devonfw guide

Reading License Information with Readers

Different Readers are available to import raw component / license information for different technologies. This chapter describes how to setup the different build / dependency management systems to create the required input and how to configure the corresponding reader.

Maven

For the export of the licenses from a maven based project the license-maven-plugin is used, which can directly be called without the need to change anything in the pom.xml.

To generate the input file required for Solicitor the License Plugin needs to be executed with the following command:

mvn org.codehaus.mojo:license-maven-plugin:1.14:aggregate-download-licenses -Dlicense.excludedScopes=test,provided

The generated output file named licenses.xml (in the directory specified in the plugin config) should look like the following:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<licenseSummary>
    <dependency>
      <groupId>org.hsqldb</groupId>
      <artifactId>hsqldb</artifactId>
      <version>2.4.1</version>
      <licenses>
        <license>
          <name>HSQLDB License, a BSD open source license</name>
          <url>http://hsqldb.org/web/hsqlLicense.html</url>
          <distribution>repo</distribution>
        </license>
      </licenses>
    </dependency>
    <dependency>
	...
    <dependency>
      <groupId>org.apache.maven</groupId>
      <artifactId>maven-compat</artifactId>
      <version>3.3.9</version>
      <licenses>
        <license>
          <name>Apache License, Version 2.0</name>
          <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
          <distribution>repo</distribution>
        </license>
      </licenses>
    </dependency>
  </dependencies>
</licenseSummary>

In Solicitor the data is read with the following reader config:

"readers" : [ {
  "type" : "maven",
  "source" : "file:target/generated-resouces/licenses.xml",
  "usagePattern" : "DYNAMIC_LINKING"
} ]

(the above assumes that Solicitor is executed in the maven projects main directory)

CSV

The CSV input is normally manually generated and should look like this:

org.eclipse;albireo;0.0.3;Eclipse Public License - v 1.0;https://www.eclipse.org/legal/epl-v10.html
org.antlr.runtime;antlr;4.6.0;BSD;https://www.antlr.org/license.html
org.eclipse.equinox.p2;Equinox p2 Provisioning for IDEs;2.3.2;Eclipse Public License - v 1.0;http://www.eclipse.org/legal/epl-v10.html
org.springframework;spring;5.0.6;Apache 2;https://www.apache.org/licenses/LICENSE-2.0
org.eclipse.xtend;xtend;2.2.0;MIT;https://spdx.org/licenses/MIT#licenseText

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "csv",
  "source" : "file:path/to/the/file.csv",
  "usagePattern" : "DYNAMIC_LINKING"
} ]

The following 5 columns need to be contained in order (separated with ";"):

groupId
artifactId
version
license name
license URL

Additionally, an optional configuration can be set in order to customize the given structure of the csv file e.g.:

"readers" : [ {
  "type" : "csv",
  "source" : "file:path/to/the/file.csv",
  "usagePattern" : "DYNAMIC_LINKING",
  "configuration" : {
	"charset" = "UTF-8",
	"artifactId" : "0",
	"version" : "1",
	"format" : "EXCEL",
	"skipHeaderRecord" : "true",
	"delimiter" : ";"
  }
} ]

The minimum of following 2 configuration settings need to be contained:

artifactId
version

With these settings one can specify the position of the value within the csv file. Additional positional settings include:

groupId
license
licenseUrl

If a charset needs to be specified, one can use the following option:

charset (string, specified charset for reader e.g. UTF-8)

Furthermore, one can configure a range of other csv structure options based on the Apache Commons CSV API:

allowDuplicateHeaderNames (boolean)
allowMissingColumnNames (boolean)
autoFlush (boolean)
commentMarker (char)
delimiter (string)
escape (char)
ignoreEmptyLines (boolean)
ignoreHeaderCase (boolean)
ignoreSurroundingSpaces (boolean)
nullString (string)
quote (char)
recordSeparator (string)
skipHeaderRecord (boolean)
trailingDelimiter (boolean)
trim (boolean)

These configurations may also be used to overwrite options of a predefined format, which can be set with:

format (string, predefined format e.g. EXCEL)

Important: In case that a component has multiple licenses attached, there needs to be a separate line in the csv file for each license.

The CSV reader currently does not fill the attribute packageUrl. Any functionality/reporting based on this attribute will be disfunctional for data read by the CSV reader.

NPM

For NPM based projects, the NPM License Checker (https://www.npmjs.com/package/license-checker) plugin can be used. The NPM License Crawler plugin is deprecated.

NPM License Checker

To install the NPM License Checker the following command needs to be executed.

npm i license-checker -g

To get the licenses, the checker needs to be executed like the following example. We require JSON output here with "--json" and developer dependencies can/should be excluded with "--production".

license-checker --production --json > /path/to/licenses.json

The export should look like the following

{
  "foo@0.0.1": {
    "licenses": "MIT",
    "repository": "https://github.com/somebody/foo",
    "url": "http://www.somebody.com/",
    "path": "C:\\Users\\me\\foo",
    "licenseFile": "C:\\Users\\me\\foo\\LICENSE"
  },
  ...
  "foo-bar@1.0.0": {
    "licenses": [
      "AFLv2.1",
      "BSD"
    ],
    "repository": "https://github.com/nobody/foo-bar",
    "path": "C:\\Users\\me\\foo-bar"
  }
}

Source: https://www.npmjs.com/package/license-checker

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "npm-license-checker",
  "source" : "file:path/to/licenses.json",
  "usagePattern" : "STATIC_LINKING"
} ]

NPM License Crawler

This reader is deprecated and should no longer be used. It requires a specific dependency (license-checker) which is not available on official npm repositories anymore and scans additional developer dependencies. Use NPM License Checker (with --production option) instead. See List of Deprecated Features.

To install the NPM License Crawler the following command needs to be executed.

npm i npm-license-crawler -g

To get the licenses, the crawler needs to be executed like the following example

npm-license-crawler --dependencies --csv licenses.csv

The export should look like the following (The csv file is "," separated)

"module name","licenses","repository","licenseUrl","parents"
"@angular-devkit/architect@0.10.6","MIT","https://github.com/angular/angular-cli","https://github.com/angular/angular-cli/raw/master/LICENSE","mythaistar-restaurant"
"@types/selenium-webdriver@2.53.43","MIT","https://github.com/DefinitelyTyped/DefinitelyTyped","https://github.com/DefinitelyTyped/DefinitelyTyped/raw/master/LICENSE","protractor:mythaistar-restaurant"
"aws-sign2@0.6.0","Apache-2.0","https://github.com/mikeal/aws-sign","https://github.com/mikeal/aws-sign/raw/master/LICENSE","loggly:mythaistar-restaurant"
"is-posix-bracket@0.1.1","MIT","https://github.com/jonschlinkert/is-posix-bracket","https://github.com/jonschlinkert/is-posix-bracket/raw/master/LICENSE","mythaistar-restaurant"
"npm-package-arg@5.1.2","ISC","https://github.com/npm/npm-package-arg","https://github.com/npm/npm-package-arg/raw/master/LICENSE","npm:npx:mythaistar-restaurant"
"readable-stream@2.0.6","MIT","https://github.com/nodejs/readable-stream","https://github.com/nodejs/readable-stream/raw/master/LICENSE","bl:mythaistar-restaurant"

Source: https://www.npmjs.com/package/npm-license-crawler

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "npm-license-crawler-csv",
  "source" : "file:path/to/licenses.csv",
  "usagePattern" : "STATIC_LINKING"
} ]

Yarn

To generate the input file required for Solicitor, yarn needs to be executed with the following command within the directory that contains the project’s package.json (we require JSON output here):

yarn licenses list --json > /path/to/yarnlicenses.json

The export should look like the following

{"type":"progressStart","data":{"id":0,"total":2}}
{"type":"progressTick","data":{"id":0,"current":1}}
{"type":"progressTick","data":{"id":0,"current":2}}
{"type":"progressFinish","data":{"id":0}}
{"type":"table","data":{"head":["Name","Version","License","URL","VendorUrl","VendorName"],"body":[["test","11.0.0","Apache-2.0","https://github.com/mrtest/test","http://test.com","Mr.Test"],["@test/testing","7.16.3","MIT","https://yarnpkg.com/package/@test/testing","TestCase"]]}}

Source: https://classic.yarnpkg.com/en/docs/cli/licenses/

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "yarn",
  "source" : "file:path/to/yarnlicenses.json",
  "usagePattern" : "STATIC_LINKING"
} ]

Pip

To generate the input file required for Solicitor, one has to follow two steps:

Capsulate software with all relevant dependencies/requirements in a virtual environment (venv)
Install the pip-licenses plugin within this virtual environment

After that, we execute following command within the virtual environment to extract the input file (we require JSON output here):

pip-licenses --from=all --format=json --with-urls --with-license-file > piplicenses.json

The export should look like the following

[
  {
    "License-Classifier": "MIT License",
    "License-Metadata": "MIT",
    "LicenseFile": "c:\\users\\path\\LICENSE.txt",
    "LicenseText": "Permission...",
    "Name": "test",
    "URL": "http://test.org",
    "Version": "2021.1"
  },
  {
    "License-Classifier": "BSD License",
    "License-Metadata": "BSD-3-Clause",
    "LicenseFile": "c:\\users\\path\\LICENSE",
    "LicenseText": "Redistribution...",
    "Name": "test2",
    "URL": "https://github.com/test/test2",
    "Version": "0.4.2"
  }
]

Source: https://pypi.org/project/pip-licenses/

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "pip",
  "source" : "file:path/to/piplicenses.json",
  "usagePattern" : "DYNAMIC_LINKING"
} ]

OSS Review Toolkit (ORT)

In order to use the analyzer library of ORT, one must first install the software and run it to generate the result file. The detailed way on installing ORT can be found here and a tutorial on how to run the analyzer library can be found here.

Usually, the command to run the analyzer and get extract the result file from a project looks like this:

docker run -v C:\\path\\to\\project/:/project ort --info analyze -f JSON -i /project -o /project/ort/analyzer

Note that this command only works for the installation via Docker and that we require JSON as the output format. For other installation methods, you need to adjust the command accordingly.

It might also be necessary to set up a customized configuration for the analyzer. This can be achieved through a configuration file. The default path for that is the .ort/config/ directory below the current user’s home directory. We can place a ort.conf file there, in which we can declare various configurations e.g. allowing dynamic versions in npm components via

analyzer {
    allowDynamicVersions = true
}

Further information about the configuration file can be found here.

The result file should look like the following

{
  "repository" : {
    "vcs" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "vcs_processed" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "config" : { }
  },
  "analyzer" : {
    "start_time" : "testStartTime",
    "end_time" : "testEndTime",
    "environment" : {
      "ort_version" : "23ca00df86",
      "java_version" : "11.0.15",
      "os" : "Linux",
      "processors" : 4,
      "max_memory" : 1631584256,
      "variables" : {
        "JAVA_HOME" : "/opt/java/openjdk",
        "ANDROID_HOME" : "/opt/android-sdk",
        "GOPATH" : "/tmp/go"
      },
      "tool_versions" : {
        "NPM" : "8.5.0"
      }
    },
    "config" : {
      "allow_dynamic_versions" : false
    },
    "result" : {
      "projects" : [ {
        "id" : "Maven:groupId:artifactId:version",
        "definition_file_path" : "",
        "declared_licenses" : [ ],
        "declared_licenses_processed" : { },
        "vcs" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "vcs_processed" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "homepage_url" : "",
        "scope_names" : [ "compile" ]
      }],
      "packages" : [ {
        "package" : {
          "id" : "Maven:testGroupId:testArtifactId:testVersion",
          "purl" : "pkg:maven/testGroupId/testArtifactId@testVersion",
          "authors" : [ "testAuthor" ],
          "declared_licenses" : [ "Apache License, Version 2.0" ],
          "declared_licenses_processed" : {
            "spdx_expression" : "Apache-2.0",
            "mapped" : {
              "Apache License, Version 2.0" : "Apache-2.0"
            }
          },
          "description" : "testDescription",
          "homepage_url" : "https://test.com/test",
          "binary_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/testGroupId/testArtifactId/testVersion/testArtifactId-testVersion.jar",
            "hash" : {
              "value" : "testHash",
              "algorithm" : "SHA-1"
            }
          },
          "source_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/testGroupId/testArtifactId/testVersion/testArtifactId-testVersion-sources.jar",
            "hash" : {
              "value" : "testHash2",
              "algorithm" : "SHA-1"
            }
          },
          "vcs" : {
            "type" : "",
            "url" : "",
            "revision" : "",
            "path" : ""
          },
          "vcs_processed" : {
            "type" : "Git",
            "url" : "https://github.com/testproject.git",
            "revision" : "",
            "path" : "testproject"
          }
        },
		"curations" : [ ]
      } ],
      "issues" : {},
      "dependency_graphs" : {},
      "has_issues" : true
    }
  },
  "scanner" : null,
  "advisor" : null,
  "evaluator" : null
}

Source: https://github.com/oss-review-toolkit/ort#analyzer

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "ort",
  "source" : "file:path/to/analyzer-result.json",
  "usagePattern" : "DYNAMIC_LINKING"
} ]

The ORT reader currently does not yet fill the attribute licenseUrl. Any functionality/reporting based on this attribute will be disfunctional for data read by the ORT reader.

Gradle (Windows)

For the export of the licenses from a Gradle based project the Gradle License Plugin is used.

To install the plugin some changes need to be done in build.gradle, like following example

buildscript {
  repositories {
    maven { url 'https://oss.jfrog.org/artifactory/oss-snapshot-local/' }
  }

  dependencies {
    classpath 'com.jaredsburrows:gradle-license-plugin:0.8.5-SNAPSHOT'
  }
}

apply plugin: 'java-library'
apply plugin: 'com.jaredsburrows.license'

Afterwards execute the following command in the console:

For Windows (Java Application)

gradlew licenseReport

The Export should look like this:

[
    {
        "project": "Apache Commons Codec",
        "description": "The Apache Commons Codec package contains simple encoder and decoders for\n     various formats such as Base64 and Hexadecimal.  In addition to these\n     widely used encoders and decoders, the codec package also maintains a\n     collection of phonetic encoding utilities.",
        "version": "1.11",
        "developers": [
			...
        ],
        "url": "http://commons.apache.org/proper/commons-codec/",
        "year": "2002",
        "licenses": [
            {
                "license": "Apache License, Version 2.0",
                "license_url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
            }
        ],
        "dependency": "commons-codec:commons-codec:1.11"
    },
    ...
    {
        "project": "XmlBeans",
        "description": "XmlBeans main jar",
        "version": "3.0.2",
        "developers": [
			...
        ],
        "url": "https://xmlbeans.apache.org/",
        "year": null,
        "licenses": [
            {
                "license": "The Apache Software License, Version 2.0",
                "license_url": "http://www.apache.org/licenses/LICENSE-2.0.txt"
            }
        ],
        "dependency": "org.apache.xmlbeans:xmlbeans:3.0.2"
    }
]

Source: https://github.com/jaredsburrows/gradle-license-plugin

In Solicitor the data is read with the following part of the config

"readers" : [ {
  "type" : "gradle2",
  "source" : "file:path/to/licenses.json",
  "usagePattern" : "DYNAMIC_LINKING"
} ]

The former reader of type gradle is deprecated and should no longer be used. See List of Deprecated Features.

Gradle (Android)

For the Export of the the Licenses from a Gradle based Android Projects the Gradle License Plugin is used.

To install the Plugin some changes need to be done in the build.gradle of the Project, like following example

buildscript {
  repositories {
    jcenter()
  }

  dependencies {
    classpath 'com.jaredsburrows:gradle-license-plugin:0.8.5'
  }
}

Also there is a change in the build.gradle of the App. Add the line in the second line

apply plugin: 'com.android.application'

Afterwards execute the following command in the Terminal of Android studio: For Windows(Android Application)

gradlew licenseDebugReport

The Export is in the following folder

$Projectfolder\app\build\reports\licenses

It should look like this:

[
    {
        "project": "Java Client for Google Maps Services",
        "description": "Use the Google Maps API Web Services in Java! https://developers.google.com/maps/documentation/webservices/",
        "version": "0.9.4",
        "developers": [
            "Mark McDonald",
            ...
        ],
        "url": "https://github.com/googlemaps/google-maps-services-java",
        "year": null,
        "licenses": [
            {
                "license": "The Apache Software License, Version 2.0",
                "license_url": "http://www.apache.org/licenses/LICENSE-2.0.txt"
            }
        ],
        "dependency": "com.google.maps:google-maps-services:0.9.4"
    },
    {
        "project": "Retrofit",
        "description": null,
        "version": "2.6.1",
        "developers": [],
        "url": null,
        "year": null,
        "licenses": [
            {
                "license": "Apache 2.0",
                "license_url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
            }
        ],
        "dependency": "com.squareup.retrofit2:retrofit:2.6.1"
    }
]

Source: https://github.com/jaredsburrows/gradle-license-plugin

In Solicitor the Data is read with the following part of the config

"readers" : [ {
      "type" : "gradle2",
      "source" : "file:$/input/licenses.json",
      "usagePattern" : "DYNAMIC_LINKING"
   	} ]

The former reader of type gradle is deprecated and should no longer be used. See List of Deprecated Features.

CycloneDX

The CycloneDX reader can read SBOMs in CycloneDX 1.4 or 1.5 format (https://cyclonedx.org/specification/overview/). CDXGEN (https://github.com/CycloneDX/cdxgen) is one tool which can create an SBOM in the required format.

To install CDXGEN, the following command needs to be executed.

sudo npm install -g @cyclonedx/cdxgen

To run CDXGEN, change into the project directory containing the build file (i.e. pom.xml, package.json). For npm projects, execute "npm-install" before running CDXGEN to create a package-lock.json.

Set the FETCH_LICENSE environmental variable, to fetch the declared licenses.

export FETCH_LICENSE=true

Then execute the following command:

cdxgen -o sbom.json

The export should look like the following

xref:devonfw-guide_solicitor.wiki_files_sbom.json[devonfw-guide/solicitor.wiki_files/sbom.json]

In Solicitor, the data is read with the following part of the config

"readers" : [ {
      "type" : "cyclonedx",
      "source" : "file:$/input/sbom.json",
      "usagePattern" : "DYNAMIC_LINKING"
   	} ]

Currently, Solicitor only has packageUrlHandlers for maven, npm and pip. For all other package types, Solicitor will ignore the packageUrl.