Python support

Black Duck Detect
Version
latest

Overview

Detect detectors for discovery of dependencies in Python:

  • Setuptools detectors

    • Setuptools CLI

    • Setuptools Parse

  • PIPENV detectors

    • Pipenv lock detector

    • Pipfile lock detector

  • PIP detectors

    • Pip Native Inspector

    • Pip Requirements File Parse

  • Poetry detector

  • UV detectors

    • UV CLI

    • UV Lock

Setuptools detectors

Setuptools detectors attempt to run on your project if a pyproject.toml file containing a build section with requires = ["setuptools"] or equivalent line is located. (Setuptools scans can be run in both build, if a pip installation is available, and buildless mode, if not.)

Note: Setuptools CLI detector should be run in a virtual environment, or environment with a clean global pip cache, where a pip install has only been performed for the project being scanned.

Detect parses the pyproject.toml file determining if the [build-system] section has been configured for Setuptools Pip via the requries = ["setuptools"] setting. If the setting is located and pip is installed in the environment, either in the default location or specified via the --detect.pip.path property, Setuptools CLI detector will execute in a virtual environment, if configured as suggested, and analyze the pyproject.toml, setup.cfg, or setup.py files for dependencies. If a configured pyproject.toml file is discovered but a pip executable is not, the Setuptools Parse detector will parse dependencies from the pyproject.toml, setup.cfg, or setup.py files but may not be able to specify exact package versions. If no dependencies are located in the pyproject.toml, setup.cfg, or setup.py files, or if the detectors fail the BDIO file output will not be generated in build or buildless mode. Detect will also attempt to run additional detectors if their execution requirements are met.

For setup.cfg and setup.py file parsing, the Setuptools detectors support direct mentioning of dependency files. For reference, see Dependency Management in Setuptools.

Tip: URL references, optional dependencies and file: <path to file> parameters found in setup.cfg are not supported. For setup.py files, Detect supports only literal install_requires=[...] lists with string literal entries. Programmatic population of the install_requires parameter (for example via variable references or function calls) is not supported.
Note: The --detect.pip.only.project.tree, --detect.pip.project.name, and --detect.pip.project.version.name properties do not apply to the Setuptools detectors.

PIPENV Detectors

Pipenv lock detector

The Pipenv lock detector attempts to run on your project if either a Pipfile or Pipfile.lock file is found.

Pipenv detector requires Python and Pipenv executables.

  • Detect looks for python on $PATH. You can override this by setting the python path property.

  • Detect looks for pipenv on $PATH.

The Pipenv detector runs pipenv run pip freeze and pipenv graph --bare --json-tree and derives dependency information from the output. The dependency hierarchy is derived from the output of pipenv graph --bare --json-tree. The output of pipenv run pip freeze is used to improve the accuracy of dependency versions.

To troubleshoot of the Pipenv detector, start by running pipenv graph --bare --json-tree, and making sure that the output looks correct since this is the basis from which Detect constructs the BDIO. If the output of pipenv graph --bare --json-tree does not look correct, make sure the packages (dependencies) are installed into the Pipenv virtual environment (pipenv install).

Note: The detect.pipfile.dependency.types.excluded property does not apply to the Pipenv detector.

Pipfile lock detector

The Pipfile lock detector attempts to run on your project if either a Pipfile.lock or Pipfile file is found AND neither of the Pip or Pipenv detectors applied.

Pipfile lock detector parses the Pipfile.lock file for dependency information. If the detector discovers a Pipfile file but not a Pipfile.lock file, it will prompt the user to generate a Pipfile.lock file by running pipenv lock and then run Detect again. Pipfile.lock dependencies can be filtered using the detect.pipfile.dependency.types.excluded property.

PIP Detectors

Pip Native Inspector

Pip Native Inspector attempts to run on your project if any of the following are true: a setup.py file is found, a pyproject.toml file is found, a requirements.txt is found, or a requirements file is provided using the --detect.pip.requirements.path property.

Pip Native Inspector requires Python and pip executables.

Pip Native Inspector runs the pip-inspector.py script, which uses Python/pip libraries to query the pip cache for the project, which may or may not be a virtual environment, for dependency information:

  1. pip-inspector.py queries for the project dependencies by project name, which can be discovered using setup.py, pyproject.toml, or provided using the --detect.pip.project.name property. If your project is installed into the pip cache, this discovers dependencies specified in setup.py, or pyproject.toml file.

  2. If one or more requirements files are found or provided, pip-inspector.py queries each requirements file for possible additional dependencies and details of each.

Tip: Only those packages which have been installed; using, for example, pip install, into the pip cache and appearing in the output of pip list, are included in the output of pip-inspector.py. There must be a match between the package version on which your project depends and the package version installed in the pip cache.
Note: If the packages are installed into a virtual environment for your project, you must run Detect from within that virtual environment.

Recommendations for Pip Detector

  • Be sure that Detect is locating the correct version of the Python executable; this can be done by running the logging level at DEBUG and then reading the log. This is a particular concern if your system has multiple versions of Python installed.

  • Create a setup.py or pyproject.toml file for your project.

  • Install your project and dependencies into the pip cache:

python setup.py install or pip install . (from directory where pyproject.toml is present)
pip install -r requirements.txt

  • Pip detector attempts to derive the project name using your setup.py or pyproject.toml file if you have one. If you do not have a setup.py or pyproject.toml file, you can provide the correct project name using the property --detect.pip.project.name.

  • If there are any dependencies specified in requirements.txt that are not specified in setup.py or pyproject.toml file, then provide the requirements.txt file using the Detect property.
    Important:
  • Ensure that the paths to the Python and pip executables are correctly configured, either via system environment variables or Detect properties. For projects using pyproject.toml file(s), incorrect paths may lead to Detect failures.
  • If you are using a virtual environment, be sure to switch to that virtual environment when you run Detect. This also applies when you are using a tool such as Poetry that sets up a Python virtual environment.

PIP Requirements File Parse

Pip Requirements File Parse is a buildless detector that acts as a LOW accuracy fallback for the Pip Native Inspector. This detector gets triggered for Pip projects that contain one or more requirements.txt files but Detect doesn't have access to a Pip executable in the environment where the scan is executed.

Requirements file selection (Default)

In the default case, the detector will search for a file named requirements.txt for parsing. If the file has references to other requirements.txt files in the project provided via the -r or --requirement option, the detector will try to resolve these file references and also include them for parsing.
Note: The options -r or --requirement expect a UNIX path (relative or absolute) to the other requirements files. If an invalid path is provided via these options, Detect will display a warning and continue parsing the initial requirements file.

Requirements file selection (Override)

The --detect.pip.requirements.path property can be used to provide an explicit comma-separated list of paths to requirements files that the parser should include. These files may be present in directories other than the source directory and may have file names other than the default file name "requirements.txt". Since this property is an override, only the files referenced via this property are considered for parsing, and the detector will not include any other file references in the given files content via the -r or --requirement option.

Parsing

This parser is a LOW accuracy, best-effort detector. In most cases, it can extract dependency information from the file when entries are in the format <dependency_name> == <version_name>. This is the typical format in requirements files generated with the Pip CLI using the pip freeze > requirements.txt command. The parser does not resolve any special wildcard characters in the version string, for example a version string present as 1.2.* is extracted exactly as it is. In cases where a range of versions is provided for a dependency, for example >= 1.2.3, <1.3, the parser will extract the lowest version in the range i.e. 1.2.3. Any extras added after the package name are not resolved by the parser. For example, for a package name declared as requests[security], only the requests package is extracted and not the extra option specified as [security].
Note: If any URL to a component is present in the requirements file being parsed, the parser will extract the entire URL as a component without a version.

Poetry detector

Poetry detector attempts to run on your project if either a poetry.lock or pyproject.toml file containing a tool.poetry section is found.

The Poetry detector parses poetry.lock for dependency information. If the detector discovers a pyproject.toml file but not a poetry.lock file, it will prompt the user to generate a poetry.lock by running poetry install and then run Detect again. The Poetry detector extracts the project's name and version from the pyproject.toml file. If it does not find a pyproject.toml file, it will defer to values derived by git, from the project's directory, or defaults.

When the --detect.poetry.dependency.groups.excluded property is specified, presence of both poetry.lock and pyproject.toml files is required for this detector to run successfully.

UV Package Manager

One of the UV detectors will run on your project if a pyproject.toml file containing section [tool.uv] is found.

The UV detectors extract the project's name and version from the pyproject.toml file. If these are not found in a pyproject.toml file, default values will be used.

UV has two detectors:

UV CLI detector

UV CLI will run if the uv executable is found along with a pyproject.toml file. It will run uv tree commands to find dependencies for the project. By default, the UV CLI detector includes main dependencies, all dependency groups, and all optional extras. Use the detect.uv.dependency.groups.excluded property to exclude specific dependency groups from the scan.

To restrict scanning to specific dependency groups while excluding standard dependencies and optional extras, use the detect.uv.dependency.groups.only property. When this property is set, Detect limits analysis to the explicitly listed dependency groups defined in the project's pyproject.toml. You can specify multiple groups as a comma-separated list (for example: detect.uv.dependency.groups.only='dev,lint'). This configuration applies exclusively to groups defined under the [dependency-groups] section; dependencies declared as extras under [project.optional-dependencies] are not included when this property is enabled. If both detect.uv.dependency.groups.only and detect.uv.dependency.groups.excluded are configured, the exclusion setting takes precedence for any overlapping groups, and Detect will log a warning to indicate the conflict.

UV Lock detector

If the uv executable is not found, the UV Lock detector will run if either uv.lock or requirements.txt file is found in the source directory of the project.

UV Lock detector will parse uv.lock, requirements.txt, or both to find project dependencies.

Note: UV Lock detector will run if there is no uv.lock file in the source; however, an uv.lock file is recommended for the highest result accuracy. Parsing only requirements.txt is considered LOW accuracy as there is no dependency source information.

Dependency and Workspace Inclusions/Exclusions

UV Properties supports exclusion of dependency groups. By default, the UV detector includes main dependencies, all dependency groups, and all optional extras. Use the detect.uv.dependency.groups.excluded property to exclude specific dependency groups from the scan. Since uv has a concept of workspaces, they can be included and excluded using the properties provided. The workspace member provided in the property should be identical to the key name under tool.uv.sources since dependencies are created under the same key name in the tree and uv.lock file. For excluding dependency groups and workspaces, presence of uv.lock or uv executable is required.