Advanced Detect topics

Detect Docker Inspector discovers dependencies in the target image by making a request to an image inspector service (running inside a container). The image inspector service works as follows:

1. Reads the target image, and constructs the file system that the container would have at time zero if you were to run the image.

2. Finds the linux package manager database in that file system.

3. Runs its own linux package manager (invoking its "list" function: "dpkg -l", or "apk info -v") on the package manager database from the target image to generate the list of packages installed in the target image. If the package manager database type of the target image does not match the image inspector's package manager type, the image inspector will redirect the request to the image inspector that can process the target image (which starts again at step 1).

4. Translates that package list to BDIO and uploads it to Black Duck SCA, or returns it to the the caller (e.g. Detect) to upload to Black Duck SCA.

5. Returns the constructed file system to the caller (e.g. Detect) for signature scanning.

The user ID and group ID of the image inspector service process will (in general) be different from the user ID and group ID of the Detect Docker Inspector process. Consequently, the the environment must be confugred so that files created by Detect Docker Inspector are readable by all. On Linux, this means an appropriate umask value (for example 002 or 022). On Windows, the working directory must be readable by all. In addition, the permissions on a Docker tarfile passed via the docker.tar property must allow read by all.

Docker may restrict file sharing to certain directories that you've configured in Docker > Settings > Resource > File Sharing. To enable Detect Docker Inspector to mount the volumes that it needs, you may need to enabling file sharing for directory $HOME/blackduck.

By default Detect Docker Inspector will produce BDIO containing a component graph consisting only of components (linux packages).

Alternatively, you can direct Detect Docker Inspector to organize components by image layer by setting property bdio.organize.components.by.layer=true. Run this way, Detect Docker Inspector will produce BDIO containing image layers at the top level of the graph, and components associated with each layer appearing as children of that layer. This structure is visible in from the Black Duck SCA project version Source display.

A side effect of this components-under-layers graph structure is the categorization by Black Duck SCA of all components as Transitive.

In the BDIO, layers are named Layer{index}_{layer digest}, where {index} is a two digit index starting at 00 to indicate layer ordering within the image, and {layer digest} is the layer digest with ":" replaced with "_". For example, the first layer of an image could be named: Layer00_sha256_1bcfbfaf95f95ea8a28711c83085dbbeceefa11576e1c889304aa5bacbaa6ac2.

This feature produces BDIO in which the same component may appear at multiple points in the graph, so only Black Duck SCA versions 2021.8.0 and newer have the ability to correctly display graphs organized by layer, if Admin > System Settings > Scan > Component Dependency Duplication Sensitivity is set high enough to avoid removal of components that appear multiple times in the graph (minimum: 2).

When organizing components by layer, you must also choose whether or not to include removed (whited-out) components in the output.

If you are interested in components from the application layers of your image, but not interested in components from the underlying platform layers, you can exclude components from platform layers from your results.

For example, if you build your application on ubuntu:latest (your Dockerfile starts with FROM ubuntu:latest), you can exclude components from the Ubuntu layer(s) so that the components generated by Detect Docker Inspector contain only components from your application layers.

Running on a non-Windows system, Docker can neither pull nor build a Windows image. Consequently, you cannot run Detect Docker Inspector on a Windows Docker image using the either the docker.image or docker.image.id property. Instead, you must, using a Windows system, pull and save the target image as a .tar file, and pass that .tar file to Detect Docker Inspector using the docker.tar property.

By default, Detect Docker Inspector starts, uses, and then stops and removes either one or two containerized image inspector services per run. This may be appropriate when scanning a single image, but when scanning many images it is highly inefficient, and it doesn't support concurrent execution of multiple Detect Docker Inspector runs.

The scanning of many images is completed significantly faster by starting the image inspector services once, and running multiple instances of Detect Docker Inspector, so that each one sends requests to the already-running image inspector services.

The following script illustrates how this is done in a Docker environment:

curl -O [source_raw_content_url_base]/[source_repo_organization]/[project_name]/master/deployment/docker/batchedImageInspection.sh

To keep the example simple, this script only starts the Alpine image inspector service. In general, you must start the Ubuntu image inspector service for inspecting images built from dpkg-based Linux distros.

If you invoke Detect Docker Inspector with an image reference; in other words, a repo:tag value versus a .tar file, it uses the docker-java library, to access the Docker registry to pull the image.

If you can pull an image using docker pull from the command line, then you will be able to configure Detect Docker Inspector to pull that image. The docker-java library can often be configured the same way as the Docker command line utility (docker).

There are also other ways to configure docker-java. For more information on configuring docker-java (and therefore Detect Docker Inspector) for your Docker registry, refer to the configuration information in the docker-java documentation.

For example, one way to configure your Docker registry username and password is to create a file in your home directory named .docker-java.properties that configures username and password:

registry.username=mydockerhubusername
registry.password=mydockerhubpassword

If you need to override the DOCKER_HOST value, set property use.platform.default.docker.host to false. When use.platform.default.docker.host is set to false, Detect Docker Inspector does not override any of the configuration settings in the code, so all other docker-java configuration methods such as properties, system properties, and system environment, are available to you.

When use.platform.default.docker.host is set to true (the default value) and Detect Docker Inspector is running on Windows, Detect Docker Inspector overrides only the DOCKER_HOST value (setting it to "npipe:////./pipe/docker_engine").

When you want to run Detect on a directory that exists within a Docker image, you can use the following approach:

1. Run Detect on the image to generate the container filesystem for the image.

2. Run Detect on a directory within that container filesystem.

Detect performs these actions without running the image/container.

To see a simple example that illustrates this approach, use the following commands to download these sample files:

curl -O [source_raw_content_url_base]/[source_repo_organization]/[project_name]/master/deployment/docker/runDetectInImageDir/runDetectInImageDir.sh
curl -O [source_raw_content_url_base]/[source_repo_organization]/[project_name]/master/deployment/docker/runDetectInImageDir/Dockerfile

Review the script before running it to make sure the side effects (files and directories that it creates) are acceptable. You must make the script executable before you run it.

To use iScan to scan a specific directory within an image:

1. Run Detect Docker Inspector on the target image to get the container file system. You can also do this using Detect using --detect.docker.passthrough.* properties. Include the following Detect Docker Inspector properties:

--upload.bdio=false                        # disable BDIO upload
--output.include.containerfilesystem=true  # tell DI to output the container file system
--output.path={your output dir}            # tell DI where to put the output

1. Locate the container file system in the output directory (*.tar.gz) and untar it.

2. cd into the directory within the untared container file system that you want to scan.

3. Invoke Detect there.

To exclude certain files and/or directories from the returned file system, you can specify that list of directories with the property --output.containerfilesystem.excluded.paths.

For example, if you are invoking Detect Docker Inspector from Detect, and want Detect to exclude the /etc and /usr/bin directories from the signature scan, you could run Detect like this:

./detect.sh --detect.docker.image=ubuntu:latest --detect.docker.passthrough.output.containerfilesystem.excluded.paths=/etc,/usr/bin

Detect Docker Inspector supports OCI image archives (.tar files) passed to it via the Detect detect.docker.tar property.

Detect Docker Inspector derives the image repo:tag value for each manifest in the index.json file from an annotation with key org.opencontainers.image.ref.name if present.

If insufficient information exists in the image's manifest.json for the repo name and tag, Detect Docker Inspector will default to using the image name, (without the .tar extension), as the project name and "latest" as the project version in the BOM. These default values can be overridden by using the detect.project.name and detect.project.version.name properties respectively.

If the OCI archive contains multiple images, Detect Docker Inspector constructs the target repo:tag from the values of Detect properties detect.docker.passthrough.docker.image.repo and detect.docker.passthrough.docker.image.tag (if detect.docker.passthrough.docker.image.tag is not set it defaults to "latest"), and looks for a manifest annotation with key org.opencontainers.image.ref.name that has value matching the constructed target repo:tag. If a match is found, Detect Docker Inspector inspects the matching image. If no match is found, or detect.docker.passthrough.docker.image.repo is not set, Detect Docker Inspector fails.