Black Duck Binary Analysis on Kubernetes

Upgrading to 2023.9.0

BDBA helm chart 2023.9.0 upgrades internal PostgreSQL to 15. If you are using external postgresql, you can ignore this.

There is an init container that performs the volume upgrade automatically. However, for the volume upgrade process to work properly, application containers need to be shut down so postgresql can shut down gracefully. First figure out NAMESPACE and PREFIX.

NAMESPACE is the Kubernetes instance where BDBA is deployed. PREFIX is the release label that BDBA pods have. You can get this for example by invoking kubeget get deployment -n $NAMESPACE.

Then invoke:

export PREFIX=<bdba-release-prefix>
export NAMESPACE=<bdba-namespace>

for deployment in "$PREFIX-bdba-beat" "$PREFIX-bdba-tasks" "$PREFIX-bdba-tasks-long" "$PREFIX-bdba-updater" "$PREFIX-bdba-webapp"; do
    kubectl delete deployment "$deployment" -n "$NAMESPACE"
done

before upgrading BDBA.

After this, you can upgrade BDBA as usual.

Recovering from unclean shutdown

Failing to stop application containers using database will likely result in database volume that is not shut down cleanly and volume upgrader container will fail. Recovering from this situation requires deleting existing postgresql statefulset and rolling back to 2023.6.0 release.

Requirements

BDBA should run on fine on any public cloud provider supporting Kubernetes. Nodes should have 7 gigabytes of memory at minimum. Examples of minimum suitable nodes are:

• AWS: m5.large
• Azure: Standard_DS2_v2 (See notes below)
• GCP: n2-standard-2

BDBA should run on any supported Kubernetes versions. Onprem BDBA is tested using current version of RKE2.

Stateless vs Stateful deployment

We recommend that production BDBA instances do not contain state. BDBA helm chart includes tempalates to deploy PostgreSQL, RabbitMQ and VersityGW, which are enabled by default for easy installation but larger scale deployments should avoid storing state in Kubernetes cluster and use managed postgresql, rabbitmq and S3 -compatible storage.

Consult the scapters "External PostgreSQL", "External RabbitMQ" and "External Object Storage" for more information and required configuration on external services.

Cluster Configuration Notes

Some Kubernetes clusters are configured with "Quaranteed QOS", which essentially means that resource limits behave like resource requests.

With "Queranteed QOS" pods won't be scheduled if nodes are not able to satisfy both CPU and memory limits.

In those cases the minimum resource requirements are higher. This helm chart does not specify hard limits for PostgreSQL, RabbitMQ and Minio due to them being essential services and having them terminated by exceeding resource limits would be catastrophic. As a rule of thumb, the cluster should have at least:

• 6 cores for BDBA pods
• 4 cores for PostgreSQL, Minio, Rabbitmq and Memcached.

Also a cluster may specify default "LimitRanges" to override limits in case limits are not specified for pod. In this case, resources should be adjusted accordingly in values.yaml. Running PostgreSQL in this kind of setup would be dangerous and it is recommended to use PostgreSQL that is not running inside the cluster if this is the case.

Azure Notes

In Azure, if monitoring addons are enabled with --enable-addons monitoring, Standard_DS2_v2 instances do not have enough free memory available for BDBA.

If monitoring is needed, minimum instance size is Standard_DS3_v2. In that case node count can be decreased to 2.

Also, Azure can conflict with some S3/minio operations. This can be resolved by settings frontend.disableEc2Metadata as true.

Deploying Black Duck Binary Analysis Using the Helm Package Manager (Stateful Deployment)

This chart bootstraps Black Duck Binary Analysis deployment on a Kubernetes cluster using the Helm package manager.

Prerequisites

Before starting, you will need:

• A Kubernetes cluster with:
  • storageClass that allows persistent volumes configured.
  • NGINX Ingress Controller (not needed with OpenShift)
  • The cluster should have enough memory, preferably at least 16 gigabytes. A good entry level deployment, for example, would be two n1-standard nodes on GCP.
• Helm 3

Install Using Black Duck Repository

$ helm repo add blackduck https://repo.blackduck.com/artifactory/sig-cloudnative

Install The Chart

To install the chart with the release name testing:

$ helm upgrade testing blackduck/bdba --install --namespace bdba
Release "testing" does not exist. Installing it now.
NAME: testing
LAST DEPLOYED: Sun Feb 16 10:50:15 2020
NAMESPACE: bdba
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
        

Configuration

This section shows an example of how you can configure a Black Duck Binary Analysis instance. It's split into multiple sections for clarity.

Secrets for Services

Black Duck Binary Analysis Helm chart uses PostgreSQL. Secrets for other services are automatically provisioned, but for PostgreSQL you need to enter the password manually.

PostgreSQL Secrets

If you use a bundled PostgreSQL database, it is recommended to configure a password for PostgreSQL. During the initial installation, a PostgreSQL database is created with that specific password, and you cannot change it afterward.

You can configure a password either with the --set postgresql.postgresqlPassword="PASSWORD" parameter or by using existing secrets. To create a random password and use that, do the following before installing:

$ kubectl create secret generic bdba-postgresql-secret -n NAMESPACE --from-literal=postgresql-password=<12 random characters>
secret/bdba-postgresql-secret created

Next, add the created secret to Helm:

--set global.postgresql.existingSecret="bdba-postgresql-secret"

This ensures that the PostgreSQL secret survives upgrades.

Summary

Storage

Black Duck Binary Analysis requires storage. It can support either existing PVC claims or automatically provision volumes if storageClass is provided.

External Object Storage

Black Duck Binary Analysis by default uses minio for storing data to persistent volumes. However, Minio can be replaced with any S3-compatible object storage, including native S3 from AWS.

To use alternative object storage, both minio and versitygw needs to be disabled.

If you are using native AWS S3, you need to consider the following:

• • Bucket names need to be unique (globally).
  • S3 Region is needed.
  • To grant permissions, create role for Kubernetes nodes that allow BDBA nodes to access S3, or create AWS user that that can access the buckets and use s3AccessKeyId and s3SecretAccessKey parameters. BDBA is able to use Instance Metadata Service, and access keys are optional if Instance Metadata Service is available.

For other object storage options (like external Minio or ceph), s3Endpoint, s3AccessKeyId and s3SecretAccessKey are needed, and possibly s3Region depending on the object storage settings.

Licensing

To access the data that Black Duck Binary Analysis needs a username and password for the licensing server are required. These are credentials needed for accessing onprem updates from Black Duck Community. Without these, the installation will not function. In case you are operating an airgapped installation, you can omit these.

Web Frontend Configuration

These are generic configuration options for customization of frontend behavior.

frontend.web.rootURL is only necessary if it differs from ingress.host and ingress.tls values.

By default, URL of the BDBA service is inferred from values specified for Ingress.

frontend.web.vacuumDays accepts days in quite liberal crontab-format. Examples are sunday, to vacuum only on sunday, mon,wed,fri,sun to vacuum on monday, wednesday, friday and sunday and mon-sun to vacuum daily.

frontend.web.csrfTrustedOrigins allows specifying list of trusted origins for unsafe requests. This is needed for example when TLS is not terminated in BDBA Ingress, but there is application load balancer terminating TLS.

frontend.web.forcedHttpsUrls allows forcing all internal absolute URL links to be https. This is required if BDBA is running behind a load balancer that does not send X-Forwarded-Proto -header.

SMTP Configuration

Black Duck Binary Analysis can send emails, for example, to invite new users or to send vulnerability notifications.

LDAP Authentication

BDBA can authenticate against LDAP servers.

To setup root certificate for LDAP, issues:

$ kubectl create secret generic bdba-ldap-root --from-file=~/root-ca/
ca.pem
secret/bdba-ldap-root created

To use this as the root certificate, add --set frontend.ldap.rootCASecret=bdba-ldap-root to the Helm command line.

Monitoring

BDBA webapp can expose several prometheus gauges via /metrics endpoint.

Monitoring is disabled by default. To enable monitoring, you need to have prometheus inside the cluster. Even if monitoring is enabled, BDBA rejects metrics requests coming thru ingress unless authentication for monitoring is enabled.

There is included grafana dashboard in contrib/bdba-metrics-grafana.json which can be added to kube-prometheus-stack deployment using kubectl. For example:

$ kubectl create configmap bdba-dashboard -n <prometheus-namespace> --file=contrib/bdba-metrics-grafana.json -o yaml --dry-run|kubectl apply -f -
$ kubectl label configmap bdba-dashboard -n <prometheus-namespace> grafana_dashboard=1

Logging

BDBA uses fluentd to centrally log. Relevant application pods are joined by fluentbit sidecar.

worker.scanSpecificLogging.enabled enables scan-specific logging in worker. After scan has been completed, it uploads the logs into object storage so they can be downloaded for troubleshooting.

Cloud provider -specific settings

Some cloud providers (for example, Azure) may interfere by providing instance metadata at the same endpoint as AWS, and cause object storage operations to fail. Usage of instance metadata can be disabled.

Worker Scaling

If worker.storageClass is left empty, scanners will be deployed as Kubernetes Deployment and use ephemeral storage for work space.

However, if nodes have limited ephemeral storage available (that is, nodes contain small root disks), worker.storageClass allows reserving work space for scanners from persistent volumes. This also makes the workers run as Kubernetes StatefulSets. Each worker pod reserves it's own workspace from persistent volume.

It is recommended to keep worker.concurrency as 1 to isolate scanners from each other and to more efficiently use cluster resources.

Worker Autoscaling

To use worker autoscaling, KEDA (https://keda.sh) is required to be installed in the cluster. See KEDA's deployment instructions how to deploy KEDA.

Use worker.keda.maxReplicaCount to specify maximum number of workers to scale up to.

worker.keda.queueLength specifies the length of job queue before scaling up. By default, BDBA starts scaling up when there are two jobs waiting in the queue. If you have lots of small scans, higher value will work better, but if you have mostly large scans, 1 will trigger new worker immediately for new scan.

worker.terminationGracePeriodSeconds is required for scaling down. As KEDA starts to scale down when there are no jobs in the queue by sending termination signal to workers, BDBA workers receiving the signal will stop accepting new scan jobs but finishing their current scans. If scan does not finish in worker.terminationGracePeriodSeconds it will be forcefully killed and fail. If scans fail abruptly when downscaling, increasing this value will help.

Networking and Security

External PostgreSQL

Black Duck Binary Analysis supports external PostgreSQL. Black Duck Binary Analysis is tested against PostgreSQL 14, 15, and 17. There are no specific version restrictions as long as it is 14 or newer.

To configure external PostgreSQL, the following parameters are supported. To omit installing PostgreSQL and use external instead, specify:

--set postgresql.enabled=false.

To inject a CA secret, use the following. rootCASecretName will be named bdba-pgroot.

$ kubectl create secret generic bdba-pgroot --from-file=ca.pem
secret/bdba-pgroot created

To inject a client keypair, use the following. clientSecretName will be named bdba-pgclient.

$ kubectl create secret tls bdba-pgclient --key key.pem --cert cert.pem
secret/bdba-pgclient created

Possible values from postgresqlSslMode are specified in https://www.postgresql.org/docs/15/libpq-ssl.html

*IMPORTANT!*: It is recommended to change the default postgresql paramater values for better performance.

• work_mem = 8MB to set postgresql working memory from default 4MB to 8MB.
• random_page_cost = 1.0 to make query planner prefer index instead of sequential scanning on large queries.

External RabbitMQ

BDBA supports external RabbitMQ with mutual TLS encryption. It also works with RabbitMQ-as-a-service offerings such as Amazon MQ.

*IMPORTANT!*: It is mandatory that RabbitMQ is configured with larger-than-default consumer timeout. Some BDBA tasks are longer than RabbitMQ defaults allow and the recommended value for it is 86400000. Without this value, BDBA containers will experience unscheduled restarts and in some cases prematurely killed jobs. Similarly, rabbitmq since 4.x has decreased the maximum message size value, which needs to be increased. To set these values, add consumer_timeout = 86400000 and max_message_size = 209715200 in /etc/rabbitmq/rabbitmq.conf if rabbitmq is running as a systemd service. With other deployment models, such as managed rabbitmq, consult the documentation on how to do this.

The configuration values are:

rabbitmq.enabled needs to be set as false to enable external RabbitMQ.

brokerUrl is the connection string for RabbitMQ service. It is in the form of amqp://<user>:<password>@<host>:<port>/<vhost>.

Root Certificate

In case TLS is in use for RabbitMQ, you need to specify brokerRootCASecretName which points to a secret that contains the root certificate that RabbitMQ uses unless the RabbitMQ server certificate is signed by a known trusted Certificate Authority.

To populate the CA secret, run:

$ kubectl create secret -n bdba generic rabbitmq-ca --from-file=ca.pem

Note that the filename MUST be ca.pem. In this case, the brokerRootCASecretName would be rabbitmq-ca.

mTLS Client Authentication

If the RabbitMQ server requires mTLS client authentication, you can pass the client certificate in the brokerClientSecretName secret.

To populate the client certificate and key, run:

$ kubectl create secret tls -n dev rabbitmq-client-cert --key="client-key.pem" --cert="client.pem"

In this case, the brokerClientSecretName would be rabbitmq-client-cert.

External Memcached

BDBA uses Memcached for certain locks and caches. Usage of external Memcached is supported. It also works with memcached-as-a-service offerings such as Amazon ElastiCache. The Memcached instance can be very lightweight.

The configuration values are:

memcached.enabled needs to be set as false to enable external Memcached.

memcachedHostPort is the <host>:<port> pair for Memcached service, for example memcached:11211.

Root Certificate

In case TLS is in use for memcached, you need to specify memcachedRootCASecretName which points to a secret that contains the root certificate that Memcached uses, unless the Memcached server certificate is signed by a known trusted Certificate Authority.

To populate the CA secret, run:

$ kubectl create secret -n bdba generic memcached-ca --from-file=ca.pem

Note that the filename MUST be ca.pem. In this case, the memcachedRootCASecretName would be memcached-ca.

mTLS Client Authentication

If the Memcached server requires mTLS client authentication, you can pass the client certificate in the memcachedClientSecretName secret.

To populate the client certificate and key, run:

$ kubectl create secret tls -n dev memcached-client-cert --key="client-key.pem" --cert="client.pem"

In this case, the memcachedClientSecretName would be memcached-client-cert.

Ingress

To create a TLS secret, invoke:

$ kubectl create secret tls bdba-tls --key key.pem --cert cert.pem
secret/bdba-tls created

Configuring

Specify each parameter using the --set key=value[,key=value] argument to helm install.

For example:

$ helm upgrade testing blackduck/bdba --install \
--set frontend.licensing.username="foo@example.com" \
--set frontend.licensing.password="secret" .

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example:

$ helm upgrade testing blackduck/bdba --install -f my-values.yaml
#

Examples

Examples, with inline documentation, are provided in the examples directory.

Secrets

Key material can be saved to Kubernetes using kubectl. To set up additional root certificates, you can set them as Kubernetes secrets. PEM encoding is assumed.

$ kubectl create secret generic bdba-root --from-file=~/root-ca/ca.pem
secret/bdba-root created

To use this as the root certificate, add --set rootCASecret=bdba-root to the Helm command line.

Backing Up Database

To take backup of internal postgresql if external PostgreSQL is not in use, you can use kubectl and pg_dump.

$ kubectl exec -it -n bdba bdba-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_dump -Fc -d bdba -U bdba' >backup.pgdump

This will create backup.pgdump which is standard PostgreSQL custom-format archive that can be restored using pg_restore.

Migration from an Existing Appliance

To migrate data from an existing VM-based appliance, backup API of the appliance can be used to perform data acquisition.

Acquiring Backup

First, run:

$ curl -X POST -u admin "https://<APPLIANCE>/api/backup/"

Next, Query until the backup is ready.

$ curl -u admin "https://<APPLIANCE>/api/backup/"|jq
{
  "meta": {
    "code": 200
  },
  "backup": {
    "status": "R",
    "started": "2019-12-03T13:02:12",
    "completed": "2019-12-03T13:03:17",
    "location": "/api/backup/appliance-NNNNN.pgdump",
    "sha1sum": "684cd634b35d26dfeb1ab8c5e9b399619126c176"
  }
}

Once it is ready, get the "location" from the response and download the database dump.

$ curl -o backup.pgdump.tar -u admin "http://<APPLIANCE>/api/backup/
appliance-NNNNN.pgdump"

Stopping Services Accessing the Database

Next, you need to stop deployments that access the database. These deployments are:

• NAME-bdba-tasks

• NAME-bdba-tasks-long
• NAME-bdba-updater
• NAME-bdba-webapp

Invoke the following commands:

$ kubectl scale --replicas=0 deployment/NAME-bdba-tasks
$ kubectl scale --replicas=0 deployment/NAME-bdba-tasks-long
$ kubectl scale --replicas=0 deployment/NAME-bdba-updater
$ kubectl scale --replicas=0 deployment/NAME-bdba-webapp

Preparing the Database for pg_restore

Before Black Duck Binary Analysis database can be restored, it is required to clean up tables found in the database.

Internal Postgresql

Kubectl into the database container and run:

$ kubectl exec -it -n NS NAME-postgresql-0 bash

In the PostgreSQL pod, enter the interactive PostgreSQL shell and execute:

$ psql -h localhost -U <database-username> -W -d <database-name>

Hosted PostgreSQL

With hosted PostgreSQL, like AWS RDS, you can access the database by launching a pod that can access the database.

kubectl run -it --env="PGPASSWORD=<database-password>" --rm --image=postgres --restart=Never --command=true psqlshell -- psql -h <database-host> -U <database-username>

DB Shell

In the interactive db shell, run:

DO $$ DECLARE
r RECORD;
BEGIN
FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = 'public')
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(r.tablename) || '
CASCADE';
END LOOP;
END $$;

This will empty the database, but authentication credentials are kept intact.

Exit the db shell with ^D to proceed.

Overwriting the Database with a Backup

Copy the database dump to the PostgreSQL pod.

$ kubectl cp database.pgdump NS/NAME-postgresql-0:/tmp

Restoring the Database on Hosted PostgreSQL

Next, restore the database. In the PostgreSQL pod, execute:

$ pg_restore -O -c -C -Fc -h localhost -U <database-username> -d <databasename>
-n public /tmp/database.pgdump

Restoring the Services

Next, restore the services:

$ kubectl scale --replicas=1 deployment/NAME-bdba-tasks
$ kubectl scale --replicas=1 deployment/NAME-bdba-tasks-long
$ kubectl scale --replicas=1 deployment/NAME-bdba-updater
$ kubectl scale --replicas=N deployment/NAME-bdba-webapp

Now, you should be all set.

Openshift

Since Openshift is incompatible with security contexts specified in helm charts, security contexts need to be disabled and have openshift manage them.

To disable security context declarations in helm charts, add the following to helm command line or your values.yaml.

--set postgresql.securityContext.enabled=false \
--set rabbitmq.securityContext.enabled=false \
--set minio.securityContext.enabled=false
--set frontend.securityContext.enabled=false \
--set worker.securityContext.enabled=false \
--set memcached.securityContext.enabled=false

Because BDBA defaults to nginx Ingress class, you also need to specify Ingress class so Openshift can create routes properly. To do this, use:

$ kubectl get ingressclass

to figure out proper ingressclass and add the parameter for helm installation command.

--set ingress.class="<ingressclass>"

By default, this is "openshift-default". You can also use --set ingress.class="" to use the default as well.

Airgapped Installation

BDBA Kubernetes can operate in airgapped mode. However, it needs manual work to be kept up-to-date.

Populating Database

By default, when BDBA is given licensing username and password, it is able to fetch data updates from https://bdba.blackduck.com/. However, when installation is airgapped, this option is not possible. However, it is still possible to manually populate the internal vulnerability database and keep it up-to-date.

To populate the database, you can download dataset from https://bdba.blackduck.com/updates/vulndata/. This requires the same credential that are used for Black Duck community. You will receive "vulndata.tar.xz" which is roughly 1GB.

This can be brought to airgapped network, and inserted into running BDBA kubernetes deployment by uploading it to http(s)://<ingress-host-name>/api/bootstrap/, for example using curl:

$ curl -T vulndata.tar.xz -u admin https://<bdba-k8s-ingress>/api/bootstrap/

Populating Component Information

After each software update, also supplemental information about components should be populated into database.

This should be done also with first installation in addition to populating the database with vulnerability data.

To achieve this, download data from https://bdba.blackduck.com/updates-v2/bootstrap/. It will return bdab-onprem-data-bootstrap-YYYYMMDD-hhmmss.tar.zst.

To update the database, push it to http(s)://<ingress-host-name>/api/nvd/, for example using curl:

$ curl -T bdba-onprem-data-bootstrap-YYYYMMDD-hhmmss.tar.zst -u admin:<adminpw> https://<bdba-k8s-ingress>/api/nvd/

Keeping Database Up-to-date

Similarly, to keep database up-to-date, you can download data from https://bdba.blackduck.com/updates-v2/. It will return bdba-onprem-data-update-YYYYMMDD-hhmmss.tar.zst..

To update database, push it to http(s)://<ingress-host-name>/api/nvd/, for example using curl:

$ curl -T bdba-onprem-data-update-YYYYMMDD-hhmmss.tar.zst -u admin:<adminpw> https://<bdba-k8s-ingress>/api/nvd/

The difference with this file to the bootstrap variant is that it contains only the delta of seven days and it is faster to apply.

Upscaling

As BDBA has varied workloads by nature, there can be no absolute guidance for configuration for example for X scans / day and how machines should be provisioned.

BDBA inherintly is an application that processes workloads using queues, stores data in PostgreSQL and provides a web interface. In practice this means that all the components except PostgreSQL can be horizontally scaled and distributed among many hardware instances. PostgreSQL is the only thing that scales only vertically.

Therefore, primary focus on scaling should be placed on PostgreSQL performance. Running PostgreSQL with enough memory and fast disks is advisable. In larger deployments external PostgreSQL should be used.

In case of BDBA workloads getting stuck, increasing replicas help. Different symptons on slowness can be for example

• Web application is slow -> increase web application replicas with frontend.web.replicas parameter.
• Scan jobs stay in the queue for long time -> increase number of workers, either enabling keda or with worker.replicas parameter.
• Post processing jobs stay in the queue for long time -> increase post processing replicas with tasks.replicas parameter.

Running without service account

By default, when BDBA is using embedded minio or rabbitmq, it creates the secrets from them in the initialization job. This job creates secrets for minio and rabbitmq, and stores them as Kubernetes secrets. This requires a service account.

If for some reason service accounts are not functioning in cluster or pods do not have access to use them, you can disable the service account and create the secrets manually.

Service account relevant parameters are:

To disable automatic secrets creation, set frontend.createSecrets as false.

Following secrets then need to be created manually

1. (If using embedded minio)

   $ kubectl create secret generic -n namespace bdba-minio-secret \
     --from-literal=accesskey=<random string> --from-literal=secretkey=<random-string>

2. (If using embedded rabbitmq)

   $ kubectl create secret generic -n <namespace> bdba-rabbitmq-password-secret \
     --from-literal=rabbitmq-password=<random string>
   
   $ kubectl create secret generic -n <namespace> bdba-rabbitmq-broker-url \
     --from-literal=host=amqp://bdba:<password-set-above>@<release-name>-rabbitmq.<namespace>
   
   $ kubectl create secret generic -n <namespace> bdba-rabbitmq-erlang-cookie-secret \
     --from-literal=rabbitmq-erlang-cookie=<random string>