Setting up the KnowledgeBase Server

This chapter lists the requirements and describes the process of setting up the Black Duck Air-gapped KnowledgeBase server.

Prerequisites

The following are the system requirements to install and configure the Air-gapped KB.

Operating System: Ubuntu 20.04 LTS or later
Docker Community Edition: Latest version
Docker Compose: Latest 1.x version
CPU: 64 cores
Memory: 256 GB RAM
Available Disk Space: Minimum of 35 TB available to /var/lib/docker, which is the default docker location.

This mount should be expandable (not on the root partition), as should the source of the storage to enable data growth.

Note: Black Duck recommends using solid state drives because of their better performance and I/O speeds.

Installing and configuring the Air-gapped KB

To install and configure the Air-gapped KB:

Copy software and data files to the air-gapped machine.

Note: The assumption is that you are copying the software and data that is downloaded from the Google Cloud bucket from portable storage on to the air-gapped system. This step might take a long time because the data is about 2 TB.

Tip: Use a tmux session to execute commands for each step. Refer to https://linuxize.com/post/getting-started-with-tmux/ for more information. The FAQ section has an example on how to get a tmux session running.
1. To copy software from portable media:
  
  Create a folder for data and a folder for software on the target machine and then copy the software as shown in the following example.
  
  For example on an Ubuntu system, to copy the software from the portable media to the air-gapped machine, on the air-gapped machine use the following commands:
```
mkdir ~/software
cp -r <source_path> ~/software
```
  The source path is on the portable storage media.
2. To copy data from portable media:
  
  Copy the data from the portable media to the storage mount point at /var/lib/docker/ into a data folder by using the following command:
```
cd /var/lib/docker/
mkdir data
cp -r <source_path> /var/lib/docker/data
```
  When all the data is copied over, create a single extract folder and extract the tarballs into that folder.
  
  Tip: If using tmux, it would be good to extract the tarballs in parallel in different tmux window sessions to save time.
  
  Note: If you don't want to use snippets, you can skip the fourth and fifth tmux sessions
```
cd /var/lib/docker/data/
mkdir extract
```
  First tmux session named solr
```
tar -xvzf kb_search_*.tar.gz -C extract/
```
  Second tmux session named kb_spider
```
tar -xvzf kb_spider_api_*.tar.gz -C extract/
```
  Third tmux session named kb_snippet
```
tar -xvzf snippet_db_*.tar.gz -C extract/
```
  Fourth tmux session named ikb_match
```
cd ikb_match
chmod +x extract.sh
./extract.sh --target=/var/lib/docker/data/extract/
```
  Fifth session named fingerprints
```
tar -xvzf fingerprints_*.tar.gz -C /var/lib/docker/data/
```
  This process might last for several hours, depending on the I/O device. After the data is extracted, you can set up the Docker images.
Set up the KB images. For this step, the Docker daemon must be up and running. Ensure Docker is installed correctly; to verify, use the sudo docker ps command.
1. Navigate to the KBServices folder and load the Docker images by using the following commands.
```
cd ~/software/KBServices/
sh load_images.sh
```
  It's helpful to check if there are any existing images and containers loaded by using the following commands:
```
docker images ls
docker container ls
```
Create and mount data volumes.
1. To create Docker volumes for the data to be mounted:
  
  Navigate to the setup folder in the kb-orch/setup/ structure, and use the following commands:
```
cd ~/software/Orchestration/
tar xf kb-onprem*.tar
cd kb-orch/setup
sh create-volumes.sh
```
  To verify volume creation, use the following command:
```
docker volume ls
```
  When the volumes are created, stay in the same setup folder and use the following steps to mount the data into the created volumes.
2. To mount the data, you must recall the path where you extracted your data archives. In the example, we used /var/lib/docker/data/extract.
  
  Note: This is a time-consuming step. To continue working in parallel, use tmux for the following set of commands.
  
  The setup files are configured to move extracted data into the mount location, and then give the docker containers access to it as the configured users. Since data is being moved instead of being copied, space is saved.
  
  Tip: If you have a data extracted on the same mount location where your volume is (same logical partition) the move should be instantaneous. However, if you are using slower storage, or copying from outside the logical partition, execute this in a tmux session.
  
  To mount data sets, use the following:
```
sh setup-api-data.sh /var/lib/docker/data/extract/
sh setup-search-data.sh /var/lib/docker/data/extract/
sh setup-match-non_baseline.sh /var/lib/docker/data/extract/
sh setup-snippet-data.sh /var/lib/docker/data/extract/
```
  When these steps are completed, data is mounted to containers and is ready for use with Black Duck SCA. The next step is to install Black Duck SCA.
Add the hostname and IP address of the KB endpoint to the bds.security.hostHeaderAllowlistCsv key in the overrides.properties file:
```
cd /var/lib/docker/Orchestration/kb-orch/config
vi overrides.properties
```
For example:
```
bds.security.hostHeaderAllowlistCsv=127.0.0.1,mykb.organization.com
```
Bring up KB Services.

Go to the following folder and use the following commands to bring up the KB services:
```
cd ~/software/Orchestration/kb-orch/
docker-compose -p hub -f docker-compose.onprem.yml up -d
```
Validate services:

To check if all KB services are up as expected, run docker ps command:
```
docker ps
```
Note: Ensure that the nginx container (blackducksoftware/kb_nginx) is running. This is the load balancer guiding your requests.
To verify this, open up a browser window from somewhere that can access the KB server, and type the following:
```
http://<ip_address>/api/authentication-health
```
Ensure specific ports are open.

For the docker containers to communicate behind the load balancer, the following ports on that machine must be opened: 443, 8080, 80 and 5432, which enable communication between docker containers.