Skip to main content
Version: 1.1

Operational Vulnerability Scanning

This guide describes how to set up and use the Operational Container Scanning (OCS) feature in GitLab. It will go over setting up the GitLab K8s Agent on a specific K8s cluster to grant GitLab access to the K8s cluster for the OCS. The guide is based on the following GitLab Documentation:

Prerequisites​

  • A running K8s cluster
  • GitLab Ultimate
  • Helm

Step-by-Step Guide​

Step 1: Configure the repository​

To use OCS, the repository must be correctly configured. Security and Compliance and Infrastructure must be activated:

  1. Navigate to Settings->General.
  2. Click on Expand for the part Visibility, project features, permissions.
  3. Scroll down and check if Security and Compliance is activated. If not, activate it.
  4. Scroll further down and check if Infrastructure is activated. If not, activate it.

To check if it worked out, check in the menu of the repository if the menu entry Secure is visible. Under Secure entries like Security Dashboard and Vulnerability report should be present. To check if Infrastructure is correctly activated, hold the mouse over the menu entry Operate. Here the entry Kubernetes clusters should be visible.

Step 2: Add config file to repository​

Create from the root of the repository a file on the path .GitLab/agents/<agent-name>/config.yaml. The file should for now just be empty.

Please Note:

  • This must be done on the main/master branch to work.
  • \<agent-name\> will also be the cluster name you can choose in the Security Dashboard when filtering after clusters. So a good idea is to give the agent a name, which corresponds to the cluster name.

Step 3: Register the agent in GitLab​

  1. Navigate to your repository in GitLab.
  2. In the menu on the left choose Operate --> Kubernetes clusters.
  3. Click on Connect a cluster and a pop-up with a dropdown will appear.
  4. Choose the agent you created the config.yaml file for and click Register.
  5. You will get a Helm command like the following:
helm upgrade --install k8s-agent GitLab/GitLab-agent \
    --namespace GitLab-agent-k8s-agent \
    --create-namespace \
    --set image.tag=v16.10.1 \
    --set config.token=some-secret-token \
    --set config.kasAddress=wss://kas.GitLab.com

Store the token in a secure location and copy the command. The token can only be seen once and is not stored in GitLab.

Step 4: Deploy the agent on the K8s cluster​

Run the Helm command from the previous step on the K8s cluster. This will deploy the GitLab K8s Agent on the K8s cluster.

Please Note:

  • If you use an older K8s version like 1.22, you might need to use an older version of the agent. Change the image.tag to v16.8.0 to make it work.
  • If you are scarce on resources or do not need high availability then use the flag --set replicas=1 to only deploy one replica of the agent. By default two replicas are deployed.

Step 5: Verify the agent is running​

  1. Navigate to your repository in GitLab.
  2. In the menu on the left choose Operate --> Kubernetes clusters.
  3. You should see the agent you registered in the list of clusters. It should have a green checkmark and the status should be Connected.

Step 6: Activate OCS​

The activation of OCS is done over the config.yaml file. The possible options which can be configured are limited. The following shows an example of a config.yaml file with most options present:

container_scanning:
    resource_requirements:
        requests:
            cpu: '0.2'
            memory: 200Mi
        limits:
            cpu: '1.0'
            memory: 2000Mi
    cadence: '0 0 * * *'
    vulnerability_report:
        namespaces:
        - default
        - kube-system

The resource_requirements are the resource requirements for the OCS scanning pod which will run per namespace on the cluster. They are optional, with default settings as follows:

requests:
    cpu: '0.1'
    memory: 100Mi
limits:
    cpu: '0.5'
    memory: 500Mi

It is often necessary to set higher values for the limits for the limits, as the default values are quite low and will often lead to OOMKilled pods. 2GB memory and 1 CPU should be a good starting point for the limits.

  • The cadence is the cronjob schedule for the OCS scanning pod.
  • The vulnerability_report.namespaces is a list of namespaces which should be scanned.

Step 7: Check the report​

After the OCS scanning pod has run, you can check the report in the repository. Navigate to Secure --> Vulnerability Report and choose the tab Operational Vulnerabilities. Here you can see the vulnerabilities found in the scanned namespaces.

Troubleshooting​

The OCS scanning pod fails to start​

If the OCS scanning pod won't start, check the logs of the GitLab Agent pod. If you get the following logs:

{"level":"error","time":"2024-03-26T06:30:10.496Z","msg":"Error running Trivy scan","mod_name":"starboard_vulnerability","error":"OCS Scanning pod exited with unknown exit code: 128, reason: StartError","agent_id":1092884}

Then the issue most likely is that the K8s version is too old for the GitLab Agent version. Try to change the image.tag to v16.8.0.

The OCS scanning pod gets OOMKilled​

If the OCS scanning pod gets OOMKilled, check the resource requirements of the pod. The default values are quite low and will often lead to OOMKilled pods. 2GB memory and 1 CPU should be a good starting point for the limits.

The OCS scanning pods won't start due to node(s) being out of resources​

This is a common issue when the cluster is scarce on resources. Problem here is that the GitLab Agent will deploy one OCS scanning pod per namespace, which will be scanned. For now there is no good solution to this problem.