Deploying Sourcegraph executors

Executors can be deployed in a variety of manners. The supported deployment options are:

• Linux Binary Service (Firecracker compatible)
• Terraform on AWS or GCP (Firecracker compatible)
• Native Kubernetes
• Docker-in-Docker on Kubernetes
• Docker-Compose

See deciding which executor deployment method to use for more information on these different deployment options.

Resource recommendations

It is recommended to set the resources based on the number of jobs an instance will process in parallel.

A single job should have the following resources available.

• CPU: 4
• Memory: 12GB
• Disk: 20GB

So, if you expect an Executor instance to process up to 4 jobs in parallel, the recommended resources for the machine are

• CPU: 16
• Memory: 48GB
• Disk: 80GB

The above recommended resources can be changed to fit your constraints. See below for configuring resources for a Job.

Job configuration

The maximum number of Jobs an Executor instance can run in parallel is configured by the Environment Variable EXECUTOR_MAXIMUM_NUM_JOBS.

The CPU and Memory usage of an individual Job is configured by the Environment Variables EXECUTOR_JOB_NUM_CPUS and EXECUTOR_JOB_MEMORY.

See executor configuration for a full list of configuration options.

AWS

It is recommended to add the following Disk configuration in AWS.

• IOPS: Equal to the Disk Size (so if Disk is 100GB, then IOPS is 100)
• Throughput: 125MiB/s

Supported infrastructures

• Operating System: Linux-based
• Architecture: AMD64

Firecracker requirements

To run Executors with Firecracker enabled requires the machine to support Kernel-based Virtual Machine. See deploying Executors binary for additional information on configuring Linux Machines.

Cloud providers

Machines on Cloud Providers have additional constraints for use with firecracker.

• AWS: machine type must be a metal instance (.metal)
• GCP: the instance must enable nested virtualization

Executor authentication

Executors must be run separately from your Sourcegraph instance.

Since they must still be able to reach the Sourcegraph instance in order to dequeue and perform work, requests between the Sourcegraph instance and the executors are authenticated via a shared secret.

Before starting any executors, generate an arbitrary secret string (with at least 20 characters) and set it as the executors.accessToken key in your Sourcegraph instance's site-config. The EXECUTOR_FRONTEND_PASSWORD environment variable on the executor instance will need to be configured with the same secret string.

Executor installation

Once the shared secret is set in Sourcegraph, you can start setting up executors that can use that access token to talk to the Sourcegraph instance.

Supported installation types

Validating executor deployments

If executor instances boot correctly and can authenticate with the Sourcegraph frontend, they will show up in the Executors page under Site admin > Maintenance.

Using private registries

If you want to use docker images stored in a private registry that requires authentication, follow this section to configure it.

Depending on the executor runtime that is being used, different options exist for provisioning access to private container registries:

• Through a special secret called DOCKER_AUTH_CONFIG, set in executor secrets in Sourcegraph.
• Through the EXECUTOR_DOCKER_AUTH_CONFIG environment variable (also available as a variable in the terraform modules for executors).
• Through the config.json file in ~/.docker. If using executors with firecracker enabled (recommended) this option is not available.

When multiple of the above options are combined, executors will use them in the following order:

• If a DOCKER_AUTH_CONFIG executor secret is configured, that will be preferred. That is so that users can overwrite the credentials being used in their user-settings. This is the only option available in Sourcegraph Cloud.
• If the EXECUTOR_DOCKER_AUTH_CONFIG environment variable is set, this will be used as the next option.
• Finally, if neither of the above are set, executors will fall back to the config.json file in the user home directory of the user that is owning the executor process. NOTE: This is not available in the firecracker runtime, as the rootfs is not shared with the host.

The docker CLI supports three ways to use credentials:

• Using static credentials
• Using credential helpers
• Using credential stores

Credential helpers and credential stores are only available for use with the config.json configuration option, as they require additional infrastructural changes. Thus, those options are not available on Sourcegraph Cloud.

Using static credentials

The EXECUTOR_DOCKER_AUTH_CONFIG environment variable and the DOCKER_AUTH_CONFIG secret expect a docker config with only the necessary properties set for configuring authentication. The format of this config supports multiple registries to be configured and looks like this:

JSON
{
  "auths": {
    "myregistry.example.com[:port]": {
      "auth": "base64(username:password)"
    },
    "myregistry2.example.com[:port]": {
      "auth": "base64(username:password)"
    }
  }
}

You can either create this config yourself by hand, or let docker do it for you by running:

BASH
TMP_FILE="$(mktemp -d)" bash -c 'echo "<password>" | docker --config "${TMP_FILE}" login --username "<username>" --password-stdin "<registryurl>" && cat "${TMP_FILE}/config.json" && rm -rf "${TMP_FILE}"'

NOTE: This doesn't work on Docker for Mac if "Securely store Docker logins in macOS keychain" is enabled, as it would store it in the credentials store instead.

You can also run the following:

BASH
echo -n "username:password" | base64

and then paste the result of that into a JSON string like this:

JSON
{
  "auths": {
    "myregistry.example.com[:port]": {
      "auth": "<the value from above>"
    }
  }
}

For Google Container Registry, follow this guide for how to obtain long-lived static credentials.

Configuring the auth config for use in executors

Now that the config has been obtained, it can be used for the EXECUTOR_DOCKER_AUTH_CONFIG environment variable (and terraform variable docker_auth_config) or you can create an executor secret called DOCKER_AUTH_CONFIG. Global executor secrets will be available to every execution, while user and organization level executor secrets will only be available to the namespaces executions.

Using custom certificates with executors

By default, executors will search for certificates in the following files and directories:

If your environment makes use of custom certificates, you can add them to one of these locations in order for executors to pick them up.

Adding certificates to a binary deployment

NOTE: see the troubleshooting guide for instructions on how to connect to cloud provider VMs.

After successfully deploying binaries, follow these steps:

1. Copy your certificates to /etc/ssl/certs.
2. If you are using systemd, run systemctl restart executor. If not, proceed to the next step.
3. Run executor run on the VM in order to restart the executor service.

Adding certificates with Firecracker

When running executors with the firecracker runtime, custom certificates need to be added in the container that is running within the Firecracker VM. To add custom certificates, you must create a new Docker image that contains the certificates. For example,

DOCKERFILE
FROM upstream:tag
 
# Copy the certificates into the container
COPY customcert.crt /usr/local/share/ca-certificates/customcert.crt
# Update the certificate store
RUN chmod 644 /usr/local/share/ca-certificates/customcert.crt && update-ca-certificates
# ...

Code Intel

Once the custom image is built, you can configure the executor to use it by setting the codeIntelAutoIndexing.indexerMap to use the custom image. For example,

JSON
"codeIntelAutoIndexing.indexerMap": {
  "go": "myregistry.company.com/scip-go:custom"
}

Adding certificates to a Kubernetes deployment using manifests

First, add the certificate data as a secret in your preferred namespace:

SHELL
SECRET_NAME=custom-certs
CERT_PATH=/path/to/cert.pem
kubectl create secret generic $SECRET_NAME --from-file=customcert.crt=$CERT_PATH

Or as a declarative manifest:

YAML
apiVersion: v1
kind: Secret
metadata:
  name: custom-certs
data:
  customcert.crt: $(base64 -i /path/to/cert.pem)
type: Opaque

Next, mount the secret in the executor deployment. Add the following snippet to spec.template.spec.volumes of each executor deployment:

YAML
- name: custom-certs
  secret:
    secretName: custom-certs

Also add this snippet to spec.template.spec.containers.volumeMounts of each executor deployment (specifically, the executor container, in case you inject any sidecars):

YAML
- mountPath: /etc/ssl/certs
  name: custom-certs
  readOnly: true

Next, apply the updated YAML manifests. Once the executors have rolled out, they should be picking up your custom certificates.

Adding certificates to a Kubernetes deployment using Helm

You may follow the same instructions for the manifest deployment to set custom certificates.

Adding certificates to a Docker Compose deployment

First, ensure that the certificate file is present on the host machine. Next, add the volume to the executor compose file:

YAML
- '/path/to/certs:/etc/ssl/certs'

Next, restart the deployment with docker-compose down and docker-compose up -d.