# Deploying Sourcegraph executors using Terraform on Google Cloud

A [Terraform module](https://github.com/sourcegraph/terraform-google-executors) is provided to
provision machines running executors on Google Cloud.

See also: [Deploying on AWS](/self-hosted/executors/deploy-executors-terraform-aws)

## Basic Definition

The following is the minimum required definition to deploy an executor on Google Cloud.

```terraform
module "executors" {
  source  = "sourcegraph/executors/google"

  # Find the latest version matching your Sourcegraph version here:
  # https://github.com/sourcegraph/terraform-google-executors/tags
  version = "<version>"

  region = "<region to provision in>"
  zone   = "<zone to provision resources in>"

  executor_sourcegraph_external_url            = "<external url>"
  executor_sourcegraph_executor_proxy_password = "<shared secret>"

  # Either:
  executor_queue_name                          = "<codeintel | batches>"
  # Or:
  executor_queue_names                         = ["codeintel", "batches"]

  executor_instance_tag                        = "<tag to filter in stackdriver monitoring>"
  executor_metrics_environment_label           = "<label to filter custom metrics>"
  executor_use_firecracker                     = true
}
```

| Variable                                       | Description                                                                                                                                                                                                                |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region`                                       | The Google Cloud region to provision the executor resources in.                                                                                                                                                            |
| `zone`                                         | The Google Cloud zone to provision the executor resources in.                                                                                                                                                              |
| `executor_sourcegraph_external_url`            | The public URL of your Sourcegraph instance. This corresponds to the `externalURL` value in your Sourcegraph instance's site configuration and must be resolvable from the provisioned executor compute resources.         |
| `executor_sourcegraph_executor_proxy_password` | The access token corresponding to the `executors.accessToken` in your Sourcegraph instance's site configuration.                                                                                                           |
| `executor_queue_name`                          | The single queue from which the executor should pull jobs - [`codeintel`](/code-navigation/auto-indexing) or [`batches`](/batch-changes/server-side). Either this or `executor_queue_names` must be set.                   |
| `executor_queue_names`                         | The multiple queues from which the executor should pull jobs - one or more of [`codeintel`](/code-navigation/auto-indexing) and [`batches`](/batch-changes/server-side). Either this or `executor_queue_name` must be set. |
| `executor_instance_tag`                        | A label tag to add to all the executors; can be used for filtering out the right instances in stackdriver monitoring.                                                                                                      |
| `executor_metrics_environment_label`           | The value for environment by which to filter the custom metrics.                                                                                                                                                           |
| `executor_use_firecracker`                     | Whether to use [Firecracker](/self-hosted/executors/firecracker) sandboxing for job execution. Requires nested virtualization support. Defaults to `true`.                                                                |
| `private_networking`                           | If `true`, the executors and Docker registry mirror will live in a private subnet and communicate with the internet through Cloud NAT. Defaults to `false`. See the [Private Single Executor](#private-single-executor) example. |
| `randomize_resource_names`                     | Use randomized names for resources. Defaults to `false`. Enabling this on existing deployments will **recreate** executor resources.                                                                                       |
| `private_ca_cert_path`                         | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. Optional.                                          |

See the [Google Cloud Terraform module variables](https://github.com/sourcegraph/terraform-google-executors/blob/main/modules/executors/variables.tf) for additional configurations.

## Terraform Version

The executor Terraform modules require Terraform `>= 1.1.0, < 2.0.0`.

## Permissions

Ensure the [IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/overview) is enabled.

<Accordion title="Required GCP permissions">

-   appengine.applications.get
-   clientauthconfig.brands.\*
-   clientauthconfig.clients.\*
-   cloudasset.assets.searchAllResources
-   cloudnotifications.activities.list
-   cloudtrace.insights.get
-   cloudtrace.insights.list
-   cloudtrace.stats.get
-   cloudtrace.tasks.\*
-   cloudtrace.traces.list
-   compute.addresses.\*
-   compute.autoscalers.\*
-   compute.disks.\*
-   compute.firewalls.\*
-   compute.globalOperations.get
-   compute.instanceGroupManagers.\*
-   compute.instanceGroups.create
-   compute.instances.\*
-   compute.instanceTemplates.\*
-   compute.networks.\*
-   compute.regionOperations.get
-   compute.subnetworks.\*
-   compute.zoneOperations.get
-   compute.zones.get
-   container.clusters.list
-   iam.roles.\*
-   iam.serviceAccountKeys.\*
-   logging.logEntries.list
-   logging.privateLogEntries.list
-   monitoring.timeSeries.list
-   oauthconfig.testusers.update
-   oauthconfig.verification.update
-   orgpolicy.policy.get
-   resourcemanager.projects.\*
-   secretmanager.locations.list
-   secretmanager.secrets.\*
-   secretmanager.versions.\*

</Accordion>

## Supported Regions

All regions are supported.

## Examples

### Single Executor

Provisions a single executor to pull from the `codeintel` queue.

-   [Google Cloud example](https://github.com/sourcegraph/terraform-google-executors/tree/main/examples/single-executor)

### Multiple Executors

Provisions two executors, one to pull from the `codeintel` queue and the other for the `batches` queue.

-   [Google Cloud example](https://github.com/sourcegraph/terraform-google-executors/tree/main/examples/multiple-executors)

### Private Single Executor

Provisions a single executor in a private subnet (no public IP). Cloud NAT is used for outbound internet traffic.

-   [Google Cloud example](https://github.com/sourcegraph/terraform-google-executors/tree/main/examples/private-single-executor)

## Step-by-step Guide

The following is a step-by-step guide on provisioning a single `codeintel` executor on Google Cloud.

### Provision

1. [Install Terraform](#terraform-version).
2. Install the [`gcloud CLI`](https://cloud.google.com/sdk/docs/install)
3. Run `gcloud auth application-default login`
4. Set up your Sourcegraph instance's Site configuration for executors
    1. Click on your profile picture in the top right corner
    2. Select **Site admin**
    3. Expand the **Configuration** section
    4. Select **Site configuration**
    5. Set the following,
        - `"externalURL": "<URL>"`
            - A URL that is accessible from the GCP VM (e.g. a public URL such as `https://sourcegraph.example.com`)
        - `"executors.accessToken": "<new long secret>"`
            - Can be generated by running `cat /dev/random | base64 | head -c 20`
            - The secret will be displayed as `REDACTED` once it's saved.
        - `"codeIntelAutoIndexing.enabled": true`
            - _This is only for `codeintel` executors._
5. Download
   the [example files](https://github.com/sourcegraph/terraform-google-executors/tree/main/examples/single-executor)
6. Change the following in `providers.tf`
    - `project` to the GCP project to provision the executor in
    - `region` to the GCP region to provision the executor in
    - `zone` to the GCP zone to provision the executor in
7. Change the following in `main.tf`
    - `executor_sourcegraph_external_url` to the URL configured in your instance's **Site configuration**
    - `executor_sourcegraph_executor_proxy_password` to the access token configured in your instance's **Site
      configuration**
8. Run `terraform init` to download the Sourcegraph executor modules.
9. Run `terraform plan` to preview the changes that will occur to your GCP infrastructure.
10. Run `terraform apply` and enter "yes" after reviewing the proposed changes to create the executor VM
    - Ensure `terraform apply` exited with code 0 and did not print any errors
11. Go back to the site admin page, expand **Executors**, click **Instances**, and check to see if your executor shows
    up in the list with a green dot 🟢

### Validation

The following can be done to troubleshoot or double-check that the executor has been properly provisioned.

Ensure the executor is listed in the Compute Engine. Either go to **Compute Engine** in the GCP Console for your project
or run the following command.

```shell
$ gcloud compute instances list
NAME                                          ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
sourcegraph-executor-h0rv                     us-central1-c  c2-standard-8               10.0.1.16                   RUNNING
sourcegraph-executors-docker-registry-mirror  us-central1-c  n2-standard-2               10.0.1.2                    RUNNING
```

You can ssh into to the instance to ensure the service is running. You can open an ssh connection either via the GCP
Console or by running the following command.

```shell
gcloud compute ssh sourcegraph-executor-h0rv
```

Then run the following command to check if the service is running.

```shell
you@sourcegraph-executor-h0rv:~$ systemctl status executor
🟢 executor.service - User code executor
     Loaded: loaded (/etc/systemd/system/executor.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-11-18 02:28:48 UTC; 19s ago
```

To check the logs, you can either query the **Log Explorer** in the GCP Console or by running the following command
while connected to the instance.

```shell
you@sourcegraph-executor-h0rv:~$ journalctl -u executor | less
Nov 18 02:31:01 sourcegraph-executor-h0rv executor[2465]: t=2021-11-18T02:31:01+0000 lvl=dbug msg="TRACE internal" host=... path=/.executors/queue/codeintel/dequeue code=204 duration=92.131237ms
Nov 18 02:31:01 sourcegraph-executor-h0rv executor[2465]: t=2021-11-18T02:31:01+0000 lvl=dbug msg="TRACE internal" host=... path=/.executors/queue/codeintel/canceled code=200 duration=90.630467ms
Nov 18 02:31:02 sourcegraph-executor-h0rv executor[2465]: t=2021-11-18T02:31:02+0000 lvl=dbug msg="TRACE internal" host=... path=/.executors/queue/codeintel/dequeue code=204 duration=91.269106ms
Nov 18 02:31:02 sourcegraph-executor-h0rv executor[2465]: t=2021-11-18T02:31:02+0000 lvl=dbug msg="TRACE internal" host=... path=/.executors/queue/codeintel/canceled code=200 duration=161.469685ms
```

Ensure the `EXECUTOR_FRONTEND_URL` and `EXECUTOR_FRONTEND_PASSWORD` in `/etc/systemd/system/executor.env` are correct

```
cat /etc/systemd/system/executor.env
```

Ensure the VM can hit your `externalURL`:

```shell
you@sourcegraph-executor-h0rv:~$ curl <your externalURL here>
<a href="/sign-in?returnTo=%2F">Found</a>
```

### Configure Auto-indexing

1. Go to the **Site admin** page
2. Expand **Code graph**,
3. Select **Configuration**
4. Click **Create new policy**, and fill in:
    - Name: `TEST`
    - Click _add a repository pattern_
    - Repository pattern #1: set this to an existing repository on your Sourcegraph instance (
      e.g. `github.com/gorilla/mux`)
    - Type: `HEAD`
    - Retention: Disabled
    - Auto-indexing: Enabled
5. Expand **Code graph**
6. Select **Auto-indexing**, and check to see if an indexing job has appeared. If nothing is there:
    - Try clicking **Enqueue**
    - Try setting a higher update frequency: `PRECISE_CODE_INTEL_AUTO_INDEXING_TASK_INTERVAL=10s`
    - Try setting a lower delay: `PRECISE_CODE_INTEL_AUTO_INDEXING_REPOSITORY_PROCESS_DELAY=10s`
7. Once you have a completed indexing job, click **Uploads** and check to see that an index has been uploaded.
8. Once the index has been uploaded, you should see the **`PRECISE`** badge in the hover! 🎉

## Auto-scaling

> NOTE: Auto scaling is currently not supported
> when [downloading and running executor binaries yourself](/self-hosted/executors/deploy-executors-binary),
> and on managed instances when using self-hosted executors, since it requires deployment adjustments.

Auto-scaling of executor instances can help to increase concurrency of jobs, without paying for unused resources. With auto-scaling, you can scale down to 0 instances when no workload exist and scale up as far as you like and your cloud provider can support. Auto-scaling needs to be configured separately.

Auto-scaling makes use of **Instance Groups** on Google Cloud. Sourcegraph's `worker` service publishes a scaling metric (that is, the number of jobs in queue) to Cloud Monitoring. Then, based on that reported value, the auto-scaler adds and removes compute resources to match the required amount of compute. The autoscaler will attempt to hold 1 instance running per each `executor_jobs_per_instance_scaling` items in queue.

For example, if `executor_jobs_per_instance_scaling` is set to `20` and the queue size is currently `400`, then `20` instances would be determined as required to handle the load. You might want to tweak this number based on the `machine_type`, `maximum_num_jobs` (concurrency per machine), and desired processing speed. See the [Google Cloud variable definitions](https://github.com/sourcegraph/terraform-google-executors/blob/main/modules/executors/variables.tf) for details.

With the Terraform variables `executor_min_replicas` and `executor_max_replicas`, you can configure the minimum and maximum number of compute machines to be run at a given time.

For auto-scaling to work, two things must be true:

1. `executor_min_replicas` must be `>= 0` and `executor_max_replicas` must be `> executor_min_replicas`.
2. The Sourcegraph instance (its `worker` service, specifically) needs to publish scaling metrics to Cloud Monitoring.

For the latter to work, the Sourcegraph instance needs to be configured with the correct credentials that allow it to access Google Cloud.

The `credentials` submodule in the [Google Cloud executor module](https://github.com/sourcegraph/terraform-google-executors/tree/main/modules/credentials) exists for that purpose. When used, the `credentials` module sets up a service account with permission to write Cloud Monitoring metrics and returns the credentials in the Terraform outputs.

Here's an example of how one would configure auto-scaling.

```terraform
module "executors" {
  source  = "sourcegraph/executors/google"
  version = "<version>"

  # Basic configuration...

  # Auto-scaling
  executor_min_replicas              = 0 # Spin down when not in use
  executor_max_replicas              = 30
  executor_jobs_per_instance_scaling = 20
}

module "my-credentials" {
  source  = "sourcegraph/executors/google//modules/credentials"
  version = "<version>"

  resource_prefix = "<optional prefix to add to created resources>"
}

output "metric_writer_credentials_file" {
  value     = module.my-credentials.metric_writer_credentials_file
  sensitive = true
}
```

After running `terraform apply`, retrieve the credentials by running the following command.

```shell
$ terraform output metric_writer_credentials_file
```

### Configuring the Sourcegraph instance

The Google Compute Engine auto-scaling groups configured by the Sourcegraph Terraform module respond to changes in metric values written to Cloud Monitoring. The target Sourcegraph instance is expected to continuously write these values.

To write the scaling metric to **Cloud Monitoring**, the `worker` service must have defined the following environment variables.

| Environment Variable                | Description                                        |
| ----------------------------------- | -------------------------------------------------- |
| `EXECUTOR_METRIC_ENVIRONMENT_LABEL` | Same value as `executor_metrics_environment_label` |
| `EXECUTOR_METRIC_GCP_PROJECT_ID`    | The GCP Project ID                                 |

Then either one of the following environment variables must be set.

| Environment Variable                                          | Description                                                                                       |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `EXECUTOR_METRIC_GOOGLE_APPLICATION_CREDENTIALS_FILE_CONTENT` | The **base64-decoded** output of `metric_writer_credentials_file`                                 |
| `EXECUTOR_METRIC_GOOGLE_APPLICATION_CREDENTIALS_FILE`         | The path to the file containing the **base64-decoded** output of `metric_writer_credentials_file` |

### Testing auto scaling

Once the environment variables have been set and the worker service has been restarted, you should be able to find the scaling metrics in **Cloud Monitoring**.

To test if the metric is correctly reported: go to the **Metrics explorer**. Select **Resource type: Global** and then **Metric: `custom/executors/queue/size`**. You should see values reported here. `0` is also an indicator that it works correctly.

Next, you can test whether the number of executors rises and shrinks as load spikes occur. Keep in mind that auto-scaling is not a real-time operation and usually takes a short moment and can have some delays between the metric going down and the desired machine count adjusting.

## Upgrading executors

Upgrading executors is relatively uninvolved. Simply follow the instructions below.
Also, check the [changelog](https://sourcegraph.com/changelog) for any Executors related breaking changes or new features or flags that you might want to configure. See [Executors maintenance](/self-hosted/executors/deploy-executors#Maintaining-and-upgrading-executors) for version compatibility.

### **Step 1:** Update the source version of the terraform modules

> NOTE: Keep in mind that only one minor version bumps are guaranteed to be disruption-free.

```diff
module "executors" {
  source = "sourcegraph/executors/google"

  # Find the latest version matching your Sourcegraph version here:
  # https://github.com/sourcegraph/terraform-google-executors/tags
-  version = "7.3.0"
+  version = "7.4.0"

  region = "<region>"
  zone   = "<zone>"

  executor_sourcegraph_external_url            = "<external url>"
  executor_sourcegraph_executor_proxy_password = "<shared secret>"

  # Either:
  executor_queue_name                          = "<codeintel | batches>"
  # Or:
  executor_queue_names                         = ["codeintel", "batches"]

  executor_instance_tag                        = "<tag to filter in stackdriver monitoring>"
  executor_metrics_environment_label           = "<label to filter custom metrics>"
  executor_use_firecracker                     = true
}
```

### **Step 2:** Reapply the terraform configuration

Simply reapply the terraform configuration and executors will be ready to go again.

```bash
terraform apply
```