This page displays the release plan of all Sourcegraph versions, including deprecation schedules, links to relevant release notes, and installation links.

This page displays the docs for legacy Sourcegraph versions less than 5.1

Explore to learn more about Sourcegaph tutorials and guides.

This document explains the Sourcegraph's default contractual Service Level Agreements and Premium Support Offerings.

This page lists a detailed comparison of the features available in each plan.

Learn about some of the most commonly asked questions about Sourcegraph.

Sourcegraph offers three different pricing plans based on your needs.

Learn about the Sourcegraph's Free plan and the features included.

Learn about the Sourcegraph's Enterprise plan and the features included.

Learn about the Enterprise Starter plan tailored for individuals and teams wanting private code indexing and search to leverage the Sourcegraph platform better.

Bring the power of Sourcegraph to your code host

Short, consumable videos to help you get started with Sourcegraph.

This page will help you learn and understand about Sourcegraph and how to use it.

Learn about the different plans available for Cody.

Learn about common reasons for errors that you might run into when using Cody and how to troubleshoot them.

This quickstart guide shows how to use Cody in the VS Code editor. You'll learn about the following tasks:

To get the best results from Cody, whether you're exploring a codebase, summarizing a pull request, or generating code, clear and effective prompts are key. This guide will help you write effective prompts to maximize your experience with Cody.

Find answers to the most common questions about Cody.

In this guide, you'll learn how to use Cody to generate unit tests for your codebase.

In this guide, you'll learn how to use Cody to build a user interface for your front-end apps.

Learn how to configure Cody via `modelConfiguration` on a Sourcegraph Enterprise instance.

This section includes examples about how to configure Cody to use Sourcegraph-provided models with `modelConfiguration`. These examples will use the following:

Along with the core features, Cody Enterprise offers additional features to enhance your coding experience.

Learn how to configure Cody via `completions` on a Sourcegraph Enterprise instance.

Learn about Cody's token limits and how to manage them.

Learn how Cody makes use to Keyword Search to gather context.

Learn about all the core concepts and fundamentals that helps Cody provide codebase-aware answers.

[Cody Enterprise](/cody/clients/enable-cody-enterprise) can be deployed via the Sourcegraph Cloud or on your self-hosted infrastructure. This page describes the architecture diagrams for Cody deployed in different Sourcegraph environments.

Understand how context helps Cody write more accurate code.

Learn how Cody Gateway powers the default Sourcegraph provider for completions, enabling Cody features for Sourcegraph Enterprise customers.

Understand what is Code Graph and how Cody use it to gather context.

Learn how to use Cody and its features with the VS Code editor.

Learn how to use Cody and its features with the Visual Studio editor.

Learn how to use Cody and its features with the Neovim editor.

Learn how to use Cody and its features with JetBrains editors.

Learn how to use Cody and its features with the Eclipse editor.

Learn how to install the cody command-line tool and using the cody chat subcommand.

There are multiple ways to use Cody: you can install its extension in your favorite IDEs, access it via the Sourcegraph web app, or use it through the Cody CLI.

This document compares features and capabilities across different clients and platforms like VS Code, JetBrains, and Sourcegraph.com (Web UI).

Cody enhances your coding experience by providing intelligent code suggestions, context-aware completions, and advanced code analysis. These docs will help you use Cody on your Sourcegraph Enterprise instance.

Learn how to use Cody in the web interface with your Sourcegraph.com instance.

This page lists all the query types that will return search results with the Sourcegraph chat.

This documentation helps you set up HTTP, HTTPS, and SOCKS proxies in VS Code and JetBrains IDEs. It also includes instructions for handling self-signed certificates on macOS and Windows.

Learn how prompts can automate and accelerate your workflow with Cody.

Use additional context sources from outside of your codebase by leveraging OpenCtx providers.

Cody offers a rich set of capabilities and features that help you write better code faster. Learn and understand more about Cody's features and core AI functionality in this section.

You can control and manage what context from your codebase is used by Cody. You can do this by using Cody Context Filters.

Learn how Cody helps you identify errors in your code and provides code fixes.

Chat with the AI assistant in your code editor or via the Sourcegraph web app to get intelligent suggestions, code autocompletions, and contextually aware answers.

Learn how Cody helps you get contextually-aware autocompletion for your codebase.

Auto-edit suggests code changes by analyzing cursor movements and typing. After you've made at least one character edit in your codebase, it begins proposing contextual modifications based on your cursor position and recent changes.

Learn about agentic context fetching, a mini-agent that uses search and tools to retrieve context.

Keep on top of events in your codebase

Anything you can search, you can track and analyze

Learn how to search code across all your repositories and code hosts.

Learn and understand more about Sourcegraph's Code Search features and core functionality.

Learn about some of the most commonly asked questions about Code Search.

This page provides docs about how Search Snippets work with Sourcegraph.

This page provides docs about using Search Subexpressions in Sourcegraph's Code Search.

This page provides docs about Search Contexts in Sourcegraph's Code Search.

This page provides docs about how Saved Searches work with Sourcegraph.

Learn and understand about Sourcegraph's Structural Search and core functionality.

Learn and understand about Sourcegraph's Fuzzy Search and core functionality.

Learn more about Sourcegraph's agentic Code Search tool Deep Search.

This page provides a visual breakdown of the Search Query Language with some examples to get you started.

This page describes the query syntax for Code Search.

Learn and walkthrough some of the handy Code Search examples.

This page describes the process of writing an indexer and details all the recommended indexers that Sourcegraph currently supports.

This guide gives specific instructions for troubleshooting code navigation in your Sourcegraph instance.

Learn and understand about Search-based Code Navigation.

Learn and understand about Precise Code Navigation.

Learn how to navigate your code and understand its dependencies with high precision.

Learn and understand more about Sourcegraph's Code Navigation features and core functionality.

These docs list supported environment variables for Code Navigation.

Learn and understand how auto-indexing works.

login password ``` Under the hood, this information will be used to write the [.netrc](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html) file that is respected by Git and Go. ### TypeScript/JavaScript For **NPM**, you can create a [secret named `NPM_TOKEN`](https://docs.npmjs.com/using-private-packages-in-a-ci-cd-workflow#set-the-token-as-an-environment-variable-on-the-cicd-server) which will be automatically picked up by the indexer. ## Language support Auto-indexing is currently available for Go, TypeScript, JavaScript, Python, Ruby and JVM repositories. See also [dependency navigation](/code-search/code-navigation/features#dependency-navigation) for instructions on how to setup cross-dependency navigation depending on what language ecosystem you use. ## Lifecycle of an indexing job Index jobs are run asynchronously from a queue. Each index job has an attached **state** that can change over time as work associated with that job is performed. The following diagram shows transition paths from one possible state of an index job to another.  The general happy-path for an index job is: `QUEUED_FOR_INDEXING`, `INDEXING`, then `INDEXING_COMPLETED`. Once uploaded, the remaining lifecycle is described in [lifecycle of an upload](/code-search/code-navigation/explanations/uploads#lifecycle-of-an-upload). Index jobs may fail to complete due to the job configuration not aligning with the repository contents or due to transient errors related to the network (for example). An index job will enter the `INDEXING_ERRORED` state on such conditions. Errored index jobs may be retried a number of times before moving into a permanently errored state. At any point, an index job record may be deleted (usually due to explicit deletion by the user). ## Lifecycle of an indexing job (via UI) Users can see precise code navigation index jobs for a particular, repository by navigating to the code graph page in the target repository's index page.  Administrators of a Sourcegraph instance can see a global view of code graph index jobs across all repositories from the **Site Admin** page.  The detail page of an index job will show its current state as well as detailed logs about its execution up to the current point in time.  The stdout and stderr of each command run during pre-indexing and indexing steps are viewable as the index job is processed. This information is valuable when troubleshooting a [custom index configuration](/code-search/code-navigation/auto_indexing_configuration) for your repository.  Once the index job completes, a code graph data file has been uploaded to the Sourcegraph instance.  ## Inference of auto-indexing jobs This feature is in beta for self-hosted customers. When a commit of a repository is selected as a candidate for [auto-indexing](/code-search/code-navigation/auto_indexing) but does not have an explicitly supplied index job configuration, index jobs are inferred from the content of the repository at that commit. The site-config setting `codeIntelAutoIndexing.indexerMap` can be used to update the indexer image that is (globally) used on inferred jobs. For example, `"codeIntelAutoIndexing.indexerMap": {"go": "lsif-go:alternative-tag"}` will cause inferred jobs indexing Go code to use the specified container (with an alternative tag). This can also be useful for specifying alternative Docker registries. This document describes the heuristics used to determine the set of index jobs to schedule. See [configuration reference](/code-search/code-navigation/auto_indexing_configuration) for additional documentation on how index jobs are configured. As a general rule of thumb, an indexer can be invoked successfully if the source code to index can be compiled successfully. The heuristics below attempt to cover the common cases of dependency resolution, but may not be sufficient if the target code requires additional steps such as code generation, header file linking, or installation of system dependencies to compile from a fresh clone of the repository. For such cases, we recommend using the inferred job as a starting point to [explicitly supply index job configuration](/code-search/code-navigation/auto_indexing#explicit-index-job-configuration). ### Go For each directory containing a `go.mod` file, the following index job is scheduled. ```json { "indexing_jobs": [ { "steps": [ { "root": "", "image": "sourcegraph/lsif-go", "commands": [ "go mod download" ] } ], "root": "", "indexer": "sourcegraph/lsif-go", "indexer_args": [ "lsif-go", "--no-animation" ] } ] } ``` For every other directory excluding `vendor/` directories and their children containing one or more `*.go` files, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/lsif-go", "indexer_args": [ "GO111MODULE=off", "lsif-go", "--no-animation" ] } ``` ### TypeScript For each directory excluding `node_modules/` directories and their children containing a `tsconfig.json` file, the following index job is scheduled. Note that there are a dynamic number of pre-indexing steps used to resolve dependencies: for each ancestor directory `ancestor(dir)` containing a `package.json` file, the dependencies are installed via either `yarn` or `npm`. These steps run in order, depth-first. ```json { "steps": [ { "root": "", "image": "sourcegraph/scip-typescript:autoindex", "commands": [ "yarn" ] }, { "root": "", "image": "sourcegraph/scip-typescript:autoindex", "commands": [ "npm install" ] }, "..." ], "local_steps": [ "N_NODE_MIRROR=https://unofficial-builds.nodejs.org/download/release n --arch x64-musl autol" ], "root": "", "indexer": "sourcegraph/scip-typescript:autoindex", "indexer_args": [ "scip-typescript", "index" ] } ``` ### Rust If the repository contains a `Cargo.toml` file, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/lsif-rust", "indexer_args": [ "lsif-rust", "index" ], "outfile": "dump.lsif" } ``` ### Java Inference for languages supported by [scip-java](https://github.com/sourcegraph/scip-java) is currently restricted to Sourcegraph.com. If the repository contains both a `lsif-java.json` file as well as `*.java`, `*.scala`, or `*.kt` files, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/scip-java", "indexer_args": [ "scip-java", "index", "--build-tool=lsif" ], "outfile": "index.scip" } ``` ## More resources # Code intelligence policies resource usage best practices This guide gives an overview of best practices when defining [auto-index scheduling](/code-search/code-navigation/auto_indexing#configure-auto-indexing) and data retention policies as it relates to resource usage (particular the disk requirement for Postgres). **Auto-index scheduling policies** define the _cadence_ at which particular repositories will be subject to have indexing jobs scheduled (depending on the repository's configuration and contents). These policies define when a commit of a repository should be marked as an auto-index job candidate. **Data retention policies** define the set of indexes which are _interesting_ to a particular Sourcegraph instance's users. This is generally not *all* commits (or even all repositories), and can change drastically from company-to-company. SCIP indexes for the head of a repository's default branch are useful to explore what is most likely the current view of the development branch. These are safe from expiration by default and do not require an explicit policy to be defined by the user. SCIP indexes for historic commits can be useful to retain in addition to the current development target (retained implicitly). SCIP indexes associated with git tags matching `v*` (for example) may be useful to index and retain indefinitely (or for some relevant length of time). These indexes are _more_ likely to be used by other projects, running in production, or used externally than any arbitrary commit of the repository. Branches with a particular format may also have significance indicating its indexes should not be garbage collected. Policies should be created in such a manner that the commits most useful to _your_ engineers are covered. This may mean defining both an auto-indexing and data retention policy, or ensuring that SCIP indexes uploaded externally (via build system) are covered by a data retention policy. However, _usefulness_ vs _cost_ can be a delicate balancing act. If all SCIP indexes uploaded to an instance were retained forever, then disk requirements for Postgres would grow continuously. This is not sustainable for any engineering team with a non-zero commit velocity. If your engineering team has zero velocity please contact Sourcegraph support for guidance. Both of these types of policies (as well as any SCIP uploads integrated into a build system) influence the usage of disk in the `codeintel-db` schema. There are two large knobs that can be turned to reduce usage: 1. Reduce index coverage (exclude repositories or targets or branches) or index less frequently 2. Retain fewer indexes as they age (e.g., do not keep indexes on the non-head of branches) The following factors also play a contributing role to the scale of data on a particular instance: - How *large* your repositories (or indexing targets) are - How *frequently* your repositories (or indexing targets) receive new commits - How *widely* your data retention policies retain individual SCIP indexes (one branch vs all branches) - How *deeply* your data retention policies retain individual SCIP indexes (the latest commit vs all commits on a branch) - How many repositories (or indexing targets) have received SCIP uploads (are there deprecated repos being indexed that will never be relevant to a user?) ## Sourcegraph.com's configuration As an example, we detail relevant configuration for [Sourcegraph.com](https://sourcegraph.com). - We support auto-indexing on repositories matching `github.com/*` and `gitlab.com/*`, but only index the `HEAD` of these repositories as there are a massive number and indexing branches or tags of both code hosts would be infeasible. Matching indexes are retained indefinitely, but there is only one set of indexes covered by this policy per repository at any given time. This policy doesn't keep unnecessary data longer than necessary. - We support indexing of all tags of `jdk` with infinite retention to support precise navigation into Java ecosystem internals when individual repositories have precise intelligence. - Experimentally, we're indexing all tags of repositories matching `maven/*` to get quantitative measurements on the number of open-source Java projects we're able to infer build/indexing requirements from given only the repository contents. # Index other languages

This guide is meant to provide instructions for how to enable precise code navigation for any programming language.

The general steps for enabling precise code navigation are as follows: ## Install Sourcegraph CLI The [Sourcegraph CLI](/cli/) is used for uploading SCIP data to your Sourcegraph instance (replace `linux` with `darwin` for macOS): ```sh curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src ``` ## Install SCIP indexer An SCIP indexer is a command line tool that performs code analysis of source code and outputs a file with metadata containing all the definitions, references, and hover documentation in a project in SCIP. The SCIP file is then uploaded to Sourcegraph to power code navigation features. Install the [indexer](/code-search/code-navigation/writing_an_indexer) for the required programming language of your repository by following the instructions in the indexer's README ## Generate SCIP data To generate the SCIP data for your repository run the command in the _generate SCIP data_ step found in the README of the installed indexer. ## Upload SCIP data The upload step is the same for all languages. Make sure the current working directory is a path inside your repository, then use the Sourcegraph CLI to upload the SCIP file: ### To a private Sourcegraph instance (on prem) ```sh SRC_ENDPOINT=https://sourcegraph.mycompany.com src code-intel upload -file=index.scip ``` ### To cloud based Sourcegraph.com ```sh src code-intel upload -github-token=YourGitHubToken -file=dump.lsif ``` The `src-cli` upload command will try to infer the repository and git commit by invoking git commands on your local clone. If git is not installed, is older than version 2.7.0 or you are running on code outside of a git clone, you will need to also specify the `-repo` and `-commit` flags explicitly. If you're using Sourcegraph.com or have enabled [`lsifEnforceAuth`](/admin/config/site_config#lsifEnforceAuth) you need to [supply a GitHub token](#proving-ownership-of-a-github-repository) supplied via the `-github-token` flag in the command above. Successful output will appear similar to the following example.  ## Automate code indexing Now that you have successfully enabled code navigation for your repository, you can automate source code indexing to ensure precise code navigation stays up to date with the most recent code changes in the repository. See our [continuous integration guide](/code-search/code-navigation/how-to/adding_lsif_to_workflows) to setup automation. ## Troubleshooting ### Testing Make sure you have configured your Sourcegraph instance and [enabled precise code navigation](/code-search/code-navigation/precise_code_navigation#setting-up-code-navigation-for-your-codebase). Once LSIF data has uploaded, open the Sourcegraph UI or your code host (i.e. GitHub) and navigate to any code file that was part of the repository that was analyzed by the LSIF indexer. Hover over a symbol, variable or function name in the file, you should now see rich LSIF metadata as the source for hover-tooltips, definitions, and references. ### Error Logs To view code graph indexer processing failures go to **Repository settings > Code graph data > Activity for this repository** in your Sourcegraph instance. Failures can occur if the LSIF data is invalid (e.g., malformed indexer output), or problems were encountered during processing (e.g., system-level bug, flaky connections, etc). If you are unable to resolve the issue, please [contact support](https://help.sourcegraph.com/hc/en-us/requests/new). ### Common Errors Possible errors that can happen during upload include: - Clone in progress: the instance doesn't have the necessary data to process your upload yet, retry in a few minutes - Unknown repository (404): check your `-endpoint` and make sure you can view the repository on your Sourcegraph instance - Invalid commit (404): try visiting the repository at that commit on your Sourcegraph instance to trigger an update - Invalid auth when using Sourcegraph.com or when [`lsifEnforceAuth`](/admin/config/site_config#lsifEnforceAuth) is `true` (401 for an invalid token or 404 if the repository cannot be found on GitHub.com): make sure your GitHub token is valid and that the repository is correct - Unexpected errors (500s): reach out to your Sourcegraph support team for assistance # Index a TypeScript or JavaScript repository

This page describes how to create an index for JavaScript and TypeScript projects and uploading it to Sourcegraph.

We will use [`scip-typescript`](https://github.com/sourcegraph/scip-typescript) to create the index and the [Sourcegraph CLI](https://github.com/sourcegraph/src-cli) to upload it to Sourcegraph. ## Indexing in CI using scip-typescript directly In this approach, you will directly install `scip-typescript` and `src-cli` in CI. This is particularly useful if you are already using some other Docker image for your build. Here is an example using GitHub Actions to create and upload an index for a TypeScript project. ```yaml jobs: create-index-and-upload: # prevent forks of this repo from uploading lsif indexes if: github.repository == '' runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Install dependencies run: npm install - name: Install scip-typescript run: npm install -g @sourcegraph/scip-typescript - name: Generate index uses: scip-typescript index - name: Install src-cli run: | curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src - name: Upload index run: src code-intel upload -github-token='${{ secrets.GITHUB_TOKEN }}' -no-progress env: SRC_ENDPOINT: https://sourcegraph.com/ ``` `src-cli` ignores index upload failures by default to avoid disrupting CI pipelines with non-critical errors. On CI providers other than GitHub Actions, you may need to explicitly install [Node.js](https://nodejs.org/) as a first step. See the [`scip-typescript` README](https://github.com/sourcegraph/scip-typescript) for the list of supported Node.js versions. Examples: - [lodash/lodash](https://github.com/sourcegraph-codeintel-showcase/lodash/blob/master/.github/workflows/lsif.yml) (JavaScript) ### Optional scip-typescript flags The exact `scip-typescript` invocation will vary based on your configuration. For example: - If you are indexing a JavaScript project instead of TypeScript, add the `--infer-tsconfig` flag. ```sh scip-typescript index --infer-tsconfig ``` - If you are indexing a project using Yarn workspaces, add the `--yarn-workspaces` flag. ```sh scip-typescript index --yarn-workspaces ``` - If you are indexing a project using Pnpm workspaces, add the `--pnpm-workspaces` flag. ```sh scip-typescript index --pnpm-workspaces ``` ## Indexing in CI using the scip-typescript Docker image Sourcegraph provides a Docker image for `sourcegraph/scip-typescript`, which bundles `src-cli` for convenience. Here is an example using the `scip-typescript` Docker image with GitHub Actions to index a TypeScript project. ```yaml jobs: create-and-upload-index: # prevent forks of this repo from uploading lsif indexes if: github.repository == '' runs-on: ubuntu-latest container: sourcegraph/scip-typescript:latest steps: - uses: actions/checkout@v3 - name: Install dependencies run: npm install - name: Generate index run: scip-typescript index - name: Upload index run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -no-progress env: SRC_ENDPOINT: https://sourcegraph.com/ ``` If you are indexing a JavaScript codebase or a project using Yarn workspaces, tweak the `scip-typescript` invocation as documented in the [Optional scip-typescript flags](#optional-scip-typescript-flags) section. ## One-off indexing using scip-typescript locally Creating one-off indexes and uploading them is valuable as a proof of concept, but indexes are not kept up to date. The steps here are similar to those in the previous GitHub Actions example. 1. Install `scip-typescript`. ```sh npm install -g @sourcegraph/scip-typescript ``` 2. Install the Sourcegraph CLI. ```sh curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src ``` The exact invocation may change depending on the OS and architecture. See the [`src-cli` README](https://github.com/sourcegraph/src-cli#installation) for details. 3. `cd` into your project's root (which contains `package.json`/`tsconfig.json`) and run the following: ```sh npm install # Enable (1) type-checking code used from external packages and (2) cross-repo navigation by installing dependencies first with npm or yarn scip-typescript index # for TypeScript projects ``` If you are indexing a JavaScript codebase or a project using Yarn workspaces, tweak the `scip-typescript` invocation as documented in the [Optional scip-typescript flags](#optional-scip-typescript-flags) section. 4. Upload the data to a Sourcegraph instance. ```sh SRC_ENDPOINT= src code-intel upload # for private instances src code-intel upload -github-token= # for public instances ``` The upload command will provide a URL you can visit to see the upload status. Once the upload has finished processing, you can visit the repo and enjoy precise code navigation! # Go SCIP indexing

This page describes how you can automate data indexing in SCIP for Go codebases or index data manually.

## Automated indexing Sourcegraph provides the Docker images `sourcegraph/scip-go` and `sourcegraph/src-cli` so that you can easily automate indexing in your favorite CI framework. Note that the `scip-go` image bundles `src-cli`, so the second image may not be necessary. The following examples show you how to set up automated indexing in a few popular frameworks. You'll need to substitute the indexer and upload commands with what works for your project locally. If you implement automated indexing in a different framework, feel free to edit this page with instructions! ### GitHub Actions ```yaml on: - push jobs: scip-go: # this line will prevent forks of this repo from uploading lsif indexes if: github.repository == '' runs-on: ubuntu-latest container: sourcegraph/scip-go:latest steps: - uses: actions/checkout@v1 - name: Generate SCIP data run: scip-go - name: Install src CLI run: | curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src - name: Upload SCIP data # this will upload to Sourcegraph.com, you may need to substitute a different command. # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure ``` The following projects have example GitHub Actions workflows to generate and upload LSIF indexes: - [golang/go](https://github.com/sourcegraph-codeintel-showcase/go/blob/f40606b1241b0ca4802d7b00a763241b03404eea/.github/workflows/lsif.yml) - [kubernetes/kubernetes](https://github.com/sourcegraph-codeintel-showcase/kubernetes/blob/master/.github/workflows/lsif.yml) - [moby/moby](https://github.com/sourcegraph-codeintel-showcase/moby/blob/master/.github/workflows/lsif.yml) ### CircleCI ```yaml version: 2.1 jobs: scip-go: docker: - image: sourcegraph/scip-go:latest steps: - checkout - run: scip-go - run: | curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src; # this will upload to Sourcegraph.com, you may need to substitute a different command. # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. - run: src code-intel upload -github-token=<> -ignore-upload-failure workflows: scip-typescript: jobs: - scip-typescript ``` The following projects have example CircleCI configurations to generate and upload SCIP indexes. - [grafana](https://github.com/sourcegraph-codeintel-showcase/grafana/blob/master/.circleci/config.yml) - [helm](https://github.com/sourcegraph-codeintel-showcase/helm/blob/62c38f152d0802719aad1ec4c1c281f01dc75173/.circleci/config.yml) - [prometheus/prometheus](https://github.com/sourcegraph-codeintel-showcase/prometheus/blob/a0a8a249fff9d1c6ce4c097ccc4f5e120c723c51/.circleci/config.yml) ### Travis CI ```yaml services: - docker jobs: include: - stage: scip-go # this will upload to Sourcegraph.com, you may need to substitute a different command. # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. script: - | docker run --rm -v $(pwd):/src -w /src sourcegraph/scip-go:latest /bin/sh -c \ "scip-go; curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src; chmod +x /usr/local/bin/src; src code-intel upload -github-token=$GITHUB_TOKEN -ignore-upload-failure" ``` The following projects have example Travis CI configurations to generate and upload SCIP indexes. - [aws/aws-sdk-go](https://github.com/sourcegraph-codeintel-showcase/aws-sdk-go/blob/92f67a061fcdd46d6a418b28838b10b6ac63a880/.travis.yml) - [etcd-io/etcd](https://github.com/sourcegraph-codeintel-showcase/etcd/blob/eae726706fe8ebf7e08b45ba29a70388595db31b/.travis.yml) - [hugo/hugo](https://github.com/sourcegraph-codeintel-showcase/hugo/blob/6704b7c125d7b21ccf2048d7bff0f1ffe2b0867d/.travis.yml) ## Manual indexing 1. Install the [Go SCIP indexer](https://github.com/sourcegraph/scip-go). 1. Install the [Sourcegraph CLI](https://github.com/sourcegraph/src-cli) with the following command: ```sh curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src ``` - **macOS**: Replace `linux` with `darwin` in the URL and choose the appropriate architecture: M1/M2 chips - `arm64`, Intel chips - `amd64`. - **Windows**: Visit [the CLI repo](https://github.com/sourcegraph/src-cli) for further instructions. 1. `cd` into your Go project's root (where the `go.mod` file lives, if you have one) and run: ```sh scip-go # generates a file named index.scip ``` 1. Upload the data to a Sourcegraph instance with: ``` sh src -endpoint= code-intel upload # for private instances src code-intel upload -github-token= # for public instances ``` The upload command will provide a URL you can visit to see the upload status. When the upload is complete, you can visit the repo and check out the difference in code navigation quality! # How-to guides - [Adding precise code navigation to CI/CD workflows](/code-search/code-navigation/how-to/adding_lsif_to_workflows) - [Combining SCIP uploads from CI/CD and auto-indexing](/code-search/code-navigation/how-to/combining_scip_uploads_from_ci_cd_and_auto_indexing) - [Go SCIP indexing](/code-search/code-navigation/how-to/index_a_go_repository) - [Index a TypeScript or JavaScript repository](/code-search/code-navigation/how-to/index_a_typescript_and_javascript_repository) - [Index other languages](/code-search/code-navigation/how-to/index_other_languages) - [Code intelligence policies resource usage best practices](/code-search/code-navigation/how-to/policies_resource_usage_best_practices) # Combining SCIP uploads from CI/CD and auto-indexing Sourcegraph Enterprise instances can serve many profiles of repository size and build complexity, and therefore provides multiple methods precise SCIP index data for your team's repositories. We currently support: 1. Uploading [SCIP data yourself](/code-search/code-navigation/how-to/index_other_languages#upload-scip-data) directly from an already-configured build or continuous integration server, as well as 2. [Auto-indexing](/code-search/code-navigation/auto_indexing), which schedules index creation within the Sourcegraph instance. There is nothing preventing users from mix-and-matching these methods (we do it ourselves), but we do have some tips for doing so successfully. First and foremost, try to that auto-indexing is disabled on repositories that will receive manually configured uploads. This can be done by ensuring that no [auto-indexing policies](/code-search/code-navigation/auto_indexing#configure-auto-indexing-policies) exist for the target repo. Even if one is not created explicitly for this repo, it may still be covered under a global policy with a matching repository pattern. If covering policies can't be changed to exclude a repo (or if you want _partially_ auto-indexed coverage), then the repository can be [explicitly configured](/code-search/code-navigation/auto_indexing#explicit-index-job-configuration) with a set of auto-indexing jobs to schedule. Simply delete the job configuration that conflicts the explicitly configured project (the directory and indexer's target language should match). Once an explicit set of jobs are configured, auto-inference will not update it. If the repository shape changes frequently (new index targets are added or removed). Therefore, this configuration should be manually refreshed as necessary. # Adding precise code navigation to CI/CD workflows ## Language-specific guides We are working on creating language specific guides for use with different indexers, so make sure to check for the documentation for your language! If there isn't a guide for your language, this general guide will help you through the precise code navigation setup process. ## Benefits of CI integration Setting up a source code indexing job in your CI build provides you with fast code navigation that gives you more control on when source code gets indexed, and ensures accuracy of your code navigation by keeping in sync with changes in your repository. Due to the large number of CI frameworks and languages we may not have specific documentation for your use case. Feel free to [contact us](https://help.sourcegraph.com/hc/en-us/requests/new) if you're having difficulties and we can help troubleshoot your setup. ## Using indexer containers We're working on creating containers that bundle indexers and the [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli). All the languages in our [language-specific guides](/code-search/code-navigation/precise_code_navigation#setting-up-code-navigation-for-your-codebase) are supported, and this section provides a general guide to using those containers. We've pinned the containers to the `latest` tag in these samples so they stay up to date, but make sure you pin them before going live to ensure reproducibility. ### Basic usage Some indexers will work out of the box by just running them in your project root. Here's an example using GitHub actions and Go: ```yaml jobs: scip-go: #this line will prevent forks of this repo from uploading lsif indexes if: github.repository == '' runs-on: ubuntu-latest container: sourcegraph/scip-go:latest steps: - uses: actions/checkout@v1 - name: Generate index run: scip-go - name: Install src CLI run: | curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src - name: Upload index # this will upload to Sourcegraph.com, you may need to substitute a different command # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure ``` ### Sub-projects If your repository contains multiple code projects in different folders, your CI framework likely provides a "working directory" option so that you can do something like: ```yaml jobs: scip-go: # Prevent forks of this repo from uploading SCIP indexes if: github.repository == '' runs-on: ubuntu-latest container: sourcegraph/scip-go:latest steps: - uses: actions/checkout@v1 - name: Generate index working-directory: backend/ run: scip-go - name: Install src CLI working-directory: backend/ run: | curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src chmod +x /usr/local/bin/src - name: Upload index # note that the upload command also needs to happen in the same directory! working-directory: backend/ # this will upload to Sourcegraph.com, you may need to substitute a different command # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure ``` ### Custom build environments Depending on which language you're using, you may need to perform some pre-indexing steps in a custom build environment. You have a couple options for this: - Add the indexer and [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli) to your environment. To make this easier, you could either directly build a Docker image from our indexer container, or use its Dockerfile as inspiration for getting the tools installed in your environment. - Do the pre-indexing steps in your environment, and then make those artifacts available to our container. This second step is easy in GitHub actions because our container can be used as an action. Here's an example for a TypeScript project: ```yaml jobs: scip-typescript: # this line will prevent forks of this repo from uploading scip indexes if: github.repository == '' runs-on: ubuntu-latest container: my-awesome-container steps: - uses: actions/checkout@v1 - name: Install dependencies run: - name: Generate index uses: docker://sourcegraph/scip-typescript:latest with: args: scip-typescript index - name: Upload index uses: docker://sourcegraph/src-cli:latest with: # this will upload to Sourcegraph.com, you may need to substitute a different command # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. args: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure ``` ### GitHub Action Examples The following projects have example GitHub Action workflows to generate and upload indexes. - [elastic/kibana](https://github.com/sourcegraph-codeintel-showcase/kibana/blob/7ed559df0e2036487ae6d606e9ffa29d90d49e38/.github/workflows/lsif.yml) - [golang/go](https://github.com/sourcegraph-codeintel-showcase/go/blob/master/.github/workflows/scip.yml) - [kubernetes/kubernetes](https://github.com/sourcegraph-codeintel-showcase/kubernetes/blob/359b6469d85cc7cd4f6634e50651633eefeaea4e/.github/workflows/lsif.yml) - [lodash/lodash](https://github.com/sourcegraph-codeintel-showcase/lodash/blob/b90ea221bd1b1e036f2dfcd199a2327883f9451f/.github/workflows/lsif.yml) - [moby/moby](https://github.com/sourcegraph-codeintel-showcase/moby/blob/380429abb05846de773d5aa07de052f40c9e8208/.github/workflows/lsif.yml) - [ReactiveX/IxJS](https://github.com/sourcegraph-codeintel-showcase/IxJS/blob/e53d323314043afb016b6deceaeb068d8d23c303/.github/workflows/lsif.yml) ### Circle CI Examples Certain frameworks like Circle CI may require you to explicitly cache artifacts between jobs. In CircleCI this might look like the following: ```yaml version: 2.1 jobs: install-deps: docker: - image: my-awesome-container steps: - checkout - - save_cache: paths: - node_modules key: dependencies jobs: scip-typescript: docker: - image: sourcegraph/scip-typescript:latest steps: - checkout - restore_cache: keys: - dependencies - run: scip-typescript index # this will upload to Sourcegraph.com, you may need to substitute a different command # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors. - run: src code-intel upload -github-token=<> -ignore-upload-failure workflows: scip-typescript: jobs: - install-deps - scip-typescript: requires: - install-deps ``` The following projects have example CircleCI configurations to generate and upload indexes. - [angular/angular](https://github.com/sourcegraph-codeintel-showcase/angular/blob/f06eec98cadab2ff7a1cef2a03ba7c42015eb399/.circleci/config.yml) - [facebook/jest](https://github.com/sourcegraph-codeintel-showcase/jest/blob/b781fa2b6683f04324edbc4b41552a94f97cd479/.circleci/config.yml) - [facebook/react](https://github.com/sourcegraph-codeintel-showcase/react/blob/e488420f686b88803cfb1bb09bbc4d3991db8c55/.circleci/config.yml) - [grafana](https://github.com/sourcegraph-codeintel-showcase/grafana/blob/664a694955ea40575a1cffe9db47a7adf4d3c2bb/.circleci/config.yml) - [helm](https://github.com/sourcegraph-codeintel-showcase/helm/blob/62c38f152d0802719aad1ec4c1c281f01dc75173/.circleci/config.yml) - [prometheus/prometheus](https://github.com/sourcegraph-codeintel-showcase/prometheus/blob/a0a8a249fff9d1c6ce4c097ccc4f5e120c723c51/.circleci/config.yml) - [ReactiveX/rxjs](https://github.com/sourcegraph-codeintel-showcase/rxjs/blob/c9d3c1a76a68273863fc59075a71b4cc43c06114/.circleci/config.yml) ### Travis CI Examples The following projects have example Travis CI configurations to generate and upload indexes. - [aws/aws-sdk-go](https://github.com/sourcegraph-codeintel-showcase/aws-sdk-go/blob/92f67a061fcdd46d6a418b28838b10b6ac63a880/.travis.yml) - [etcd-io/etcd](https://github.com/sourcegraph-codeintel-showcase/etcd/blob/eae726706fe8ebf7e08b45ba29a70388595db31b/.travis.yml) - [expressjs/express](https://github.com/sourcegraph-codeintel-showcase/express/blob/bd1ae153f19656183257ed223d518aeb9f5091ec/.travis.yml) - [hugo/hugo](https://github.com/sourcegraph-codeintel-showcase/hugo/blob/6704b7c125d7b21ccf2048d7bff0f1ffe2b0867d/.travis.yml) - [Microsoft/TypeScript](https://github.com/sourcegraph-codeintel-showcase/TypeScript/blob/f37f1dee1b3e63b12df2935590c8707a5ec3993b/.travis.yml) - [moment/moment](https://github.com/sourcegraph-codeintel-showcase/moment/blob/eedccdc2c07fb5abe931b427d50f5b3c3f44ac95/.travis.yml) - [sindresorhus/got](https://github.com/sourcegraph-codeintel-showcase/got/blob/164d55a029512cea7f245de870cbb1eaba114734/.travis.yml) ## CI from scratch If you're indexing a language we haven't documented yet in our [language-specific guides](/code-search/code-navigation/precise_code_navigation#setting-up-code-navigation-for-your-codebase), follow the instructions in this section. We want to have containers available for every language with a robust indexer, so please [contact](https://help.sourcegraph.com/hc/en-us/requests/new) to let us know we're missing one. ### Set up your CI machines Your CI machines will need two command-line tools installed. Depending on your build system setup, you can do this as part of the CI step, or you can add it directly to your CI machines for use by the build. 1. The [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli). 2. The [indexer](/code-search/code-navigation/writing_an_indexer) for your language. ### Add steps to your CI 1. **Generate an index** for a project within your repository by running the indexer in the project directory (consult your indexer's docs). 1. **[Upload that generated index](/code-search/code-navigation/precise_code_navigation#setting-up-code-navigation-for-your-codebase)** to your Sourcegraph instance. ## Recommended upload frequency We recommend that you start with a CI job that runs on every commit (including branches). If you see too much load on your CI, your Sourcegraph instance, or a rapid decrease in free disk space on your Sourcegraph instance, you can instead index only the default branch, or set up a periodic job (e.g. daily) in CI that indexes your default branch. With periodic jobs, you should still receive precise code navigation on non-indexed commits on lines that are unchanged since the nearest indexed commit. This requires that the indexed commit be a direct ancestor or descendant no more than [100 commits](https://github.com/sourcegraph/sourcegraph-public-snapshot/blob/e7803474dbac8021e93ae2af930269045aece079/lsif/src/shared/constants.ts#L25) away. If your commit frequency is too high and your index frequency is too low, you may find commits with no precise code navigation at all. In this case, we recommend you try to increase your index frequency if possible. ## Uploading indexes to Sourcegraph.com Indexes can be uploaded to a self-hosted Sourcegraph instance or to [Sourcegraph.com](https://sourcegraph.com). Using the [Sourcegraph.com](https://sourcegraph.com) endpoint will surface code navigation for your public repositories directly on GitHub via the [Sourcegraph browser extension](/integration/browser_extension) and at `https://sourcegraph.com/github.com//`. Using the [Sourcegraph.com](https://sourcegraph.com) endpoint is free and your index is treated as User-Generated Content (you own it, as covered in our [Sourcegraph.com terms of service](https://about.sourcegraph.com/terms-dotcom#3-proprietary-rights-and-licenses)). If you run into trouble, or a situation arises where you need all of your index expunged, please reach out to us at [support@sourcegraph.com](mailto:support@sourcegraph.com). # Code Graph Data uploads

How Code Graph indexers analyze code and generate and index file.

[Code graph indexers](/code-search/code-navigation/writing_an_indexer) analyze source code and generate an index file, which is subsequently [uploaded to a Sourcegraph instance](/code-search/code-navigation/how-to/index_other_languages#upload-scip-data) using [Sourcegraph CLI](/cli/) for processing. Once processed, this data becomes available for [precise code navigation queries](/code-search/code-navigation/precise_code_navigation). ## Lifecycle of an upload Uploaded index files are processed asynchronously from a queue. Each upload has an attached `state` that can change as work associated with that data is performed. The following diagram shows the possible transition paths from one `state` of an upload to another.  The typical sequence for a successful upload is: `UPLOADING_INDEX`, `QUEUED_FOR_PROCESSING`, `PROCESSING`, and `COMPLETED`. In some cases, the processing of an index file may fail due to issues such as malformed input or transient network errors. When this happens, an upload enters the `PROCESSING_ERRORED` state. Such error uploads may undergo multiple retry attempts before moving into a permanent error state. At any point, an uploaded record may be deleted. This can happen due to various reasons, such as being replaced by a newer upload, due to the age of the upload record, or by explicit deletion initiated by the user. When deleting a record that could be used for code navigation queries, it transitions first into the `DELETING` state. This temporary state allows Sourcegraph to manage the set of Code Graph uploads smoothly. Changing the state of an upload to or from `COMPLETED` requires updating the [repository commit graph](#repository-commit-graph). This process can be computationally expensive for the worker service or Postgres database. ## Lifecycle of an upload (via UI) After successfully uploading an index file, the Sourcegraph CLI will provide a URL on the target instance to track the progress of that upload. You can view the data uploads for a specific repository by navigating to the Code Graph page on the target repository's index page. Alternatively, website administrators of a Sourcegraph instance can access a global view of Code Graph data uploads across all repositories from the **Site Admin** page. ## Repository commit graph Sourcegraph maintains a mapping from a commit of a repository to the set of upload records that can resolve a query for that commit. When an upload record transitions to or from the `COMPLETED` state, the set of eligible uploads changes, and this mapping must be recalculated. Upon a state change in an upload, we flag the repository as needing an update. Subsequently, the worker service updates the commit graph and asynchronously clears the flag for that repository. When an upload changes state, the repository is flagged as requiring an update status. Then the [`worker` service](/admin/workers#codeintel-commitgraph) will update the commit graph and unset the flag for that repository asynchronously. While this flag is set, the repository's commit graph is considered `stale`. This means there may be some upload records in a `COMPLETED` state that aren't yet used to resolve code navigation queries. The state of a repository's commit graph can be seen in the code graph data page on the target repository's index page. Once the commit graph has updated (and no subsequent changes to that repository's uploads have occurred), the repository commit graph is no longer considered `stale`. # Inference of auto-indexing jobs This feature is in beta for self-hosted customers. When a commit of a repository is selected as a candidate for [auto-indexing](/code-search/code-navigation/auto_indexing) but does not have an explicitly supplied index job configuration, index jobs are inferred from the content of the repository at that commit. The site-config setting `codeIntelAutoIndexing.indexerMap` can be used to update the indexer image that is (globally) used on inferred jobs. For example, `"codeIntelAutoIndexing.indexerMap": {"go": "scip-go:alternative-tag"}` will cause inferred jobs indexing Go code to use the specified container (with an alternative tag). This can also be useful for specifying alternative Docker registries. This document describes the heuristics used to determine the set of index jobs to schedule. See [configuration reference](/code-search/code-navigation/auto_indexing_configuration) for additional documentation on how index jobs are configured. As a general rule of thumb, an indexer can be invoked successfully if the source code to index can be compiled successfully. The heuristics below attempt to cover the common cases of dependency resolution, but may not be sufficient if the target code requires additional steps such as code generation, header file linking, or installation of system dependencies to compile from a fresh clone of the repository. For such cases, we recommend using the inferred job as a starting point to [explicitly supply index job configuration](/code-search/code-navigation/auto_indexing#explicit-index-job-configuration). ## Go For each directory containing a `go.mod` file, the following index job is scheduled. ```json { "index_jobs": [ { "steps": [ { "root": "", "image": "sourcegraph/scip-go", "commands": [ "go mod download" ] } ], "root": "", "indexer": "sourcegraph/scip-go", "indexer_args": [ "scip-go", "--verbose" ] } ] } ``` For every _other_ directory excluding `vendor/` directories and their children containing one or more `*.go` files, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/scip-go", "indexer_args": [ "GO111MODULE=off", "scip-go", "--verbose" ] } ``` ## TypeScript For each directory excluding `node_modules/` directories and their children containing a `tsconfig.json` file, the following index job is scheduled. Note that there are a dynamic number of pre-indexing steps used to resolve dependencies: for each ancestor directory `ancestor(dir)` containing a `package.json` file, the dependencies are installed via either `yarn` or `npm`. These steps run in order, depth-first. ```json { "steps": [ { "root": "", "image": "sourcegraph/scip-typescript:autoindex", "commands": [ "yarn" ] }, { "root": "", "image": "sourcegraph/scip-typescript:autoindex", "commands": [ "npm install" ] }, "..." ], "local_steps": [ "N_NODE_MIRROR=https://unofficial-builds.nodejs.org/download/release n --arch x64-musl autol" ], "root": "", "indexer": "sourcegraph/scip-typescript:autoindex", "indexer_args": [ "scip-typescript", "index" ] } ``` ## Rust If the repository contains a `Cargo.toml` file, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/lsif-rust", "indexer_args": [ "lsif-rust", "index" ], "outfile": "dump.lsif" } ``` ## Java Inference for languages supported by [scip-java](https://github.com/sourcegraph/scip-java) is currently restricted to Sourcegraph.com. If the repository contains both a `lsif-java.json` file as well as `*.java`, `*.scala`, or `*.kt` files, the following index job is scheduled. ```json { "root": "", "indexer": "sourcegraph/scip-java", "indexer_args": [ "scip-java", "index", "--build-tool=lsif" ], "outfile": "index.scip" } ``` # Private Resources on on-prem data center via Sourcegraph Connect agent This feature is in the Experimental stage. Please contact Sourcegraph directly via [preferred contact method](https://about.sourcegraph.com/contact) for more information. As part of the [Enterprise tier](https://sourcegraph.com/pricing), Sourcegraph Cloud supports connecting private resources on any on-prem private network by running Sourcegraph Connect tunnel agent in customer infrastructure. ## How it works Sourcegraph will set up a tunnel server in a customer dedicated GCP project. Customer will start the tunnel agent provided by Sourcegraph with the provided credential. After start, the agent will authenticate and establish a secure connection with Sourcegraph tunnel server. Sourcegraph Connect consists of three major components: Tunnel agent: deployed inside the customer network, which uses its own identity and encrypts traffic between the customer code host and client. Agent can only communicate with permitted customer code hosts inside the customer network. Only agents are allowed to establish secure connections with tunnel server, the server can only accept connections if agent identity is approved. Tunnel server: a centralized broker between client and agent managed by Sourcegraph. Its purpose is to set up mTLS, proxy encrypted traffic between clients and agents and enforce ACL. Tunnel client: forward proxy clients managed by sourcegraph. Every client has its own identity and it cannot establish a direct connection with the customer agent, and has to go through tunnel server. [link](https://link.excalidraw.com/readonly/453uvY8infI8wskSecGJ)