Search Jobs

Use Search Jobs to search code at scale for large-scale organizations.

Search Jobs allows you to run search queries across your organization's codebase (all repositories, branches, and revisions) at scale. It enhances the existing Sourcegraph's search capabilities, enabling you to run searches without query timeouts or incomplete results.

With Search Jobs, you can start a search, let it run in the background, and then download the results from the Search Jobs UI when it's done. Site administrators can enable or disable the Search Jobs feature, making it accessible to all users on the Sourcegraph instance.

Using Search Jobs

To use Search Jobs, you need to:

• Run a search query from your Sourcegraph instance
• Click the result menu below the search bar to see if your query is supported by Search Jobs

• If your query is valid, click Create a search job to initiate the search job
• You will be redirected to the "Search Jobs UI" page at /search-jobs, where you can view all your created search jobs. If you're a site admin, you can also view search jobs from other users on the instance
• For each search job, you can download the results by clicking the Download button. The results are formatted as JSON lines, see Search results format for more details.

Search results format

The downloaded results are formatted as JSON lines. The JSON objects have the same format as the (event-type) matches served by the Stream API.

Here is an example of a JSON lines results file for a search job that ran the query r:github.com/sourcegraph/sourcegraph handlePanic. The search returned 4 results in 2 files:

JSON
{"type":"content","path":"cmd/symbols/shared/main.go","repositoryID":399,"repository":"github.com/sourcegraph/sourcegraph","branches":["HEAD"],"commit":"9b47c6430cc3c9cb16695b4fa0a1a277cbbdb327","hunks":null,"chunkMatches":[{"content":"func handlePanic(logger log.Logger, next http.Handler) http.Handler {","contentStart":{"offset":4526,"line":129,"column":0},"ranges":[{"start":{"offset":4531,"line":129,"column":5},"end":{"offset":4542,"line":129,"column":16}}]},{"content":"\thandler = handlePanic(logger, handler)","contentStart":{"offset":3570,"line":98,"column":0},"ranges":[{"start":{"offset":3581,"line":98,"column":11},"end":{"offset":3592,"line":98,"column":22}}]}],"language":"Go"}
{"type":"content","path":"cmd/embeddings/shared/main.go","repositoryID":399,"repository":"github.com/sourcegraph/sourcegraph","branches":["HEAD"],"commit":"9b47c6430cc3c9cb16695b4fa0a1a277cbbdb327","hunks":null,"chunkMatches":[{"content":"func handlePanic(logger log.Logger, next http.Handler) http.Handler {","contentStart":{"offset":6585,"line":190,"column":0},"ranges":[{"start":{"offset":6590,"line":190,"column":5},"end":{"offset":6601,"line":190,"column":16}}]},{"content":"\thandler = handlePanic(logger, handler)","contentStart":{"offset":2844,"line":81,"column":0},"ranges":[{"start":{"offset":2855,"line":81,"column":11},"end":{"offset":2866,"line":81,"column":22}}]}],"language":"Go"}

Configure Search Jobs

Search Jobs requires an object storage to store the results of your search jobs. By default, Search Jobs stores results using our bundled blobstore service. If the blobstore service is deployed, and you want to use it to store results from Search Jobs, you don't need to configure anything.

To use a third party managed object storage service, see instructions in externalizing object storage.

Supported result types

Search jobs supports the following result types:

• file
• path
• diff
• commit

The following result types are not supported:

• symbol
• repo

Limitations

The following elements of our query language are not supported:

• file predicates, such as file:has.content, file:has.owner, file:has.contributor, file:contains.content
• catch-all .* regexp search
• Multiple rev filters
• Queries with index: filter

Disable Search Jobs

To disable Search Jobs, set DISABLE_SEARCH_JOBS=true in your frontend and worker services.

FAQ

How can I run a search job against all branches and all repositories?

You can use a combination of the repo: and rev: filter to search across all branches and repositories. For example, r: rev:*/refs/heads/* foo will search for foo across all branches and all repositories. See the search syntax documentation for more details.

Is there a time limit for a search job?

We break down a search job into smaller tasks and run them in parallel. Each task is limited to a single revision and a single repository. This allows us to run searches across large codebases. The chance of hitting timeouts is very low, but it's not impossible. If you do hit a timeout, please reach out to support and we will help you troubleshoot the issue. As an immediate measure, you can break down your search query into smaller queries and run them separately.

A search job failed, how can I find out what went wrong?

We mark a search job as failed if any of the tasks fail. You can access the log of a search job from the search jobs UI. The log contains one line per task. If a task fails, the log will contain the error message.