HTTP Tracing

HTTP traces are a powerful debugging tool to help you see how your Sourcegraph requests are processed under the hood - like having X-ray vision into how long each part takes and where errors occur.

To enable HTTP traces on your Sourcegraph Instance:

  1. Deploy and / or configure a tracing backend

  2. Configure tracing in your Site Configuration settings, to match your tracing backend

Backends

The quickest way to get started with HTTP tracing is to deploy our bundled Jaeger backend. You can also configure an external, OpenTelemetry-compatible backend of your choice.

Jaeger

To deploy our bundled Jaeger backend, follow the instructions for your deployment type:

Then configure your Site Configuration:

  1. Ensure your externalURL is configured
  2. Configure observability.tracing > urlTemplate
  3. Optionally, configure observability.client, for Sourcegraph clients to also report traces, ex. src cli
JSON
  "externalURL": "https://sourcegraph.example.com",
  "observability.tracing": {
    "urlTemplate": "{{ .ExternalURL }}/-/debug/jaeger/trace/{{ .TraceID }}"
  },
  "observability.client": {
    "openTelemetry": {
      "endpoint": "/-/debug/otlp"
    }
  },

Where:

  • {{ .ExternalURL }} is the value of the externalURL setting in your Sourcegraph instance's Site Configuration
  • {{ .TraceID }} is the TraceID which gets generated while processing the request

Once deployed, the Jaeger web UI will be accessible at /-/debug/jaeger

External OpenTelemetry-Compatible Platforms

If you prefer to use an external, OTel-compatible platform, you can configure Sourcegraph to export traces to it instead. See our OpenTelemetry documentation for further details.

Then configure your Site Configuration:

  1. Configure observability.tracing > urlTemplate
  2. Optionally, configure observability.client, for Sourcegraph clients to also report traces, ex. src cli

For example, if you export your traces to Honeycomb, your Site Configuration may look like:

JSON
  "observability.tracing": {
    "urlTemplate": "https://ui.honeycomb.io/YOUR-HONEYCOMB-ORG/environments/YOUR-HONEYCOMB-DATASET/trace?trace_id={{ .TraceID }}"
  },
  "observability.client": {
    "openTelemetry": {
      "endpoint": "/-/debug/otlp"
    }
  },

Where:

  • {{ .TraceID }} is the TraceID which gets generated while processing the request

How to use traces

We generally use the following approach when using traces to help root-cause an issue:

  1. Reproduce the problematic user request, with the trace=1 parameter in the URL
  2. Get the link to the trace in the tracing backend, from the x-trace-url response header
  3. Explore the request tree in the the tracing backend's UI, and take note of:
    1. Items near the leaves which take up a significant portion of the overall request time
    2. Spans which have errors attached to them
  4. Search your Sourcegraph instance logs for events which include the corresponding TraceId or SpanId values
  5. Include this information in your Sourcegraph support ticket, by attaching the trace JSON file, and / or screenshots

Trace a search query

To trace a search query, run a search on your Sourcegraph instance with the ?trace=1 parameter in the URL.

Depending on your Sourcegraph instance version, a link to the exported trace may appear in the UI:

link to trace

Trace a GraphQL request

To trace a GraphQL request, include a X-Sourcegraph-Should-Trace: true header when you send the request.

The response will include an x-trace-url header, which will include a URL to the exported trace.

Trace Formats

As the OTel (OpenTelemetry) HTTP trace format has gained broad industry adoption, we've centralized our support for HTTP traces on the OTel format, whether with our bundled Jaeger, or an external backend of your choice.

As Jaeger has also switched to the OTel format, we've removed support for Jaeger's deprecated format. We've also removed support for Go's net/trace format.

Basic sampling modes

Three basic sampling modes are available in the observability.tracing Site Configuration:

JSON
  "observability.tracing": {
    "urlTemplate": "{{ .ExternalURL }}/-/debug/jaeger/trace/{{ .TraceID }}",
    "sampling": "selective"
  }
  • selective
    • Default
    • Only exports a trace when the trace=1 parameter is in the request URL
  • all
    • Exports traces for all requests
    • Not recommended, as it can be memory and network intensive, while very few traces are actually needed
  • none
    • Disables tracing