How to evaluate Sourcegraph on your own codebase

Does Sourcegraph make a coding agent better on your codebase? That depends on what better means for your team: completing more tasks, finding the right code faster, reducing cost, or making difficult work more reliable. A public benchmark cannot answer that question directly.

The answer depends less on codebase size than on how the relevant work is distributed across it. An agent can navigate a very large monorepo cheaply when the code it needs sits in one obvious directory. It can struggle in a much smaller environment when understanding one behavior requires tracing several services, repositories, ownership boundaries, and historical changes. Retrieval difficulty is a property of both the codebase and the task.

We saw that distinction directly while evaluating the Sourcegraph MCP server. Across 288 tasks where both arms could reach the same code at the same revisions, structured retrieval raised file-level F1 from 0.091 to 0.240 and recall from 0.120 to 0.272. Aggregate task-completion reward was effectively unchanged. Cost improved when the baseline agent searched broadly, but worsened when it could localize the work quickly.

Those results are not contradictory. Retrieval quality, task completion, and cost measure different parts of the system. Measuring only one can produce a technically correct result that still leads to the wrong rollout decision.

The practical question is not whether structured code retrieval wins on average. It is whether it helps on the work your engineers and agents have difficulty localizing. What follows is a method for running that comparison with your own repositories, tasks, models, and cost structure.

One evaluation, three different answers

The same 288 tasks produced three different results, all under full information parity: both arms could reach the same code at the same revisions.

Sourcegraph improved retrieval, raising file-level F1 from 0.091 to 0.240 and recall from 0.120 to 0.272. Aggregate task-completion reward was effectively unchanged. Cost moved in a third direction and depended on the task. Structured retrieval saved 15% to 51% where the baseline agent searched broadly, but cost an average of $0.15 more when the baseline localized the work in seven searches or fewer.

We measured completion with deterministic, task-specific verifiers rather than model judgment. We report both continuous reward and binary pass rate because they answer different questions: how close the agent came and whether it met the task's acceptance threshold.

The flat completion result is easy to misread. A later census found that roughly 74% of the corpus was solvable with grep alone. Thirty-eight percent of prompts named a path, symbol, or directory outright, while only about a quarter combined a behavioral instruction with an answer dispersed across the codebase. Most tasks therefore left little room for better retrieval to affect completion.

That task mix will dilute the aggregate effect of any retrieval system. When building your own evaluation set, first remove tasks that already tell the agent where to look, then report easily localized work separately from tasks that require genuine discovery.

Search difficulty matters more than codebase size

Classify the work you care about by localization difficulty before measuring anything.

Easy-to-localize tasks name a path, symbol, or directory, or yield the answer after a few searches. Hard-to-localize tasks describe a behavior whose implementation spans services, repositories, generated code, ownership boundaries, or git history.

Our retrieval-difficulty weighting, defined before we examined outcomes, assigns more weight to repository span and directory dispersion than to total lines of code. A 40-million-line monorepo behaves like a small repository when the relevant code sits in one directory. A two-million-line estate becomes the harder retrieval problem when the answer is scattered across four services.

Two forms of dispersion are especially important to include in a pilot. The first is history-dependent work: questions such as which change removed a line, across any branch or repository, cannot be answered from the current tree alone. git log -S can search that history, but only one repository at a time. The second is code that is generated, vendored, or owned by another team, where the answer exists but the person defining the task does not already know where to look. Both are common in incident response and compliance investigations, but uncommon in benchmark corpora, including ours.

You do not have to rely on a taxonomy to identify these tasks. Your baseline agent traces already provide a stronger signal. Before adding Sourcegraph, measure how much searching the agent performs: repeated queries, broad file reads, long trajectories, and high context consumption are better indicators of retrieval difficulty than repository size or workflow category.

Cost tracked most strongly with this observed search burden. Across 113 paired tasks, the correlation between Sourcegraph's cost advantage and baseline search breadth was (r = -0.57): the more the baseline agent searched, the more likely structured retrieval was to reduce cost.

In our runs, the tasks that benefited tended to cross four thresholds together: roughly 20 or more searches, 15 or more file reads, 30 or more turns, and 75,000 or more payload tokens in the baseline arm. These are reference points from one corpus, not universal cutoffs. Measure the same quantities against your own baseline before deciding where retrieval is likely to pay off.

The work that clustered above those thresholds included incident response, compliance investigations, migrations, and cross-repository tracing. In each case, the agent began with a symptom or behavior rather than a known location.

domain-160 shows why observed search burden matters more than a task label. We classified it as difficult because the instruction was behavioral and the answer was dispersed, but the baseline localized it in only two searches. Structured retrieval therefore cost more. The more useful predictor was not the category we assigned, but how much searching the baseline agent actually had to do.

Nine controls your evaluation needs

For task completion, the practical question is usually whether Sourcegraph performs at least as well while reducing search, time, or cost. State that objective before you run, let retrieval metrics carry the case for improvement, and apply the following nine controls.

1. Choose real work. Mine merged pull requests, incidents, migrations, compliance investigations, and cross-service changes. Tasks mined from private repository history substantially reduce the risk that the model has already seen the task or its solution, but a real change is not automatically a valid evaluation task. OpenAI's audit of SWE-Bench Pro estimated that roughly 30% of its tasks were broken, most often because prompts and tests disagreed, tests enforced an unstated implementation, or incomplete solutions could pass. Check that each prompt states what the verifier enforces, that the tests accept functionally valid alternatives, and that the reference change is not the only solution that can pass.
2. Remove trivial tasks. Reject prompts that name the answer's file, directory, or symbol. In our corpus, cost win rate fell monotonically as prompts revealed more about the answer, so this filter materially changes what the evaluation can measure.
3. Pin the code. Both arms must reach the same repositories at the same revisions. When we audited this, tasks where the baseline was missing repositories showed a +0.091 reward delta, compared with −0.007 under full parity. The former measures having code rather than finding it.
4. Hold the agent constant. Use the same model, verifier, budgets, runtime, and task text in both arms. Keep the task instruction byte-identical. Any Sourcegraph-specific preamble or tool description is part of the treatment and should be recorded and published with the result.
5. Compare access methods. Compare baseline local search with Sourcegraph-assisted retrieval. For a substitution test, remove local source access rather than asking the agent to prefer Sourcegraph, because an agent can fall back under pressure and turn the experiment into a test of instruction-following. This isolation condition is for attribution, not necessarily the configuration you would use in production.
6. Repeat each task at least three times. Average runs within each task before comparing arms. Single-trial designs overstate deltas by 40% to 60%, and within-task variance routinely exceeds the effect being estimated.
7. Measure the whole system. Record completion, retrieval, wall time, total tokens, total cost, search calls, and the amount of code entering context. Define the token-accounting boundary before the experiment and apply it identically to both arms. For implementation tasks, assign zero reward when the agent produces no required diff, so doing nothing cannot score the same as attempting the task and failing.
8. Segment before aggregating. Report easily localized tasks separately from dispersed and search-heavy ones. Pooling them can produce a number that answers neither deployment question.
9. Audit actual tool use. Configuration does not prove that the tool was available or adopted. Verify connectivity with observed tool calls, report zero-call trials as adoption or configuration failures, and separate them from analyses of retrieval capability.

How to interpret the result

Result	What it likely means	What to do next
Better completion and lower cost	Strong evidence that Sourcegraph improves both capability and efficiency	Expand the evaluation and prioritize rollout
Equivalent completion with lower cost or latency	Sourcegraph provides an efficiency advantage without reducing task quality	Deploy first to workflows with the largest savings
Better completion at higher cost	Sourcegraph enables work the baseline misses, but the added capability has a price	Decide whether the completion gain justifies the cost for that workflow
Better retrieval, equivalent completion, higher cost	The agent finds more relevant code, but that improvement is not reaching the final outcome efficiently	Examine the invocation surface, result shaping, and task population
Gains concentrated in dispersed or search-heavy tasks	The aggregate is hiding a workflow-specific advantage	Target Sourcegraph to those tasks rather than requiring a universal win
No gain and little baseline search activity	The selected work may not require indexed retrieval	Add tasks with behavioral prompts, repository span, or important history
Large gains when the baseline lacks required repositories	The experiment measures access to information rather than retrieval quality	Restore repository and revision parity before interpreting the result
Lower cost with worse completion	The agent is doing less work, not necessarily working more efficiently	Treat completion as the binding constraint and investigate the failure mode
Many zero-tool-call trials	The tools may be unavailable, poorly described, or not adopted by the agent	Diagnose connectivity and adoption before estimating retrieval capability

Your first experiment, start to finish

Nothing in the checklist above requires a particular harness. If you already have a system that runs an agent against a task and scores the result, you have most of the machinery. The missing pieces are usually clean isolation between arms, repeated trials, and an audit of whether the retrieval tools were actually available and used.

We run this evaluation with CodeProbe. It mines tasks from repository history, constructs the comparison arms, runs repeated trials, and reports paired per-task deltas. CodeProbe is still in alpha and is currently being tested internally and with partners, so expect breaking changes. Treat the within-experiment differences as the useful output; the absolute scores are not yet calibrated across experiments.

To evaluate Sourcegraph on your own repositories during the partner-testing period, contact me at stephanie.jarmak@sourcegraph.com.

The shape of the run:

# 1. Can this repository produce valid tasks at all?
codeprobe assess /path/to/repo

# 2. Build 40 tasks from merged pull requests. Ground truth comes from two
#    independent backends; --no-llm keeps mining inside your boundary.
codeprobe mine /path/to/repo --org-scale --count 40 \
    --consensus-backends ast,grep --no-llm

# 3. Define the comparison and its two arms.
codeprobe experiment init /path/to/repo --name sourcegraph-eval

codeprobe experiment add-config <exp> \
    --label local-baseline --agent <agent> --model <model>   # arm 1: local search

codeprobe experiment add-config <exp> \
    --label sourcegraph --agent <agent> --model <model> \
    --mcp-config '<sourcegraph-config>' \
    --mcp-mode strict --hide-local-source scaffold           # arm 2: retrieval only

# 4. Price it before you run it, then run it.
codeprobe run <exp> --dry-run                   # projected cost, launches no agents
codeprobe run <exp> --repeats 3 --parallel 5    # 240 runs: 40 tasks x 2 arms x 3 repeats

# 5. Paired per-task deltas, plus the validity gate described below.
codeprobe interpret <exp> --format html

This experiment asks one question: across 40 tasks mined from your own merged pull requests, does the same agent and model complete more work, or complete it at lower cost, when it reaches the code through Sourcegraph rather than local search? The two flags on the Sourcegraph configuration isolate the access method. --mcp-mode strict preserves the configured retrieval tools and write access while blocking local reads, grep, glob, and shell commands. This is an experimental condition, not a recommended production configuration. In production, an agent can use Sourcegraph alongside local tools. Isolation is useful here because it makes attribution possible, but strict mode also removes shell capabilities unrelated to retrieval. Its result therefore measures the complete isolated tool surface, not the retrieval backend alone. --hide-local-source scaffold replaces source files with zero-byte placeholders at their original paths, then overlays the agent's edits onto the real code before verification. The agent cannot inspect the local implementation, but the verifier still evaluates a real repository. Use hide instead when the required output is a text artifact rather than a code change.

Run --dry-run first and inspect the projected cost. Forty tasks across two arms and three repeats produce 240 agent runs. --max-cost-usd caps total campaign spending, while --parallel controls concurrency. A pilot of this size is useful for exposing integration failures and identifying directional differences between configurations, but it is not large enough to support a stable effect estimate.

Two checks belong in CI from the first run. The cross-validation gate fails when independent ground-truth derivations disagree. The validity gate in interpret refuses to report results while unresolved infrastructure failures remain in the analysis population. This matters because repository history provides realistic work, not automatically valid evaluation tasks: tests written for a particular pull request may encode that implementation rather than define the full set of acceptable solutions.

What this evaluation cannot tell you yet

Results from public repositories do not automatically transfer to private enterprise code, which is why this evaluation belongs on your own repositories. A small pilot can expose integration failures and directional differences, but it cannot establish a stable effect. In our analysis, roughly 80 paired tasks was the statistical floor, while 200 provided a more defensible basis for a deployment claim.

Retrieval scoring also cannot replace task verification. An agent can find every relevant file and still produce an incorrect answer, or arrive at the right answer without using retrieval at all. The invocation surface matters too: how an agent accesses retrieval can change cost as much as the retrieval results themselves. The most important population to test is cross-repository work grounded in code history, because that is where indexed retrieval differs most from local search and where our own corpus had the least coverage.

A useful pilot does not need to show that Sourcegraph improves every task. It needs to identify the work where indexed retrieval changes the outcome: tasks involving broad search, dispersed ownership, cross-repository dependencies, or important history.

Start with 40 representative tasks to validate the integration and surface obvious differences. Expand to at least 80 paired tasks before treating the measured effect as stable, and closer to 100 when retrieval quality is the primary endpoint, since that was the hardest effect to estimate precisely in our power analysis. Segment the results by baseline search breadth, and deploy first where completion remains steady while search, time, or cost declines.

Treat the numbers in this post as a hypothesis about your codebase, not as a benchmark score. If your results point in the opposite direction from ours, that is the result worth investigating, and we'd love to hear about it.