Announcing the LLM Litmus test
Cody gets even better with multi-repo context support, faster completions, improved commands, and much more. Read on for all the details.

Cody gets even better with multi-repo context support, faster completions, improved commands, and much more. Read on for all the details.
Today, I'm excited to announce that we're launching a new initiative called Sourcegraph Labs. It's our place to run AI experiments that can help make us more productive coders.
The first AI experiment we've created is called the LLM Litmus Test! This is a tool that lets you compare different large language models (LLMs) like GPT-4 Turbo, Mixtral 8x22b, and Claude 3 Opus.
We've given you the choice of which LLM, but today we're providing a tool to help you pick YOUR LLM.
These are the main things that you would find valuable with this tool:
Try out the LLM Litmus Test ->

Here at Sourcegraph, we believe in giving users the choice of which LLM to use for their coding tasks.
When you use Cody, we let you pick your LLM for:
We're a big believer in being LLM agnostic. Each LLM has its own strengths for specific tasks. But we've always gotten the question:
Which LLM is right for me?
As with almost all things in software development, the answer is "it depends". Now that the LLM Litmus Test has been launched, we can say "try them all out for yourself".
In addition to comparing LLMs, you can also add a GitHub repo, or repos, and turn on Cody Context. This means that we will search that repo for relevant code snippets to pass to an LLM so that you can get the most accurate and relevant answers.

You can compare the same LLM with and without Cody Context to see how big of a difference having the correct context makes in your answers.
Right now, you can compare the following LLMs.

Here are some examples that we've created so that you can dive right in and see LLM comparisons.
This example is using Claude 2.1 vs Claude 3 Opus.

In this example, I am comparing GPT-4 Turbo (April 2024) vs Claude 3 Opus. I have added the recharts library which is a React charting library.

We have plenty of AI experiments coming up next. We want Sourcegraph Labs to be a place we can iterate quickly to create practical AI projects that you would actually want to use.
Here's a few experiments we're noodling on:
We think the LLM Litmus Test will be a handy tool to have in your developer toolbox. It can help you choose the right AI assistant for your needs and even teach you new techniques from your own code.Â
The LLM Litmus Test is available starting today - just head over to s0.dev to take it for a spin. Let us know what you think!

With Sourcegraph, the code understanding platform for enterprise.
Schedule a demo