Sourcegraph 3.7: Improved performance, efficiency, accuracy, and reliability

Sourcegraph is the standard developer platform for code search and navigation at many of the largest and most exacting technology companies. With Sourcegraph, every company has access to the same kind of tools that Google and Facebook developers use every day.

🏎 Symbol search performant at scale Symbol searches are now indexed for default branches.

🔎 Search performance, efficiency, and reliability improvements Speed improvements for repohasfile:, more frequent garbage collections, and support for Unicode search results.

🧠 More accurate TypeScript code intelligence Jump to definition is more accurate, with fewer false positives.

🐇 Improved efficiency of Git requests Git requests use protocol version 2 where possible, increasing request efficiency to your code host.

⏎ Multi-line (\n) search on all branches

🧩 Language extensions get icons

🛠️ Sourcegraph configuration temporary overrides in the UI Admins can now override config files temporarily to test out changes.

👀 In case you missed it... Other Sourcegraph updates.

📝 Changelog Every detail that changed in this release.

🎖️ Thank you Sourcegraph couldn't be what it is without the community.

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

Symbol search performant at scale

View on Vimeo

Symbol search (type:symbol) is a powerful feature for finding specific symbols, such as a function, variable, or package, and not just all text occurrences of your query. More and more users are taking advantage of symbol search results, which was leading to performance issues for some users with large instances. Symbol search is now indexed for default branches, which is a major improvement for both users running symbol searches and admins with large instances that were having performance issues. On large symbol searches, we've seen speed ups by factor of 20x (benchmarks).

Upon upgrading, Sourcegraph will seamlessly & automatically re-index repositories in the background. Once finished, you'll see substantially improved symbol search performance. In the meantime, Sourcegraph will perform as it would normally.

Deployment note

Prior to upgrading, please consult our 3.7.2+ migration guide to ensure you have enough free disk space.

Search performance, efficiency, and reliability improvements

As we continue to make incremental improvements, Sourcegraph search is getting faster, more efficient, and more reliable across the board. In Sourcegraph 3.7:

• Symbol search is now performant at scale (see above)
• Searches making use of the repohasfile: filter are now faster.
• The indexed-search code path, Zoekt, now runs garbage collections twice as frequently, which reduces Sourcegraph's memory consumption.
• We've improved support for Unicode search results, so that combined characters now highlight properly.

More accurate TypeScript code intelligence

View on Vimeo

Out-of-the-box TypeScript code intelligence has been improved with an updated ctags version that includes a built-in TypeScript parser. This means that jump to definition is more accurate for TypeScript files, and the symbols sidebar recognizes more functions and variables with fewer false positives.

Improved efficiency of Git requests

We now use Git protocol version 2 with compatible code hosts. This can significantly increase the efficiency and performance of a git fetch by reducing the amount of data transmitted over the wire, especially when Sourcegraph's copy is only a few commits behind the remote (a common situation).

As an illustrative example, when Git protocol version 2 is used, git fetch on an up-to-date clone of Kubernetes is up to 2x faster and consumes up to 4x less client CPU (averaged across 10 runs on a 2015 MacBook Pro).

You can read this blog post for more technical details and performance measurements.

Multi-line (\n) search on all branches

In Sourcegraph 3.5 we introduced the ability to do a multi-line search in by using \n in queries, however, it was limited to only indexed default (e.g. master) branches. In 3.7 we expand this option to include unindexed branches so you can now perform multi-line searches on any branch.

Language extensions get icons

Icons have been added to the language extensions in the extension registry, and will soon be added to the external service extensions, such as Codecov and Sentry.

Sourcegraph configuration temporary overrides in the UI

Sourcegraph 3.4 introduced an optional way to load Sourcegraph configuration from a file or Kubernetes config map. It is helpful for teams with Sourcegraph as a critical piece of infrastructure, to be able to check the Sourcegraph configuration into version control. We received feedback that it would be nice to be able to make temporary edits in the web UI and diverge from the files on disk intentionally (i.e., to try an option out before committing it).

You can now set EXTSVC_CONFIG_ALLOW_EDITS=true and SITE_CONFIG_ALLOW_EDITS=true to allow the active Sourcegraph configuration to diverge from the file, thus allowing temporary edits. For complete details, see the documentation.

In case you missed it...

We were proud to host the official liveblog for GopherCon 2019, and all 28 talks are now live.

Some of our favorites were:

• Two Go programs, three different profiling techniques, in 50 minutes by Dave Cheney
• Generics in Go by Ian Lance Taylor
• Get Going with Web Assembly by Johan Brandhorst
• How Uber "Go"es by Elena Morozova

Check out the rest of the talks!

Changelog

3.7.0

Added

• Indexed search now supports symbol queries. This feature will require re-indexing all repositories. This will increase the disk and memory usage of indexed search by roughly 10%. You can disable the feature with the configuration search.index.symbols.enabled. #3534
• Multi-line search now works for non-indexed search. #4518
• When using SITE_CONFIG_FILE and EXTSVC_CONFIG_FILE, you may now also specify, e.g. SITE_CONFIG_ALLOW_EDITS=true to allow edits to be made to the config in the application which will be overwritten on the next process restart. #4912

Changed

• In the GitHub external service config it's now possible to specify orgs without specifying repositoryQuery or repos too.
• Out-of-the-box TypeScript code intelligence is much better with an updated ctags version with a built-in TypeScript parser.
• Sourcegraph uses Git protocol version 2 for increased efficiency and performance when fetching data from compatible code hosts.
• Searches with repohasfile: are faster at finding repository matches. #4833.
• Zoekt now runs with GOGC=50 by default, helping to reduce the memory consumption of Sourcegraph. #3792
• Upgraded the version of Go in use, which improves security for publicly accessible Sourcegraph instances.

Fixed

• Disk cleanup in gitserver is now done in terms of percentages to fix #5059.
• Search results now correctly show highlighting of matches with runes like 'İ' that lowercase to runes with a different number of bytes in UTF-8 #4791.
• Fixed an issue where search would sometimes crash with a panic due to a nil pointer. #5246

3.6.2

Fixed

• Fixed Phabricator external services so they won't stop the syncing process for repositories when Phabricator doesn't return clone URLs. #5101

3.6.1

Added

• New site config option branding.brandName configures the brand name to display in the Sourcegraph element.</li> <li><strong><code>repositoryPathPattern</code></strong> option added to the "Other" external service type for repository name customization.</li> </ul> <p>The <a href="https://github.com/sourcegraph/sourcegraph/blob/main/CHANGELOG.md#370"><strong>changelog for this and previous releases</strong></a> is available on GitHub.</p> <h2 id="thank-you">Thank you</h2> <p>Thank you to the many people who contributed to Sourcegraph since the last release!</p> <ul> <li><a href="https://github.com/YouveGotMeowxy"><strong>@YouveGotMeowxy</strong></a></li> <li><a href="https://github.com/thinktopdown"><strong>@thinktopdown</strong></a></li> <li><a href="https://github.com/JoseFdri"><strong>@JoseFdri</strong></a></li> <li><a href="https://github.com/wxio"><strong>@wxio</strong></a></li> <li><a href="https://github.com/abeyerpath"><strong>@abeyerpath</strong></a></li> <li><a href="https://github.com/asinwang"><strong>@asinwang</strong></a></li> <li><a href="https://github.com/marco-c"><strong>@marco-c</strong></a></li> <li><a href="https://github.com/Warchant"><strong>@Warchant</strong></a></li> <li><a href="https://github.com/pplanel"><strong>@pplanel</strong></a></li> <li><a href="https://github.com/VeselaHouba"><strong>@VeselaHouba</strong></a></li> <li><a href="https://github.com/Phrohdoh"><strong>@Phrohdoh</strong></a></li> <li><a href="https://github.com/trollknurr"><strong>@trollknurr</strong></a></li> <li><a href="https://github.com/pheex"><strong>@pheex</strong></a></li> <li><a href="https://github.com/xingx-xx"><strong>@xingx-xx</strong></a></li> </ul>