Improvements

• Cody Gateway Add new claude-3-5-haiku-latest model

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

  Cody Gateway: add new claude-3-5-haiku-latest & claude-3-5-opus-latest models to Cody Gateway allow list Cody Gateway: add new claude-3-5-haiku-latest dotcom models list
  Backport 11e7481ba3c810ae5b47d32b8cf32066e2f0b2bb from #1470

• Cody Add more API docs content

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Cody Add embedded API docs via OpenAPI/Redocly
  • New OpenAPI API docs within the instance at the URL $SRC_ENDPOINT/api/openapi/public. Alternatively, reach the page at "User > Settings > OpenAPI Reference".
• Analytics Log estimated LLM cost telemetry (CODY-4007)
  • Adds cost estimates to the ModelConfiguration returned by .api/llm/supported-models.json
  • Adds llmTokenUsageCostEstimate telemetry value with estimates of each requests LLM cost in pennies
• Analytics Add check for PGDATASOURCE

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Administration Site config 'telemetry: { disableLocalEventLogs }' to disable event_logs

  Long-term local retention of user telemetry as 'event logs' can now be disabled entirely via the telemetry: { disableLocalEventLogs } site configuration.

• Add embedded file snippet page
  • Added embeddable page for file snippets
• Cody Gateway Deprecate old claude 3.5 sonnet

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Cody Add Claude 3.5 Sonnet (Latest) & Claude 3 Opus (Latest)

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Sub_repo_perms Add conditional logging for seeing if sub_repo_permisisons are working

  The sub repository permissions implementation now emits logs that describe the rules that were evaluated for a given request whenever tracing is enabled. (We limit these logs for only when tracing is enabled since they can be quite verbose).

• Analytics Add source.server.server_client.user_agent to telemetry payload
  • Add the sourcegraph API client user-agent to telemetry payload types under source.server.server_client.user_agent.
• Cody Expose token usage in the LLM API
  • The /.api/completions/stream API now includes used input/output tokens and the upstream model name in the response body when using the query parameter api-version=5 or higher. Does not apply to: Vertex Anthropic, Azure OpenAI (mo model name, and no token usage in streaming response), Gemini (no model name in streaming response)
  • LLM chat completions now support stream: false when using Azure OpenAI and OpenAI-compatible providers.
• Analytics Propagate new x-sourcegraph-api-client-* headers to request clients
  • Propagates new X-Sourcegraph-API-Client-Name and X-Sourcegraph-API-Client-Version headers to request clients
• Ci Record Test infrastructure failures
  • add library that writes to TEST_INFRASTRUCTURE_FAILURE_FILE if it is defined for test failures
• Batch Changes Show name of additional fields that cause errors

  feat(batches): show name of additional fields that cause errors

• Cody Add support to format raw API responses for Cody
  • The raw HTTP API now accepts the query paramter format=cody or header Accept: application/cody to render files and directories as context items for Cody. Example request path: /github.com/sourcegraph/cody/-/raw/agent/src/index.ts?format=cody.

Fixes

• Batch Changes Enable request splitting by default

  fix(batches): enable request splitting by default

• Source Correct Name() documentation for gitserver's ReadDirIterator

  The documentation for gitserver's ReadDir method has been clarified to reflect that the return iterator's Name() method returns the full path of the file, as opposed to just the basename.

• Analytics Only log Cody API events to a remote data store, do n…

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Codyapi Restrict audit log retrieval to site admins only

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Dev Correctly propagate error key-value pairs

  Fixes a bug where logs and traces were sometimes missing key-value pairs recorded alongside errors.

• Source Clarify exclude docs in github connection schema to say that individual expressions within block are `and`-ed together

  The documentation for the "exclude" section in the github code host connection schema has been clarified to explain that each block is OR'd together, and the expressions within each block are AND'd together.

• Source Multiple GitHub external accounts to the same URL now refresh correctly
  • When there are multiple auth providers configured that point to the same GitHub URL, as can be the case when using private GitHub Apps, user external account tokens will now refresh correctly.
• Code Search (new web ui) Make blob view readonly

  Prevents default browser shortcuts from modifying the file content locally.

• Batch Changes Handle spec being nil for suffix salt

  fix(batches): handle spec being nil for suffix salt

• Code Intel Add more limits in auto-indexing inference code

  Introduces new limits for auto-indexing inference to reduce the risk of continuously growing auto-indexing queues.

  • The number of jobs spawned per round of auto-indexing inference per repo is capped (default: 100) to reduce risk of clogging of auto-indexing queues. Excess jobs will be discarded.
  • The number of paths inspected for a single round of auto-indexing inference per repo is capped (default: 500) to reduce risk of timeouts. Excess paths will be discarded.
• Code Intel Handle annotated tag hashes passed as 'commit' for uploads

  Fixes handling of SCIP uploads when the hash passed to the -commit flag of src-cli corresponds to the hash of an annotated tag instead of the hash of a commit. Previously, these uploads were not accessible for code navigation.

• Source The sync of a GitHub repositoryQuery failing will no longer cause repos to be deleted
  • GitHub code host connections using repositoryQuery will no longer delete repositories if the sync fails for reasons like a GitHub outage or a token expiring.
• Batch Changes Enable GQL request splitting via BATCH_CHANGES_REQUEST_SPLITTING

  fix(batches): enable GQL request splitting via BATCH_CHANGES_REQUEST_SPLITTING

• Codyapi Add usage to completions/chat (CODY-3498)

  Added token usage information to .api/llm/chat/completions

• Code Nav Trigger hovers for all programming languages

  Fixes a bug where hovers were not triggered for less mainstream languages like F#, Standard ML, Visual Basic, Pkl, Hack, MATLAB etc.

• Code Search Exclude content filters from phrase boosting

  This fixes a bug where we would apply a phrase boost if a content: filter was specified with keyword search enabled. This led to inconsistent behavior (regexp vs keyword search) and it also rendered the content: filter ineffective, because we ran a general text search instead of a just a content search.

• Code Monitor Fail monitors with missing/unverified primary email address

  fix(code-monitor): fail monitors with missing/unverified primary email address

• Release Add stitched migration graph override (#935)

  NA

• Batch Changes Display "Deleted Namespace" instead of throwing an error

  fix(batches): display "Deleted Namespace" instead of throwing an error

• Code Intel Propagate error collector to fix retention tab GraphQL API call

  Fixes a bug where the retention tab for precise indexes would show a nil pointer exception instead of retention information.

Other

• Release: Prep stitched migration graph for release (#1388)

  n/a

• Enable reranker by default if Gateway is enabled

  Cody Chat: the context engine now uses the reranker by default everywhere where Cody Gateway is enabled.

• Add code llama 7B model for ab test
  • adding code llama 7B for completions
• Hide auto submit checkbox and add promoted badge

  OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Worker: Fix configuration error reporting

  A bug in configuration validation in the worker service has been fixed. Configuration mistakes are now correctly reported again.

• Creating/Editing Prompts Re-design
  • The prompt creation/edit page has been redesigned.
• Add recommended flag to Prompt Templates
  • Adds ability to mark prompts as recommended and then sort the recommended prompts at the top.
• Analytics Report token usage per request to telemetry

  Added LLM token usage telemetry

• Code Search Remove all non-single job pod code from Executors on Kubernetes.
  • The environment variable KUBERNETES_SINGLE_JOB_STEP_IMAGE is now KUBERNETES_JOB_STEP_IMAGE.
  • Both environment variables are read, with preference given to KUBERNETES_JOB_STEP_IMAGE, to preserve backward compatibility, but reading of KUBERNETES_SINGLE_JOB_STEP_IMAGE may be removed in a future version.
• Bugfix: escape spaces in repo filter names
  • Fixed a bug that would cause filters added from the search sidebar to not be correctly escaped
• Add prompt mode
  • Adds mode (CHAT, EDIT, INSERT) to prompt.
• Feature: add internal/tracelog: a log.Logger that only logs if tracing is enabled

  Added a new internal/tracelog package that provides a conditional logger for tracing. This logger only logs when tracing is enabled, improving performance by reducing unnecessary logging.

• Feature/source: implement core logic for parsing / evaluating Perforce IP addresses for sub repo permissions

  For the Perforce IP permissions implementation, this PR introduces machinery that implements the ability to parse the IP address expressions from the perforce protections table and compare them against the user's IP address (that's advertised from the incoming X-FORWARDED-FOR header.

  Namely, this PR:

  • Introduced parsePerforceIPString, which parses Perforce IP strings (e.g., CIDR expressions, specific IP addresses) into an ipMatcher for IP validation.

  • Implemented multiple ipMatcher types, including:

    • addrMatcher for exact IP matches.
    • cidrMatcher for checking IPs within a CIDR range.
    • alwaysTrueMatcher for wildcard * expressions.

  • Added toggleableIPMatcher, which either enables or disables IP enforcement based on site configuration settings.

  • Created matcherCache for caching recent IP string to ipMatcher translations, improving efficiency.

  • Updated NewRequestClientIPSource to properly derive IP from the x-forwarded-for header.

• (feature): site-config: add ipParseCacheSize to schema for enforce IP restrictions schema

  Added a new ipParseCacheSize configuration option for sub-repo permissions to control the caching of Perforce "Host" to IP address translations.

• Feature/site-config: add site configuration setting for enforcing IP restrictions

  Added enforceIPRestrictions setting to the SubRepoPermissions section in site configuration, allowing IP-based enforcement using the X-FORWARDED-FOR header.

  Updated JSON schema to require SubRepoPermissions to be enabled when enforceIPRestrictions is enabled.

  Added validation in Perforce external service configuration to ensure IgnoreRulesWithHost and enforceIPRestrictions cannot be enabled simultaneously.

  Updated internal implementation to associate each Perforce rule with an IPMatcher for IP-based rule enforcement.

• Feature/plumbing: sub_repo_perms: do pumbling to thread comparing IP addresses alongside paths

  This PR introduces logic for IP-based permissions in the authz provider for Perforce: 1. Modified interfaces in internal/authz/sub_repo_perms.go to accept caller's IP address for enforcing IP permissions. 2. Introduced a new IPSource interface to provide IP addresses for sub-repo permissions implementation. 3. Updated rule implementation to include an IPMatcher for comparing provided IP addresses with Perforce rule IP addresses.

  This change lays the groundwork for supporting IP-based permissions in Perforce authz provider.

• Add autoSubmit & includeViewerDrafts to Cody prompts
  • Adds options to set prompts to "Auto Submit", which would automatically execute the prompts in one-click.
• Fix bug where text pushed buttons out of the container
• Add ability to duplicate existing prompts
  • Users can now duplicate existing prompts.
• Security Update rules_apko and fix issues with sg wolfi v2
  • Build containers using latest version of apko and rules_apko
• Blob View: Ensure copy button copies full path and not displayed path
  • Fixes a bug where certain copy path buttons were only copying the visible path and not the full path of a file.

Other

• Security Upgrade src-cli version to address CVE

  Upgrade src-cli version to 5.8.1 to address CVE-2024-24788, CVE-2024-24790, CVE-2024-34156

  
  Backport b8da20f6fa148f30fe97d9267a536bd5a84502f1 from #1112

• Update src-cli MinimumVersion from 5.5.0 to 5.8.1

  chore(release): Update minimum supported version of src-cli to 5.8.1

  
  Backport 8f039bdcb5425071dd6e4dfddfb2cb436e124b05 from #1079

Improvements

• Local Add cody-gateway version endpoints
  • add cody-gateway environments for subcommand live
• Code Search Bump Zoekt for ranking and memory improvements
  • Reduce peak memory required for Zoekt indexing
  • Improve search ranking by using repo freshness as a scoring signal
• Code Intel Updates the autoindexing images for TypeScript and Ruby
  • TypeScript: Fixes references to object properties in various places
  • Ruby: Fixes references to Opus:: in Ruby tests
• Ci Enable workflows telemetry
  • enable aspect workflows ci telemetry
• Code Search Enable similarity search by default

  With this change we rank repository search results by similarity. Before, repository search results were ordered by star count instead. This affects suggestions and repository search results. To disable this feature and return to the previous behavior, set { "experimentalFeatures": { "disableOrderBySimilarity": true}} in global settings.

• Cody Add basic support for server-rendered prompts
  • /.api/completions/stream endpoint now accepts "file" and "repo" parts alongside "text" and "image_url"
• Release Use releaseregistry for release banner [REL-145]
  • implement using the release-registry for the banner
• Ci Convert onconflict gen to bazel
  • generate constraints.go using bazel for onconflict linter
• Cody Add vision support
  • /.api/completions/stream now supports vision using the OpenAI-compatible base64 encoding of images. Example "content": [{"type": "image_url", "image_url": { "url": "data:image/png;base64,{{IMAGE_BASE64}}" } }]. Requires the query parameter api-version=3 or higher.
• Source Add ssh auth support for more code hosts
  • BitBucket Cloud, BitBucket Server, GitLab, Gerrit, Gitolite, AWS CodeCommit Azure DevOps code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.
• Source Add support for ssh auth to github
  • GitHub and generic Git code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.
• Batch Changes Verbose error logging for batch syncer with BATCH_CHANGES_SYNCER_VERBOSE_ERROR_LOGGING

  feat(batches): verbose error logging for batch syncer with BATCH_CHANGES_SYNCER_VERBOSE_ERROR_LOGGING

• Release Remove test from promotion ci during release pipeline
  • Remove upgradetest from the release promotion ci
• Code Search Respect match order for search.results.repositories resolver

  The GraphQL endpoint search.results.repositories now returns the list of repositories in the order of the matches we found. Before, repositories were sorted by id.

• Embeddings: v2
  • Introduce a new experimental embeddings index and context retriever.
• Wolfi Update server + postgresql-12 images
  • add pgvector to server and postgresql-12 images
• Wolfi Add pgvector-12 package
  • build and add pgvector extension
• Batch Changes Add experimental feature to add a fork name suffix to avoid name collisions

  feat(batches): add experimental feature to add a fork name suffix to avoid name collisions

• Release Stop releases from going out if there's a PR with the release-blocker label
  • Allow PRs to block a release from going out 😈

Fixes

• Code Search Fix client-side query validation for rev: filters

  We have updated the client-side query validation to allow combining rev: filters with query-based search contexts.

• Code Intel Correctly escape special characters in repo names and identifiers

  The new usagesForSymbol API correctly searches repos whose names contain spaces or other special characters.

• Code Search In search jobs, don't time out fetches

  For very large repositories, search jobs could time out while fetching the repository. Now, search jobs can fetch and search over repositories, even if they take a long time to fetch.

• Batch Changes Report correct version in AMI
  • Fixed incorrect executors version reporting which indicated 0.0.0+dev even though it was the correct executor AMI version running under the hood.
• Batch Changes Stamp the executor binary with version information
  • override internal/version.version with x_defs at build time in executor
• Cody Disable ModelAPIEnabled for Dotcom

  modelsAPIEnabled is always false for dotcom

• Adjust formatting when using unix sockets
  • fix PGDATASOURCE format when using unix sockets
• Code Search Add index for search jobs janitor

  We added a new database index to speed up a janitor job which is run as part of Search Jobs. Before, the janitor job might have significantly delayed migrations during an upgrade.

• Analytics Ensure telemetry export only occurs once at a time
  • Fix an issue where duplicate telemetry can be exported when the worker service is scaled horizontally
• Cody Allow empty `finish_reason` in `/.api/llm/chat/completions`
  • The /.api/llm/chat/completions endpoint will now return an empty string "" for finish_reason instead of the string "unknown_please_report_bug()".
• Batch Changes Disabled "select all" checkbox if all nodes are already disabled

  fix(batches): disabled "select all" checkbox if all nodes are already disabled

• Local Sg - return the error when checking for remote branch/commits
  • dump more information out about the branch if it is out of sync
• Use preferred mermaid APIs
  • Fixes a bug that can cause errors when rendering many mermaid diagrams on a page.
• Code Intel Fetch tags for scip-go auto-indexing jobs

  Go cross-repo navigation for tagged versions should work better when using scip-go for auto-indexing. Navigation based on SCIP indexes generated using scip-go in CI or cron jobs should be unaffected.

• Release Remove embeddings from promotion script (#360)
  • remove embeddings in promotion script

Other

• Update lockfiles with newer version of apko

  
  Backport 7f2c3bacf6778fef5f2ef444db2ebd873903c5e6 from #881

• Rel Prepare stitch graph for 5.8

  
  Backport 6b58d4b62bba8f558c097fa97a751e51b178c537 from #850

• Code Intel Bump autoindexing image SHAs

  Upgrades default auto-indexing images to the latest versions (1) scip-go to v0.1.21 (2) scip-typescript to v0.3.14

• Adding code qwen 2p5 and deepseek long prompt optimized model
  • adding code qwen 2p5 and deepseek long prompt optimized model #811
• Enable reranker by default
  • Reranker for context retrieval is turned on by default
• Local/nix: add pgvector extension
  • enable pgvector in nix managed postgres db
• Code Intel Bump autoindexing image SHAs
  • Upgrades the default scip-go auto-indexing image to v0.1.20
• Final ranker config for context v2

  Changing the ranker configuration -- number of items fetched from zoekt

• Context: return multiple chunks per file from Zoekt results
  • We can improve recall by fetching multiple chunks per file from Zoekt results by turning on the cody-reranker feature flag.
• Context: improve ranker integration + loosen the tap
  • Ranker for chat context is now available behind use-reranker feature flag.
• Add(cody): `api-version=2` for incremental streaming LLM responses
  • The /.api/completions/stream endpoint now accepts an api-version=2 query parameter that returns incremental text text responses when using "stream": true to improve performance and reduce bandwidth.
• Ci Add annotation for no merge base
  • post an annotation if there is no merge base
• Svelte: fix loading for ref panel
  • Fix loading behavior for the reference panel in the web app rewrite
• Svelte: fix text wrapping for file popover
  • Fixed a bug in the web app rewrite where long paths would cause file popovers to wrap
• Retrieve and display change log during upgrades.
  • [Appliance] Retrieve live changelog and display information about the release to be upgraded.
• Release: specifiy download directory for changelog
  • specify download directory for changelog
• Code Intel Bump auto-indexing image SHAs

  Bumps default auto-indexing scip-go image to use Go 1.23.1 and scip-go v0.1.19

• Appliance Admin UI
  • [Appliance]: Adds the upgrade path for the Appliance Admin UX
• Ci Update gomod lint
  • remove lint that checks for promotheus/common 0.32.1 since we are using 0.48
• Ci Upgrade golang.org/x/tools to 0.24
  • address printf linter flagged issues
• Add(cody): `api-version=2` incremental streaming LLM response
  • The /.api/completions/stream endpoint now accepts an api-version=2 query parameter that returns incremental text text responses when using "stream": true to improve performance and reduce bandwidth.
• Svelte: add symbol tree
  • Adds a symbol tree to file page in the experimental webapp

Fixes

• Code Search Remove query expansion

  This fixes a bug where we added "readme" too often to the context.
  Backport 28ff196a663f537c6cb6340f976a91431509a90e from #582

• Batch Changes Workaround for a bug in GitHub

  fix(batches): workaround for a bug in GitHub
  Backport aad3a04f8c93561a61c404e69132e70a22d0acba from #576

• Release Remove the other embedding reference

  n/a

Improvements

• Batch Changes Add a container registry deny list
  • Add a container registry deny list to complement the allow list.
  • Configure in site config via "batchChanges.containerRegistryDenylist".
  • Mutually exclusive with "batchChanges.containerRegistryAllowlist".
• Cody Add `/.api/cody/context` API
  • New POST /.api/cody/context REST endpoint to retrieve a list of relevant source locations given a natural language query.

Fixes

• Local Check for rogue files and folders in svelte routes
  • prevent web-sveltekit commands from running if there are untracked files under src/routes
• Code Search Skip if git diff not found in hybrid

  When searching an unindexed commit we would consult indexed commits for speeding up results. If our index contained a commit that no longer existed in git we would error out due to a regression in v5.4.5099. This is now fixed.

• Sg Make start commands cancel fn be sync.OnceFunc
  • the cancel funcs used by sg commands are now wrapped in sync.OnceFunc to prevent duplicate execution
• Sg Add deprecation notice to sg wolfi update-hashes
  • sg: fix panic when using wolfi update-hashes
  • sg: add deprecation notice for wolfi update-hashes
• Cody Correctly parse queries containing 'or'

  Fixes a regression in Cody context where questions containing the word 'or' could return noisy or no results.

• Repo Updater Add WARN level logs every time we sync a code host

  repo-updater now emits logs that log the result of every code host sync.

• Sg Check if we are ephemeral before getting lease time
  • sg - fix panic in Cloud Ephemeral listing when listing instances that are not Ephemeral
• Sg Clamp deployment name consistently in cloud ephemeral
  • ensure deployment / instance names are clamped in all places for cloud ephemeral
• Batch Changes Avoid "Name already exists on this account" from creating fork by fetching the repo when the error happens

  fix(batches): avoid "Name already exists on this account" from creating fork by fetching the repo when the error happens

• Cody Fix error handling in LLM API
  • LLM API endpoints (/.api/llm) now return JSON-encoded HTTP bodies for non-200 status codes.
• Cody Return valid `finish_reason` in `/.api/llm/chat/completions`
  • LLM API /.api/chat/completions now returns OpenAI-compatible finish_reason.
• Cody Allow `Bearer TOKEN` header for all LLM APIs
  • For compatibility with OpenAI clients, it's possible to use Bearer TOKEN header with all API endpoints that start with the prefix /.api/llm.

Other

• Siteadmin: Make recoverUsers idempotent

  The recoverUsers endpoint failed with a spurious error when some of the given users were already active. It is now idempotent.

• Wofli Update images
  • update images to use latest p4-fusion binary
• Add option to cache label data with p4-fusion
  • Perforce connections now support a cacheLabels option to cache Perforce label data from the server, speeding up consecutive syncs on systems with a large number of labels.
• Release Simplify changelog download in release
  • simplify changelog cli download by using gh cli
• Configure and activate Admin UI
  • [Appliance] Activate appliance updates on Code Search admin UI
• Bug: Fix file/directory popover regression

  Fix File and directory popovers in the file tree when code search is scoped to a perforce depot.

• Web: encode file path for blame
  • Fixes an issue that would cause blame view to fail on files that contain some special characters.
• Fix main lint
  • [Fix] code linting
• Add perforce support for git references table and labels
  • Code Search now supports labels for Perforce Depots
• Code Search Deprecate experimental site setting

  The site setting experimentalFeatures.searchJobs is not read anymore. To disable Search Jobs, set DISABLE_SEARCH_JOBS=true for the "frontend" and "worker" services.

• Migrate httpserver to use sg/log
  • update httpserver to use sourcegraph/log instead of log15
• [Stream API]: Add external_service_type field to the SearchedRepo type

  Searched repo, commit, path, symbol and file match responses will now include external service type.