fix: prompt templates should not fail when pasting linebreaks

• You can now create a GitHub App with all the right permissions/events for code review agents. Previously, you had to manually customize the apps.
  Backport 87e8d77f464912cbd4356a8c3c39fba1d099e3b6 from #3473

Fixes a bug in SCIP index processing for instances with a long history of processing large uploads.
Backport 48e7b47898ee7710f12270c6861c335a2ef75f48 from #3595

• Errors are now rendered as strings in agent run logs. Previously, they rendered as Source: {}, which wasn't helpful.
  Backport d1f18d280fc9d3a985a284c31e819412d2add606 from #3514

• HTTP header auth username and email headers, if configured, are no longer blocked by CORS.
  Backport 782b98a780dac335576b8f43affb4b1a10123882 from #3512

fix(rel): remove TimescaleDB from existing database if found during upgrade to Postgres 16 on the codeinsights database.
Backport 71b4af3d6faef054803db0151b2cc7b151bb1c0e from #3556

fix: transformChanges.group.directory now ignores file names
Backport 3b76fe4ab146565b0e736231353b1e24f1468241 from #3576

• Fixes an (unreleased) issue that could cause duplicate search results when switching between tabs.
  Backport 746a29a25d6e54dfe7ab38f70855a9f58a652426 from #3528

• Code review agents can now automatically run on GitHub Pull Requests (actions: opened and synchronize) based on a feature flag. Both boolean (true/false) and rollout (percentage-based ) feature flags are supported. For example, this means you can enable automatic reviews on 10% of all opened PRs.

• The Code Review Agent now reports live status with the GitHub Commit Status API making it possible to open Agent logs directly from GitHub.

Code reviews deduplicate diagnostics from historical reviews

Adds a feedback UI for diagnostics within the Agents app.

• You can now request review from the Review Agent by posting a pull request comment with a configurable substring

• Add GET /.api/agents/runs to list runs of an agent, and other related endpoints including the ability to view logs

• The Review agent can now review larger diffs.

Adds HTTP handlers for /.api/conversations for creating and filtering conversations as well as a DB method for querying conversations.

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Move Gemini 2.0 Flash from Experimental to GA, add Gemini 2.0 Flash-Lite Preview Experimental and Gemini 2.0 Pro Experimental

• feat: add init cmd subsection to release.yaml parser
• feat: add init section to release.yaml

• allow the upgradetest to be run locally without knowing the bazel invocation required to stamp the build etc
• unlock local minor and major branch upgradetests

feat: prompt templates editor supports dynamic @ mentions

• Creating a new agent now redirects to the settings page for further setup

• Rule include/exclude patterns are now interpreted as globs (*.go) instead of regexp (.*\.go). Negative include patterns like !*.go will be interpreted as exclude patterns, and vice versa.

fix/auth: "http-header" auth provider can "allowSignup": false to disable automatic account creation

fix(rel): fix migrator upgrade check

Fix bug where the URL of a Gerrit code host connection could not be updated.

The logic that translates Bitbucket clone URLs to repository names has been fixed to support:

• URLs that have no scheme (like "bitbucket.sgdev.org/sourcegraph/sourcegraph")
• SSH clone urls that don't have a ssh:// scheme prefix (like [email protected]:sourcegraph/sourcegraph.git)

A bug on the user's account security page that could result in duplicated / buggy entries has been fixed.

• single docker server checks for 5.10-reindex.completed instead of 5.1-reindex.completed

fix: prompt templates should not fail when pasting linebreaks

Move Gemini 2.0 Flash from Experimental to GA, add Gemini 2.0 Flash-Lite Preview Experimental and Gemini 2.0 Pro Experimental

Co-authored-by: arafatkatze [email protected] (cherry picked from commit 4aa5aa41cc0f5e2be80c77e8e8709e198ff54b94)

PR description tips: https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e

feat: prompt templates editor supports dynamic @ mentions

PR description tips: https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e

Fix bug where the URL of a Gerrit code host connection could not be updated.

(cherry picked from commit e6da1ceb9586bf109339f06220c1fdbbf570a6d9)

Commit listing should work correctly for repos which contain arbitrary characters in commit messages. This also affects downstream functionality such as commit graph updates needed for precise code navigation.

Backport 2eae8e1 from #3359

fix: retry graphql queries that failed with 502

Combines Cody chat and Code Search into a single unified interface.

Improved highlighting for JSON files

• The Review agent now requires the program file .sourcegraph/review-agent.json with at least the content "{}" to enable posting Pull Requests review through the GitHub App. The JSON file may contain additional configuration options to override defaults of the ReviewRequestConfiguration object (see OpenAPI spec)

• The Review Agent now includes the rule ID when posting comments on pull requests. By default, the minimum review severity is errors (previously it was warning). When the minimum severity is overridden to be warnings, then the review comments will also include the severity.

• Configure custom review rules for the Review Agent by adding *.rule.md files nested under .sourcegraph/ directories in any ancestor director of files in the diff.

• New /.api/agents/** endpoints to manage the creation of Sourcegraph Agents (EAP feature).
• New support to automatically review GitHub Pull Requests with a Review Agent via GitHub Apps (EAP feature).

• New GET /.api/tools endpoint to list tool definitions for Sourcegraph API endpoints to use with the LLM /chat/completions endpoint for function calling.
• New POST /.api/review/diff endpoint to get a list of code review comments given a git diff (including GitHub pull requests and Gitlab merge requests).

• Normalize drift detection to handle for postgres 12 to 16 upgrade

feat: prompt templates now support specific @ mentions

• For broad matches, we now center the best-scoring line in search results

• The /.api/llm/chat/completions endpoint now support function calling via the tools property. This feature works when using the LLM providers: OpenAI, Anthropic, Fireworks, AWS Bedrock, and Gemini.

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• fixes 404 errors when trying to load "authSidebar.js" when the Code Search VS Code extension is used with the VSCode Remote SSH extension

In the site admin configuration for code graph policies, restricts the creation of policies for syntactic indexing to HEAD-only. This restriction may be lifted in the future. Policies for precise auto-indexing are not affected.

Backport 9639850855bb169f7dee4fbcff869c77c17f9328 from #3007

• Fixed bug in Review Agent where it didn't post review comments even for small diffs Backport 4710b59 from #3019

We are changing the behavior of session expiration to be a hard-cap instead of a rolling window. auth.sessionExpiry will now set a hard-limit on the sessions lifetime, and users will have to re-authenticate after this period. To preserve the previous behavior where sessions would be invalidated after they haven't been used for a while, a new configuration option named auth.maxSessionIdleDuration has been introduced that controls session expiration based on inactivity.

Fixed an issue where Sourcegraph user sessions initiated via OAuth integrations did not respect the auth.sessionExpiry configuration and expired much earlier.

Backport dc8deea005067145855b03a3344675386bc75be7 from #3078

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

N/A

N/A

fix(rel): add handling for permissions errors when handling custom Postgres configurations in database containers.

Backport 67bddb2 from #3041

NA - no customer facing changes

NA

fix: don't show current prompt in suggestions

Publish signatures for release containers.
Backport c11764be4adb377521e4067c4852953d7c434e24 from #2697

• server: runs postgresql from /var/run/postgresql instead of /run/postgresql
  Backport 36c78cdf596e1b22079512931fa526346c2ec821 from #2711

fix(rel): fix issue with Postgres docker container failing upgrades after an unclean database shutdown.
Backport ca5185ce2528b67e1ce6bcb5170f96d3699e11e0 from #2660

• Addresses errors similar to: failed to prepare workspace: failed to create ext4 filesystem in backing file: "mke2fs 1.45.5 (07-Jan-2020)\nThe file losetup: /var/lib/firecracker/loop-devices/workspace-loop-47749-3763347387: Warning: file is smaller than 512 bytes; the loop device may be useless or invisible for system tools.\n/dev/loop13 does not exist and no size was specified.\n": exit status 1
• Affects reading and using the environment variable EXECUTOR_FIRECRACKER_DISK_SPACE
  Backport 22db2a97a0393cdd233a92da7ade96cd59bb99ef from #2562

Special uppercase regex tokens are no longer inverted for repository name search. eg repo:foo\B correctly interprets \B as not at ASCII word boundary. Previously it would transform the expression into repo:foo\b.

• Adds configurable limits for file size and parsing time for syntactic indexing

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• release: nightly release pipeline is working

Sourcegraph now supports App-level GitHub App connections that aren't tied to a specific Installation ID.

feat(batches): more telemetry events

feat: extend prompt tags with pagination and increased limit

feat: editing tags on prompts

feat: management page prompt categories

feat: add tags as filter to prompts and their navigation

feat: add prompt labels to the GraphQL API

feat: calculate contributor stats

• Also validates KUBERNETES_RUN_AS_USER and KUBERNETES_RUN_AS_GROUP.
• Must be integers in the range [0, 2147483647].

Properly report symbols for TypeScript type aliases

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Narrowed down the stopwords list used in Cody context, to fix an issue where we could miss highly relevant files.

Fixes a bug in Cody context where airgapped instances would attempt to use the reranker model, which increased the context retrieval latency.

N/A

N/A

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

N/A

N/A

• refactor and fix bugs in release test

<details>

➜  sourcegraph git:(wg/rel/post-release-5-10-fixup) ✗ bazel run //testing/tools/upgradetest:sh_upgradetest_run -- all -svs 5.9.0 -mvs 5.8.0 -avs 5.8.0 -pv 5.10.2832
INFO: Analyzed target //testing/tools/upgradetest:sh_upgradetest_run (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //testing/tools/upgradetest:sh_upgradetest_run up-to-date:
  bazel-bin/testing/tools/upgradetest/sh_upgradetest_run
Aspect @@rules_rust//rust/private:clippy.bzl%rust_clippy_aspect of //testing/tools/upgradetest:sh_upgradetest_run up-to-date (nothing to build)
INFO: Elapsed time: 1.165s, Critical Path: 0.72s
INFO: 5 processes: 1 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/testing/tools/upgradetest/sh_upgradetest_run testing/tools/upgradetest/go_upgradetest_/go_upgradetest cmd/migrator/image_tarball.sh cmd/frontend/no_client_bundle.image_tarball.sh docker-images/postgresql-16/image_tarball.sh docker-images/postgresql-16-codeinsights/image_tarball.sh internal/database/_codeinsights_squashed.sql internal/database/_codeinsights_tenant_data_squashed.sql internal/database/_codeintel_squashed.sql internal/database/_codeintel_tenant_data_squashed.sql internal/database/_frontend_squashed.sql internal/database/_frontend_tenant_data_squashed.sql internal/database/_schema.codeinsights.json internal/database/_schema.codeinsights.md internal/database/_schema.codeintel.json internal/database/_schema.codeintel.md internal/database/_schema.json internal/database/_schema.md all -svs 5.9.0 -mvs 5.8.0 -avs 5.8.0 -pv 5.10.2832
Loaded image: migrator:candidate
Loaded image: frontend:candidate
Loaded image: postgresql-16:candidate
Loaded image: postgresql-16-codeinsights:candidate
👉 Upgrade test ...
Latest stable release version:  5.10.2832
Latest minor version:  5.10.0
Target version:  5.10.2832
Migrator image used to upgrade:  sourcegraph/migrator:5.10.2832
Standard Versions: [5.9.0]
Multiversion Versions: [5.8.0]
Autoupgrade Versions: [5.8.0]
auto:  5.8.0
mvu:  5.8.0
std:  5.9.0
--- 🕵️  Standard Upgrade Tests:
✅ 5.9.0 Passed -- 1m16.366600375s
--- 🕵️  Multiversion Upgrade Tests:
✅ 5.8.0 Passed -- 1m17.939247958s
--- 🕵️  Auto Upgrade Tests:
✅ 5.8.0 Passed -- 1m24.54765675s

</details>
Backport 8dc831953f43bbe9f6b9cb5467ef263b757425ac from #2374

fix: remove surplus typename

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
Backport 3f079a57978179aa2ad3f310195346a8c574f9ce from #2349

We no longer log the postgresql DSN when it changes. Previously this could contain the secret PGPASSWORD.
Backport 2bd9c09969fff536c6fb29108656658d3f875047 from #2272

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
Backport 964eceae584e4eff56df9711da68fef7047cbc40 from #2254

fix(rel): change pg_upgrade behavior to use copy instead of hard links
Backport 13caa807d96dd4b5dd872fec2a38322d1888134a from #2192

fix(rel): address issue with readiness check outputting passwords to logs.
Backport 655a739e8e791d7aab1977d2c26543a3a4871809 from #2132

Disabled an indexed search optimization which would skip files accidentally (ZOEKT_DISABLE_GOGIT_OPTIMIZATION=true).
Backport 34ada948bdcee3d75499c98f4db5c32986943e88 from #2050

Cody context now incorporates filename information in reranking, improving context quality when the reranker is enabled.

• adds systemPreInstruction to the modelConfiguration key in site admin config that allows injecting a prelude prompt into every chat request for an enterprise

• Cody Gateway: add support for OpeanAI predicted outputs

A new GraphQL query, GithubAppRepositoriesForInstallation, has been added that provides a paginated list of all the GitHub repositories that are accessible to the GitHub app with the provided installation id.

A new graphql endpoint has been added, GithubAppInstallationsForUser, that returns installation information for the global GitHub multi tenant app when running in multitenant mode.

The GitHub app authentication package now supports authenticating a GitHub app via the OAuth client ID following the announcement of https://github.blog/changelog/2024-05-01-github-apps-can-now-use-the-client-id-to-fetch-installation-tokens/

• Cody Gateway: add gpt-4o-mini to Cody Gateway allow list
• Cody Gateway: add gpt-4o-mini to DotCom models list

Feat

Dev

• Adding sg command to request Entitle bundles #1370

The Github API client's GetUserInstallations route, which lists of GitHub App installations the user has access to, now has pagination support.

A simple helper routine to the multitenantenv package that automatically populates a github app struct with the provided validated credentials.

The routing logic for multitenant mode now has a new route that uses the "last seen tenant" cookie to route github app login authorization callbacks to the appropriate tenant.

feat(rel): Add Postgresql 16 and Postgresql 16 codeinsights images to published image list.

feat(rel): Add self updating to Postgres 16 codeinsights db image.

feat(rel): Add self updating to Postgres 16 container image.

feat(rel): Add Postgres 16 CodeInsights Wolfi image

feat(rel): Add Postgres 16 Wolfi image

• refactor upgradetest
• introduce proper handling of the postgres version upgrade
  Backport 9ccdf4200e3e08cea56bffe5779ca8a6cda2909c from #1894

• support get, list, and delete workspaces to sg

A new worker job has been added that updates the database with the credentials for the global github app when running in multitenant mode.

N/A

• Added dynamic filters and the ability to aggregate by repo metadata and repo topic

• Search filters sidebar is now collapsible

• Render .mdx files as markdown

We sometimes would emit patternType:regex instead of patternType:regexp. We now always do regexp as well as treating regex as an alias for regexp.
Backport a095b39ac39cfcbe3526ecf85ed6d50cb5fa3d9d from #1808

fix(batches): titles now break over multiple lines if they are too long

Code insights should show correct line counts when enhanced language detection is turned off (note: this setting is on by default).

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

The github_apps table has removed an erroneous unique constraint that accidentally incorporated an app's slug (which is mutable) as part of the unique identifier for an app.

The logic in the new multitenant GraphQL resolvers has been reworked to use the new helper methods introduced in the Multitenant GitHub app configuration logic introduced in https://app.graphite.dev/github/pr/sourcegraph/sourcegraph/1758/.

This change reduces the risk of secrets being emitted in executor logs even if there are bugs in the secret redaction logic.

NA
Backport a943412a99852332f921b52a0ee3179dc3331d20 from #1958

fix(rel): Add vacuum to Postgres upgrade process

N/A

• add migrations to handle database drift caused during postgres 12 to postgres 16 upgrade

• Fix bug in local upgrade test preventing final stage drift check via private monorepo

• When cloning Perforce depots, Sourcegraph will now decode encoded paths correctly (paths that include characters like @ and #)

When using an instance with a Cody-only license, accessing the home page now correctly redirects you to /cody/dashboard, instead of a non-existent /cody page.
Backport a4cb5a0723bad18e1c215d81231db457d1abfbdb from #1621

feat(batches): show name of additional fields that cause errors

Long-term local retention of user telemetry as 'event logs' can now be disabled entirely via the telemetry: { disableLocalEventLogs } site configuration.

• Adds cost estimates to the ModelConfiguration returned by .api/llm/supported-models.json
• Adds llmTokenUsageCostEstimate telemetry value with estimates of each requests LLM cost in pennies

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Add the sourcegraph API client user-agent to telemetry payload types under source.server.server_client.user_agent.

• Propagates new X-Sourcegraph-API-Client-Name and X-Sourcegraph-API-Client-Version headers to request clients

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• New OpenAPI API docs within the instance at the URL $SRC_ENDPOINT/api/openapi/public. Alternatively, reach the page at "User > Settings > OpenAPI Reference".

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• The /.api/completions/stream API now includes used input/output tokens and the upstream model name in the response body when using the query parameter api-version=5 or higher. Does not apply to: Vertex Anthropic, Azure OpenAI (mo model name, and no token usage in streaming response), Gemini (no model name in streaming response)
• LLM chat completions now support stream: false when using Azure OpenAI and OpenAI-compatible providers.

• The raw HTTP API now accepts the query paramter format=cody or header Accept: application/cody to render files and directories as context items for Cody. Example request path: /github.com/sourcegraph/cody/-/raw/agent/src/index.ts?format=cody.

• add library that writes to TEST_INFRASTRUCTURE_FAILURE_FILE if it is defined for test failures

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Cody Gateway: add new claude-3-5-haiku-latest & claude-3-5-opus-latest models to Cody Gateway allow list Cody Gateway: add new claude-3-5-haiku-latest dotcom models list
Backport 11e7481ba3c810ae5b47d32b8cf32066e2f0b2bb from #1470

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

The sub repository permissions implementation now emits logs that describe the rules that were evaluated for a given request whenever tracing is enabled. (We limit these logs for only when tracing is enabled since they can be quite verbose).

• Added embeddable page for file snippets

Prevents default browser shortcuts from modifying the file content locally.

This fixes a bug where we would apply a phrase boost if a content: filter was specified with keyword search enabled. This led to inconsistent behavior (regexp vs keyword search) and it also rendered the content: filter ineffective, because we ran a general text search instead of a just a content search.

Fixes a bug where hovers were not triggered for less mainstream languages like F#, Standard ML, Visual Basic, Pkl, Hack, MATLAB etc.

Introduces new limits for auto-indexing inference to reduce the risk of continuously growing auto-indexing queues.

• The number of jobs spawned per round of auto-indexing inference per repo is capped (default: 100) to reduce risk of clogging of auto-indexing queues. Excess jobs will be discarded.
• The number of paths inspected for a single round of auto-indexing inference per repo is capped (default: 500) to reduce risk of timeouts. Excess paths will be discarded.

Fixes handling of SCIP uploads when the hash passed to the -commit flag of src-cli corresponds to the hash of an annotated tag instead of the hash of a commit. Previously, these uploads were not accessible for code navigation.

Fixes a bug where the retention tab for precise indexes would show a nil pointer exception instead of retention information.

fix(batches): enable request splitting by default

fix(batches): handle spec being nil for suffix salt

fix(batches): enable GQL request splitting via BATCH_CHANGES_REQUEST_SPLITTING

fix(batches): display "Deleted Namespace" instead of throwing an error

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

fix(code-monitor): fail monitors with missing/unverified primary email address

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Added token usage information to .api/llm/chat/completions

Fixes a bug where logs and traces were sometimes missing key-value pairs recorded alongside errors.

NA

The documentation for gitserver's ReadDir method has been clarified to reflect that the return iterator's Name() method returns the full path of the file, as opposed to just the basename.

The documentation for the "exclude" section in the github code host connection schema has been clarified to explain that each block is OR'd together, and the expressions within each block are AND'd together.

• When there are multiple auth providers configured that point to the same GitHub URL, as can be the case when using private GitHub Apps, user external account tokens will now refresh correctly.

• GitHub code host connections using repositoryQuery will no longer delete repositories if the sync fails for reasons like a GitHub outage or a token expiring.