Sourcegraph on Kubernetes upgrade notes

This page lists the changes that are relevant for upgrading Sourcegraph on Kubernetes with Kustomize and Helm.

For upgrade procedures or general info about sourcegraph versioning see the links below:

• Kubernetes Kustomize Upgrade Procedures
• Kubernetes Helm Upgrade Procedures
• General Upgrade Info
• Technical changelog

Attention: These notes may contain relevant information about the infrastructure update such as resource requirement changes or versions of dependencies (Docker, Kubernetes, externalized databases).

If the notes indicate a patch release exists, target the highest one.

6.10.0

• Port names have been updated on services and pods, to unblock Kubernetes service meshes from auto-configuring themselves for the traffic type, since most inter-pod communication has been switched from http to gRPC [PR 756]. If you are using Istio (or similar service meshes), and our Envoy filter example, and your pods fail to communicate after upgrading to >= v6.10, please remove the Envoy filter and try again so your service mesh can auto-configure itself based on the new port names.

6.4.0

• The repo-updater service is no longer needed and will be removed from deployment methods going forward.
• The symbols service and searcher service have been merged and symbols will be removed from deployment methods going forward. Consider moving env vars set on symbols to the searcher deployment before upgrading and reallocating resources from symbols to searcher.

6.2.2553

Known issues

Customers running Sourcegraph versions prior to v6.2.2553 and using the Sourcegraph provided PostgreSQL containers may encounter PostgreSQL collation version mismatch warnings after upgrading to more recent Sourcegraph versions due to an underlying glibc version update.

When logging into the database via psql or similar tools you may see the following warning:

Warning: database "sg" has a collation version mismatch DETAIL: The database was created using collation version 2.40, but the operating system provides version 2.41.

Mismatched collation versions can lead to database index corruption if left unchecked.

Affected Services

• pgsql container
• codeintel-db container
• codeinsights-db container

Only self-hosted customers using the Sourcegraph provided PostgreSQL container images are affected.

Self-hosted customers using external databases, such as AWS RDS, GCP CloudSQL, or another self-managed solution are NOT affected.

See our PostgreSQL Collation Version Mismatch Resolution notes for more details.

6.0.0

• Sourcegraph 6.0.0 no longer supports PostgreSQL 12, admins must upgrade to PostgreSQL 16. See our postgres 12 end of life notice! As well as supporting documentation and advisements on how to upgrade.

Warning: The Kubernetes Helm deployment type does not support MVU from Sourcegraph v5.9.45 versions and earlier to Sourcegraph v6.0.0. Admins seeking to upgrade to Sourcegraph v6.0.0 should upgrade to v5.11.6271 then use the standard upgrade procedure to get to v6.0.0. This is because migrator v6.0.0 will no longer connect to Postgres 12 databases. For more info see our PostgreSQL upgrade docs.

5.10.1164

• This release resolves an issue in the v5.10.0 release which prevented multiversion upgrades from working. You may now target v5.10.1164 using migrator's upgrade command.

Note: For versions prior to 6.12.0, you could also use autoupgrade by setting the environment variable SRC_AUTOUPGRADE_IGNORE_DRIFT=true on the frontend deployment. Autoupgrade has been removed in Sourcegraph 6.12.0 and later.

5.10.0

Warning: Admins are advised to upgrade directly to v5.10.1164 circumventing this release.

Warning: This release updates the database container images from Postgres 12 to Postgres 16, and begins using Wolfi based images. Customers are advised to take a database backup before upgrading! See our postgres 12 end of life notice!

Warning: automatic and migrator upgrade command will not work for this release, please upgrade directly to v5.10.1164, or to a 5.9 version and conduct a standard upgrade using migrator's default up command!

Notes:

• The container image for pgsql and codeintel-db have been renamed from postgres-12-alpine and codeintel-db respectively to postgresql-16. The codeinsights-db container has been renamed to postgresql-16-codeinsights.
• Admins using external dbs who have not yet upgraded from postgres 12 to postgres 16, can expect to see database drift after upgrading to 5.10.0. The new expected schema definition for Sourcegraph is based on postgres 16. The schema drift is the result of automatic changes made to the schema by pg_upgrade utils, and will not cause issues in the application.
  • Admins should not run migrators suggested drift fixes, and should instead upgrade their database from postgres 12 to postgres 16.
• Postgres 12 image containers cannot be started with data volumes which have been upgraded by postgres 16.

5.2.0

Notes:

• The GitHub proxy service has been removed and is no longer required. You can safely remove it.

5.1.7

Notes:

• v5.1.7 of the deploy-sourcegraph-helm repo was initially released with the precise-code-intel worker service unable to write to /tmp. The release was overwritten, users who have not yet upgraded will be unaffected. Users who have already upgraded may ammend this issue by pulling in the fix with helm repo update and rerunning helm upgrade.

5.1.5

Notes:

• Upgrades from versions v5.0.3, v5.0.4, v5.0.5, and v5.0.6 to v5.1.5 are affected by an ordering error in the frontend databases migration tree.

4.5.0

Notes:

• Our new kustomize repo is introduced. Admins are advised to follow our migrate procedure to migrate away from our legacy deployment

  • See our note on multiversion upgrades coinciding with this migration. Admins are advised to stop at this version, migrate, and then proceed with upgrading.

• This release introduces a background job that will convert all LSIF data into SCIP. This migration is irreversible and a rollback from this version may result in loss of precise code intelligence data. Please see the migration notes for more details.

Kubernetes with Helm

• Searcher and Symbols now use StatefulSets and PVCs to avoid large ephermeralStorage requests #242
• This release updates searcher and symbols services to be headless.
  • Before upgrading, delete your searcher and symbols services (ex: kubectl delete svc/searcher svc/symbols) #250
• An env var CACHE_DIR was renamed to SYMBOLS_CACHE_DIR in sourcegraph/sourcegraph. This change was missed in the Helm charts, which caused a permissions issue during some symbols searches. For more details, see the PR to fix the env var: #258.
  • A revision to the 4.5.1 chart (4.5.1-rev.1) was released to address the above issue. Use this revision for upgrades to 4.5.1. (ex: helm upgrade --install --version 4.5.1-rev.1) #259

4.4.1

Notes:

• Users attempting a multi-version upgrade to v4.4.0 may be affected by a bug in which an outdated schema migration is included in the upgrade process. This issue is fixed in patch v4.4.2
  • The error will be encountered while running upgrade, and contains the following text: "frontend": failed to apply migration 1648115472.
    • To resolve this issue run migrator with the args 'add-log', '-db=frontend', '-version=1648115472'.
    • If migrator was stopped while running upgrade the next run of upgrade will encounter drift, this drift should be disregarded by providing migrator with the --skip-drift-check flag.

4.2.1

Notes:

• The worker-executors Service object is now included in manifests generated using kustomize. This object was already introduced in the base manifest, but omitted from manifests generated using kustomize. Its purpose is to enable ingested executor metrics to be scraped by Prometheus. It should have no impact on behavior.

• minio has been replaced with blobstore. Please see the update notes here

• This upgrade adds a node-exporter DaemonSet, which collects crucial machine-level metrics that help Sourcegraph scale your deployment.

  • Note: Similarly to cadvisor, node-exporter:
    • runs as a DaemonSet
    • needs to mount various read-only directories from the host machine (/, /proc, and /sys)
    • ideally shares the machine's PID namespace

  For more information, see deploy-sourcegraph-helm's Changelog or contact customer support.

4.0

Notes:

• jaeger-agent sidecars have been removed in favor of an OpenTelemetry Collector DaemonSet + Deployment configuration. See Configure a tracing backend section.
• Exporting traces to an external observability backend is now available. Read the documentation to configure.
• The bundled Jaeger instance is now disabled by default. It can be enabled if you do not wish to utilise your own external tracing backend.

3.41

Notes:

• The Postgres DBs frontend and codeintel-db are now given 1 hour to begin accepting connections before Kubernetes restarts the containers. #4136

3.40

Notes:

• cadvisor now defaults to run in privileged mode. This allows cadvisor to collect out of memory events happening to containers which can be used to discover underprovisoned resources. This is disabled by default in non-privileged overlay. #4126
• Updated the Nginx ingress controller to v1.2.0. Previously this image originated from quay.io, now it is pulled from the official k8s repository. A redeployment of the ingress controller may be necessary if your deployment used the manifests provided in configure/ingress-nginx. #4128
• The alpine-3.12 docker images used as init containers for some deployments have been replaced with images based on alpine-3.14. #4129

3.39

Notes:

• Thecodeinsights-db container no longer uses TimescaleDB and is now based on the standard Postgres image sourcegraph/deploy-sourcegraph#4103. Metrics scraping is also enabled.
• CAUTION: If you use a custom Code Insights postgres config, you must update the shared_preload_libraries list to remove timescaledb. The above PR demonstrates this change.

3.37

Notes:

• This release adds a new migrator initContainer to the frontend deployment to run database migrations. Confirm the environment variables on this new container match your database settings. Docs
• If performing a multiversion upgrade from an instance prior to this version see our upgrading early versions documentation

3.36

Notes:

• The backend service has been removed, so if you deploy with a method other than kubectl-apply-all.sh, a manual removal of the service may be necessary.

3.35

Notes:

• The query-runner deployment has been removed, so if you deploy with a method other than the kubectl-apply-all.sh, a manual removal of the deployment may be necessary. Follow the standard upgrade procedure to upgrade your deployment.
• There is a known issue with the Code Insights out-of-band settings migration not reaching 100% complete when encountering deleted users or organizations.

3.31

WARNING: This upgrade must originate from v3.30.3.

Notes:

• The built-in main Postgres (pgsql) and codeintel (codeintel-db) databases have switched to an alpine-based Docker image. Upon upgrading, Sourcegraph will need to re-index the entire database.

3.30

WARNING: If you have already upgraded to 3.30.0, 3.30.1, or 3.30.2 please see the rebuild corrupt postgres indexes guide for recovery steps.

Notes:

• This upgrade removes the non-root overlay, in favor of using only the non-privileged overlay for deploying Sourcegraph in secure environments. If you were previously deploying using the non-root overlay, you should now generate overlays using the non-privileged overlay.

3.29

Notes:

• This upgrade adds a new worker service that runs a number of background jobs that were previously run in the frontend service. See notes on deploying workers for additional details. Good initial values for CPU and memory resources allocated to this new service should match the frontend service.

3.28

Notes:

• All Sourcegraph images now have a registry prefix. #2901
• The memory requirements for redis-cache and redis-store have been increased by 1GB. See https://github.com/sourcegraph/deploy-sourcegraph/pull/2898 for more context.

3.27

WARNING: Sourcegraph 3.27 now requires Postgres 12+.

Notes:

WARNING: We have updated the default replicas for sourcegraph-frontend and precise-code-intel-worker to 2. If you use a custom value, make sure you do not merge the replica change.

NOTE: The Postgres 12 database migration scales with the size of your database, and the resources provided to the container. Expect to have downtime relative to the size of your database. Additionally, you must ensure that have enough storage space to accommodate the migration. A rough guide would be 2x the current on-disk database size

• If you are using an external database, upgrade your database to Postgres 12 or above prior to upgrading Sourcegraph. No action is required if you are using the supplied database images.
• If performing a multiversion upgrade from an instance prior to this version see our upgrading early versions documentation

3.25

Notes:

• Go 1.15 introduced changes to SSL/TLS connection validation which requires certificates to include a SAN. This field was not included in older certificates and clients relied on the CN field. You might see an error like x509: certificate relies on legacy Common Name field. We recommend that customers using Sourcegraph with an external database and and connecting to it using SSL/TLS check whether the certificate is up to date.
  • AWS RDS customers please reference AWS' documentation on updating the SSL/TLS certificate for steps to rotate your certificate.

3.22

Notes:

• This upgrade removes the code intel bundle manager. This service has been deprecated and all references to it have been removed.
• This upgrade also adds a MinIO container that doesn't require any custom configuration. You can find more detailed documentation in here.

3.21

Notes:

• This release introduces a second database instance, codeintel-db. If you have configured Sourcegraph with an external database, then update the CODEINTEL_PG* environment variables to point to a new external database as described in the external database documentation. Again, these must not point to the same database or the Sourcegraph instance will refuse to start.

3.19

Notes:

• WARNING: If you use an overlay that does not reference one of the provided overlays, please add - /admin/bases/pvcs as an additional base to your kustomization.yaml file. Otherwise the PVCs could be pruned if kubectl apply -prune is used.