Alerts reference

This document contains a complete reference of all alerts in Sourcegraph's monitoring, and next steps for when you find alerts that are firing. If your alert isn't mentioned here, or if the next steps don't help, ontact us](mailto:[email protected]) for assistance.

To learn more about Sourcegraph's alerting and how to set up alerts, see our alerting guide.

frontend: 99th_percentile_search_request_duration

99th percentile successful search request duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_request_duration"
]

frontend: 90th_percentile_search_request_duration

90th percentile successful search request duration over 5m

Descriptions

• warning frontend: 15s+ 90th percentile successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_request_duration"
]

frontend: hard_timeout_search_responses

hard timeout search responses every 5m

Descriptions

• warning frontend: 2%+ hard timeout search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_responses"
]

frontend: hard_error_search_responses

hard error search responses every 5m

Descriptions

• warning frontend: 2%+ hard error search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_responses"
]

frontend: partial_timeout_search_responses

partial timeout search responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_responses"
]

frontend: search_alert_user_suggestions

search alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search alert user suggestions shown every 5m for 15m0s

Next steps

• This indicates your user`s are making syntax errors or similar user errors.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_search_alert_user_suggestions"
]

frontend: page_load_latency

90th percentile page load latency over all routes over 10m

Descriptions

• warning frontend: 2s+ 90th percentile page load latency over all routes over 10m

Next steps

• Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels.
• Investigate potential sources of latency by selecting Explore and modifying the sum by(le) section to include additional labels: for example, sum by(le, job) or sum by (le, instance).
• Trace a request to see what the slowest part is: https://sourcegraph.com/docs/admin/observability/tracing
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_page_load_latency"
]

frontend: 99th_percentile_search_codeintel_request_duration

99th percentile code-intel successful search request duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile code-intel successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_codeintel_request_duration"
]

frontend: 90th_percentile_search_codeintel_request_duration

90th percentile code-intel successful search request duration over 5m

Descriptions

• warning frontend: 15s+ 90th percentile code-intel successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_codeintel_request_duration"
]

frontend: hard_timeout_search_codeintel_responses

hard timeout search code-intel responses every 5m

Descriptions

• warning frontend: 2%+ hard timeout search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_codeintel_responses"
]

frontend: hard_error_search_codeintel_responses

hard error search code-intel responses every 5m

Descriptions

• warning frontend: 2%+ hard error search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_codeintel_responses"
]

frontend: partial_timeout_search_codeintel_responses

partial timeout search code-intel responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_codeintel_responses"
]

frontend: search_codeintel_alert_user_suggestions

search code-intel alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search code-intel alert user suggestions shown every 5m for 15m0s

Next steps

• This indicates a bug in Sourcegraph, please open an issue.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_search_codeintel_alert_user_suggestions"
]

frontend: 99th_percentile_search_api_request_duration

99th percentile successful search API request duration over 5m

Descriptions

• warning frontend: 50s+ 99th percentile successful search API request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_api_request_duration"
]

frontend: 90th_percentile_search_api_request_duration

90th percentile successful search API request duration over 5m

Descriptions

• warning frontend: 40s+ 90th percentile successful search API request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_api_request_duration"
]

frontend: hard_error_search_api_responses

hard error search API responses every 5m

Descriptions

• warning frontend: 2%+ hard error search API responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_api_responses"
]

frontend: partial_timeout_search_api_responses

partial timeout search API responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search API responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_api_responses"
]

frontend: search_api_alert_user_suggestions

search API alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search API alert user suggestions shown every 5m

Next steps

• This indicates your user`s search API requests have syntax errors or a similar user error. Check the responses the API sends back for an explanation.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_search_api_alert_user_suggestions"
]

frontend: frontend_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "frontend" instances)

Descriptions

• critical frontend: 300s+ maximum duration since last successful site configuration update (all "frontend" instances)

Next steps

• This indicates that one or more "frontend" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "frontend" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_frontend_frontend_site_configuration_duration_since_last_successful_update_by_instance"
]

frontend: internal_indexed_search_error_responses

internal indexed search error responses every 5m

Descriptions

• warning frontend: 5%+ internal indexed search error responses every 5m for 15m0s

Next steps

• Check the Zoekt Web Server dashboard for indications it might be unhealthy.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_internal_indexed_search_error_responses"
]

frontend: internal_unindexed_search_error_responses

internal unindexed search error responses every 5m

Descriptions

• warning frontend: 5%+ internal unindexed search error responses every 5m for 15m0s

Next steps

• Check the Searcher dashboard for indications it might be unhealthy.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_internal_unindexed_search_error_responses"
]

frontend: 99th_percentile_gitserver_duration

99th percentile successful gitserver query duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile successful gitserver query duration over 5m

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_gitserver_duration"
]

frontend: gitserver_error_responses

gitserver error responses every 5m

Descriptions

• warning frontend: 5%+ gitserver error responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_gitserver_error_responses"
]

frontend: observability_test_alert_warning

warning test alert metric

Descriptions

• warning frontend: 1+ warning test alert metric

Next steps

• This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_observability_test_alert_warning"
]

frontend: observability_test_alert_critical

critical test alert metric

Descriptions

• critical frontend: 1+ critical test alert metric

Next steps

• This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_frontend_observability_test_alert_critical"
]

frontend: cloudkms_cryptographic_requests

cryptographic requests to Cloud KMS every 1m

Descriptions

• warning frontend: 15000+ cryptographic requests to Cloud KMS every 1m for 5m0s
• critical frontend: 30000+ cryptographic requests to Cloud KMS every 1m for 5m0s

Next steps

• Revert recent commits that cause extensive listing from "external_services" and/or "user_external_accounts" tables.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_cloudkms_cryptographic_requests",
  "critical_frontend_cloudkms_cryptographic_requests"
]

frontend: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning frontend: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical frontend: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_mean_blocked_seconds_per_conn_request",
  "critical_frontend_mean_blocked_seconds_per_conn_request"
]

frontend: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning frontend: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_container_cpu_usage"
]

frontend: container_memory_usage

container memory usage by instance

Descriptions

• warning frontend: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_container_memory_usage"
]

frontend: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning frontend: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_long_term"
]

frontend: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning frontend: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
• Docker Compose: Consider increasing memory: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_long_term"
]

frontend: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning frontend: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_short_term"
]

frontend: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning frontend: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_short_term"
]

frontend: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning frontend: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_container_oomkill_events_total"
]

frontend: go_goroutines

maximum active goroutines

Descriptions

• warning frontend: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_go_goroutines"
]

frontend: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning frontend: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_go_gc_duration_seconds"
]

frontend: pods_available_percentage

percentage pods available

Descriptions

• critical frontend: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod (frontend\|sourcegraph-frontend) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (frontend\|sourcegraph-frontend).
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_frontend_pods_available_percentage"
]

frontend: email_delivery_failures

email delivery failure rate over 30 minutes

Descriptions

• warning frontend: 0%+ email delivery failure rate over 30 minutes
• critical frontend: 10%+ email delivery failure rate over 30 minutes

Next steps

• Check your SMTP configuration in site configuration.
• Check sourcegraph-frontend logs for more detailed error messages.
• Check your SMTP provider for more detailed error messages.
• Use sum(increase(src_email_send{success="false"}[30m])) to check the raw count of delivery failures.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_email_delivery_failures",
  "critical_frontend_email_delivery_failures"
]

frontend: mean_successful_sentinel_duration_over_2h

mean successful sentinel search duration over 2h

Descriptions

• warning frontend: 5s+ mean successful sentinel search duration over 2h for 15m0s
• critical frontend: 8s+ mean successful sentinel search duration over 2h for 30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_mean_successful_sentinel_duration_over_2h",
  "critical_frontend_mean_successful_sentinel_duration_over_2h"
]

frontend: mean_sentinel_stream_latency_over_2h

mean successful sentinel stream latency over 2h

Descriptions

• warning frontend: 2s+ mean successful sentinel stream latency over 2h for 15m0s
• critical frontend: 3s+ mean successful sentinel stream latency over 2h for 30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_mean_sentinel_stream_latency_over_2h",
  "critical_frontend_mean_sentinel_stream_latency_over_2h"
]

frontend: 90th_percentile_successful_sentinel_duration_over_2h

90th percentile successful sentinel search duration over 2h

Descriptions

• warning frontend: 5s+ 90th percentile successful sentinel search duration over 2h for 15m0s
• critical frontend: 10s+ 90th percentile successful sentinel search duration over 2h for 3h30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_successful_sentinel_duration_over_2h",
  "critical_frontend_90th_percentile_successful_sentinel_duration_over_2h"
]

frontend: 90th_percentile_sentinel_stream_latency_over_2h

90th percentile successful sentinel stream latency over 2h

Descriptions

• warning frontend: 4s+ 90th percentile successful sentinel stream latency over 2h for 15m0s
• critical frontend: 6s+ 90th percentile successful sentinel stream latency over 2h for 3h30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_sentinel_stream_latency_over_2h",
  "critical_frontend_90th_percentile_sentinel_stream_latency_over_2h"
]

gitserver: cpu_throttling_time

container CPU throttling time %

Descriptions

• warning gitserver: 75%+ container CPU throttling time % for 2m0s
• critical gitserver: 90%+ container CPU throttling time % for 5m0s

Next steps

• • Consider increasing the CPU limit for the container.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_cpu_throttling_time",
  "critical_gitserver_cpu_throttling_time"
]

gitserver: disk_space_remaining

disk space remaining

Descriptions

• warning gitserver: less than 15% disk space remaining
• critical gitserver: less than 10% disk space remaining for 10m0s

Next steps

• On a warning alert, you may want to provision more disk space: Disk pressure may result in decreased performance, users having to wait for repositories to clone, etc.
• On a critical alert, you need to provision more disk space. Running out of disk space will result in decreased performance, or complete service outage.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_disk_space_remaining",
  "critical_gitserver_disk_space_remaining"
]

gitserver: running_git_commands

git commands running on each gitserver instance

Descriptions

• warning gitserver: 50+ git commands running on each gitserver instance for 2m0s
• critical gitserver: 100+ git commands running on each gitserver instance for 5m0s

Next steps

• Check if the problem may be an intermittent and temporary peak using the "Container monitoring" section at the bottom of the Git Server dashboard.
• Single container deployments: Consider upgrading to a Docker Compose deployment which offers better scalability and resource isolation.
• Kubernetes and Docker Compose: Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the Sourcegraph resource estimator.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_running_git_commands",
  "critical_gitserver_running_git_commands"
]

gitserver: echo_command_duration_test

echo test command duration

Descriptions

• warning gitserver: 0.02s+ echo test command duration for 30s
• critical gitserver: 1s+ echo test command duration for 1m0s

Next steps

• Single container deployments: Upgrade to a Docker Compose deployment which offers better scalability and resource isolation.
• Kubernetes and Docker Compose: Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the Sourcegraph resource estimator.
• If your persistent volume is slow, you may want to provision more IOPS, usually by increasing the volume size.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_echo_command_duration_test",
  "critical_gitserver_echo_command_duration_test"
]

gitserver: repo_corrupted

number of times a repo corruption has been identified

Descriptions

• critical gitserver: 0+ number of times a repo corruption has been identified

Next steps

• Check the corruption logs for details. gitserver_repos.corruption_logs contains more information.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_gitserver_repo_corrupted"
]

gitserver: repository_clone_queue_size

repository clone queue size

Descriptions

• warning gitserver: 25+ repository clone queue size

Next steps

• If you just added several repositories, the warning may be expected.
• Check which repositories need cloning, by visiting e.g. https://sourcegraph.example.com/site-admin/repositories?filter=not-cloned
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_repository_clone_queue_size"
]

gitserver: gitserver_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "gitserver" instances)

Descriptions

• critical gitserver: 300s+ maximum duration since last successful site configuration update (all "gitserver" instances)

Next steps

• This indicates that one or more "gitserver" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "gitserver" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_gitserver_gitserver_site_configuration_duration_since_last_successful_update_by_instance"
]

gitserver: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning gitserver: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical gitserver: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_mean_blocked_seconds_per_conn_request",
  "critical_gitserver_mean_blocked_seconds_per_conn_request"
]

gitserver: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning gitserver: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_container_cpu_usage"
]

gitserver: container_memory_usage

container memory usage by instance

Descriptions

• warning gitserver: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_container_memory_usage"
]

gitserver: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning gitserver: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the gitserver service.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_long_term"
]

gitserver: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning gitserver: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_short_term"
]

gitserver: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning gitserver: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of gitserver container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_container_oomkill_events_total"
]

gitserver: go_goroutines

maximum active goroutines

Descriptions

• warning gitserver: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_go_goroutines"
]

gitserver: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning gitserver: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_gitserver_go_gc_duration_seconds"
]

gitserver: pods_available_percentage

percentage pods available

Descriptions

• critical gitserver: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod gitserver (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p gitserver.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_gitserver_pods_available_percentage"
]

postgres: connections

active connections

Descriptions

• warning postgres: less than 5 active connections for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_connections"
]

postgres: usage_connections_percentage

connection in use

Descriptions

• warning postgres: 80%+ connection in use for 5m0s
• critical postgres: 100%+ connection in use for 5m0s

Next steps

• Consider increasing max_connections of the database instance, learn more
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_usage_connections_percentage",
  "critical_postgres_usage_connections_percentage"
]

postgres: transaction_durations

maximum transaction durations

Descriptions

• warning postgres: 0.3s+ maximum transaction durations for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_transaction_durations"
]

postgres: postgres_up

database availability

Descriptions

• critical postgres: less than 0 database availability for 5m0s

Next steps

• Kubernetes:
  • Determine if the pod was OOM killed using kubectl describe pod (pgsql\|codeintel-db\|codeinsights) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
  • Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (pgsql\|codeintel-db\|codeinsights).
  • Check if there is any OOMKILL event using the provisioning panels
  • Check kernel logs using dmesg for OOMKILL events on worker nodes
• Docker Compose:
  • Determine if the pod was OOM killed using docker inspect -f '\{\{json .State\}\}' (pgsql\|codeintel-db\|codeinsights) (look for "OOMKilled":true) and, if so, consider increasing the memory limit of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
  • Check the logs before the container restarted to see if there are panic: messages or similar using docker logs (pgsql\|codeintel-db\|codeinsights) (note this will include logs from the previous and currently running container).
  • Check if there is any OOMKILL event using the provisioning panels
  • Check kernel logs using dmesg for OOMKILL events
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_postgres_postgres_up"
]

postgres: invalid_indexes

invalid indexes (unusable by the query planner)

Descriptions

• critical postgres: 1+ invalid indexes (unusable by the query planner)

Next steps

• Drop and re-create the invalid trigger - please contact Sourcegraph to supply the trigger definition.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_postgres_invalid_indexes"
]

postgres: pg_exporter_err

errors scraping postgres exporter

Descriptions

• warning postgres: 1+ errors scraping postgres exporter for 5m0s

Next steps

• Ensure the Postgres exporter can access the Postgres database. Also, check the Postgres exporter logs for errors.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_pg_exporter_err"
]

postgres: migration_in_progress

active schema migration

Descriptions

• critical postgres: 1+ active schema migration for 5m0s

Next steps

• The database migration has been in progress for 5 or more minutes - please contact Sourcegraph if this persists.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_postgres_migration_in_progress"
]

postgres: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning postgres: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights) service.
• Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_long_term"
]

postgres: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning postgres: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights) service.
• Docker Compose: Consider increasing memory: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_long_term"
]

postgres: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning postgres: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_short_term"
]

postgres: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning postgres: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_short_term"
]

postgres: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning postgres: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_postgres_container_oomkill_events_total"
]

postgres: pods_available_percentage

percentage pods available

Descriptions

• critical postgres: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod (pgsql\|codeintel-db\|codeinsights) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (pgsql\|codeintel-db\|codeinsights).
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_postgres_pods_available_percentage"
]

precise-code-intel-worker: codeintel_upload_queued_max_age

unprocessed upload record queue longest time in queue

Descriptions

• warning precise-code-intel-worker: 18000s+ unprocessed upload record queue longest time in queue

Next steps

• An alert here could be indicative of a few things: an upload surfacing a pathological performance characteristic, precise-code-intel-worker being underprovisioned for the required upload processing throughput, or a higher replica count being required for the volume of uploads.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_codeintel_upload_queued_max_age"
]

precise-code-intel-worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning precise-code-intel-worker: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical precise-code-intel-worker: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_mean_blocked_seconds_per_conn_request",
  "critical_precise-code-intel-worker_mean_blocked_seconds_per_conn_request"
]

precise-code-intel-worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_cpu_usage"
]

precise-code-intel-worker: container_memory_usage

container memory usage by instance

Descriptions

• warning precise-code-intel-worker: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_memory_usage"
]

precise-code-intel-worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the precise-code-intel-worker service.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_long_term"
]

precise-code-intel-worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning precise-code-intel-worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the precise-code-intel-worker service.
• Docker Compose: Consider increasing memory: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_long_term"
]

precise-code-intel-worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_short_term"
]

precise-code-intel-worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning precise-code-intel-worker: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_short_term"
]

precise-code-intel-worker: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning precise-code-intel-worker: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_oomkill_events_total"
]

precise-code-intel-worker: go_goroutines

maximum active goroutines

Descriptions

• warning precise-code-intel-worker: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_goroutines"
]

precise-code-intel-worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning precise-code-intel-worker: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_gc_duration_seconds"
]

precise-code-intel-worker: pods_available_percentage

percentage pods available

Descriptions

• critical precise-code-intel-worker: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod precise-code-intel-worker (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p precise-code-intel-worker.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_precise-code-intel-worker_pods_available_percentage"
]

redis: redis-store_up

redis-store availability

Descriptions

• critical redis: less than 1 redis-store availability for 10s

Next steps

• Ensure redis-store is running
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_redis_redis-store_up"
]

redis: redis-cache_up

redis-cache availability

Descriptions

• critical redis: less than 1 redis-cache availability for 10s

Next steps

• Ensure redis-cache is running
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_redis_redis-cache_up"
]

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-cache service.
• Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-cache service.
• Docker Compose: Consider increasing memory: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning redis: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

redis: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning redis: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-cache container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_container_oomkill_events_total"
]

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-store service.
• Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-store service.
• Docker Compose: Consider increasing memory: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning redis: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

redis: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning redis: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-store container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_redis_container_oomkill_events_total"
]

redis: pods_available_percentage

percentage pods available

Descriptions

• critical redis: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod redis-cache (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p redis-cache.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

redis: pods_available_percentage

percentage pods available

Descriptions

• critical redis: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod redis-store (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p redis-store.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

worker: worker_job_codeintel-upload-janitor_count

number of worker instances running the codeintel-upload-janitor job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-upload-janitor" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-upload-janitor"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-upload-janitor_count",
  "critical_worker_worker_job_codeintel-upload-janitor_count"
]

worker: worker_job_codeintel-commitgraph-updater_count

number of worker instances running the codeintel-commitgraph-updater job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-commitgraph-updater" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-commitgraph-updater"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-commitgraph-updater_count",
  "critical_worker_worker_job_codeintel-commitgraph-updater_count"
]

worker: worker_job_codeintel-autoindexing-scheduler_count

number of worker instances running the codeintel-autoindexing-scheduler job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-autoindexing-scheduler" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-autoindexing-scheduler"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-autoindexing-scheduler_count",
  "critical_worker_worker_job_codeintel-autoindexing-scheduler_count"
]

worker: codeintel_commit_graph_queued_max_age

repository queue longest time in queue

Descriptions

• warning worker: 3600s+ repository queue longest time in queue

Next steps

• An alert here is generally indicative of either underprovisioned worker instance(s) and/or an underprovisioned main postgres instance.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_codeintel_commit_graph_queued_max_age"
]

worker: perms_syncer_outdated_perms

number of entities with outdated permissions

Descriptions

• warning worker: 100+ number of entities with outdated permissions for 5m0s

Next steps

• Enabled permissions for the first time: Wait for few minutes and see if the number goes down.
• Otherwise: Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_perms_syncer_outdated_perms"
]

worker: perms_syncer_sync_duration

95th permissions sync duration

Descriptions

• warning worker: 30s+ 95th permissions sync duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_perms_syncer_sync_duration"
]

worker: perms_syncer_sync_errors

permissions sync error rate

Descriptions

• critical worker: 1+ permissions sync error rate for 1m0s

Next steps

• Check the network connectivity the Sourcegraph and the code host.
• Check if API rate limit quota is exhausted on the code host.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_worker_perms_syncer_sync_errors"
]

worker: insights_queue_unutilized_size

insights queue size that is not utilized (not processing)

Descriptions

• warning worker: 0+ insights queue size that is not utilized (not processing) for 30m0s

Next steps

• Verify code insights worker job has successfully started. Restart worker service and monitoring startup logs, looking for worker panics.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_insights_queue_unutilized_size"
]

worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning worker: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical worker: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_mean_blocked_seconds_per_conn_request",
  "critical_worker_mean_blocked_seconds_per_conn_request"
]

worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning worker: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_container_cpu_usage"
]

worker: container_memory_usage

container memory usage by instance

Descriptions

• warning worker: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_container_memory_usage"
]

worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the worker service.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_long_term"
]

worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the worker service.
• Docker Compose: Consider increasing memory: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_long_term"
]

worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_short_term"
]

worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning worker: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_short_term"
]

worker: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning worker: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_container_oomkill_events_total"
]

worker: go_goroutines

maximum active goroutines

Descriptions

• warning worker: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_go_goroutines"
]

worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning worker: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_worker_go_gc_duration_seconds"
]

worker: pods_available_percentage

percentage pods available

Descriptions

• critical worker: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod worker (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p worker.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_worker_pods_available_percentage"
]

worker: worker_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "worker" instances)

Descriptions

• critical worker: 300s+ maximum duration since last successful site configuration update (all "worker" instances)

Next steps

• This indicates that one or more "worker" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "worker" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_worker_worker_site_configuration_duration_since_last_successful_update_by_instance"
]

repo-updater: src_repoupdater_max_sync_backoff

time since oldest sync

Descriptions

• critical repo-updater: 32400s+ time since oldest sync for 10m0s

Next steps

• An alert here indicates that no code host connections have synced in at least 9h0m0s. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
• Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
• Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
• Check the repo-updater logs for errors about syncing.
• Confirm that outbound network connections are allowed where repo-updater is deployed.
• Check back in an hour to see if the issue has resolved itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_max_sync_backoff"
]

repo-updater: src_repoupdater_syncer_sync_errors_total

site level external service sync error rate

Descriptions

• warning repo-updater: 0.5+ site level external service sync error rate for 10m0s
• critical repo-updater: 1+ site level external service sync error rate for 10m0s

Next steps

• An alert here indicates errors syncing site level repo metadata with code hosts. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
• Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
• Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
• Check the repo-updater logs for errors about syncing.
• Confirm that outbound network connections are allowed where repo-updater is deployed.
• Check back in an hour to see if the issue has resolved itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_syncer_sync_errors_total",
  "critical_repo-updater_src_repoupdater_syncer_sync_errors_total"
]

repo-updater: syncer_sync_start

repo metadata sync was started

Descriptions

• warning repo-updater: less than 0 repo metadata sync was started for 9h0m0s

Next steps

• Check repo-updater logs for errors.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_start"
]

repo-updater: syncer_sync_duration

95th repositories sync duration

Descriptions

• warning repo-updater: 30s+ 95th repositories sync duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_duration"
]

repo-updater: source_duration

95th repositories source duration

Descriptions

• warning repo-updater: 30s+ 95th repositories source duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_source_duration"
]

repo-updater: syncer_synced_repos

repositories synced

Descriptions

• warning repo-updater: less than 0 repositories synced for 9h0m0s

Next steps

• Check network connectivity to code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_syncer_synced_repos"
]

repo-updater: sourced_repos

repositories sourced

Descriptions

• warning repo-updater: less than 0 repositories sourced for 9h0m0s

Next steps

• Check network connectivity to code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_sourced_repos"
]

repo-updater: purge_failed

repositories purge failed

Descriptions

• warning repo-updater: 0+ repositories purge failed for 5m0s

Next steps

• Check repo-updater`s connectivity with gitserver and gitserver logs
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_purge_failed"
]

repo-updater: sched_auto_fetch

repositories scheduled due to hitting a deadline

Descriptions

• warning repo-updater: less than 0 repositories scheduled due to hitting a deadline for 9h0m0s

Next steps

• Check repo-updater logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_sched_auto_fetch"
]

repo-updater: sched_known_repos

repositories managed by the scheduler

Descriptions

• warning repo-updater: less than 0 repositories managed by the scheduler for 10m0s

Next steps

• Check repo-updater logs. This is expected to fire if there are no user added code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_sched_known_repos"
]

repo-updater: sched_update_queue_length

rate of growth of update queue length over 5 minutes

Descriptions

• critical repo-updater: 0+ rate of growth of update queue length over 5 minutes for 2h0m0s

Next steps

• Check repo-updater logs for indications that the queue is not being processed. The queue length should trend downwards over time as items are sent to GitServer
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_sched_update_queue_length"
]

repo-updater: sched_loops

scheduler loops

Descriptions

• warning repo-updater: less than 0 scheduler loops for 9h0m0s

Next steps

• Check repo-updater logs for errors. This is expected to fire if there are no user added code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_sched_loops"
]

repo-updater: src_repoupdater_stale_repos

repos that haven't been fetched in more than 8 hours

Descriptions

• warning repo-updater: 1+ repos that haven't been fetched in more than 8 hours for 25m0s

Next steps

• Check repo-updater logs for errors. Check for rows in gitserver_repos where LastError is not an empty string.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_stale_repos"
]

repo-updater: sched_error

repositories schedule error rate

Descriptions

• critical repo-updater: 1+ repositories schedule error rate for 25m0s

Next steps

• Check repo-updater logs for errors
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_sched_error"
]

repo-updater: src_repoupdater_external_services_total

the total number of external services

Descriptions

• critical repo-updater: 20000+ the total number of external services for 1h0m0s

Next steps

• Check for spikes in external services, could be abuse
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_external_services_total"
]

repo-updater: repoupdater_queued_sync_jobs_total

the total number of queued sync jobs

Descriptions

• warning repo-updater: 100+ the total number of queued sync jobs for 1h0m0s

Next steps

• Check if jobs are failing to sync: "SELECT * FROM external_service_sync_jobs WHERE state = errored";
• Increase the number of workers using the repoConcurrentExternalServiceSyncers site config.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_queued_sync_jobs_total"
]

repo-updater: repoupdater_completed_sync_jobs_total

the total number of completed sync jobs

Descriptions

• warning repo-updater: 100000+ the total number of completed sync jobs for 1h0m0s

Next steps

• Check repo-updater logs. Jobs older than 1 day should have been removed.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_completed_sync_jobs_total"
]

repo-updater: repoupdater_errored_sync_jobs_percentage

the percentage of external services that have failed their most recent sync

Descriptions

• warning repo-updater: 10%+ the percentage of external services that have failed their most recent sync for 1h0m0s

Next steps

• Check repo-updater logs. Check code host connectivity
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_errored_sync_jobs_percentage"
]

repo-updater: github_graphql_rate_limit_remaining

remaining calls to GitHub graphql API before hitting the rate limit

Descriptions

• warning repo-updater: less than 250 remaining calls to GitHub graphql API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_github_graphql_rate_limit_remaining"
]

repo-updater: github_rest_rate_limit_remaining

remaining calls to GitHub rest API before hitting the rate limit

Descriptions

• warning repo-updater: less than 250 remaining calls to GitHub rest API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_github_rest_rate_limit_remaining"
]

repo-updater: github_search_rate_limit_remaining

remaining calls to GitHub search API before hitting the rate limit

Descriptions

• warning repo-updater: less than 5 remaining calls to GitHub search API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_github_search_rate_limit_remaining"
]

repo-updater: gitlab_rest_rate_limit_remaining

remaining calls to GitLab rest API before hitting the rate limit

Descriptions

• critical repo-updater: less than 30 remaining calls to GitLab rest API before hitting the rate limit

Next steps

• Try restarting the pod to get a different public IP.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_gitlab_rest_rate_limit_remaining"
]

repo-updater: repo_updater_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "repo_updater" instances)

Descriptions

• critical repo-updater: 300s+ maximum duration since last successful site configuration update (all "repo_updater" instances)

Next steps

• This indicates that one or more "repo_updater" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "repo_updater" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_repo_updater_site_configuration_duration_since_last_successful_update_by_instance"
]

repo-updater: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning repo-updater: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical repo-updater: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_mean_blocked_seconds_per_conn_request",
  "critical_repo-updater_mean_blocked_seconds_per_conn_request"
]

repo-updater: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning repo-updater: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_container_cpu_usage"
]

repo-updater: container_memory_usage

container memory usage by instance

Descriptions

• critical repo-updater: 90%+ container memory usage by instance for 10m0s

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_container_memory_usage"
]

repo-updater: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning repo-updater: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the repo-updater service.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_long_term"
]

repo-updater: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning repo-updater: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the repo-updater service.
• Docker Compose: Consider increasing memory: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_long_term"
]

repo-updater: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning repo-updater: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_short_term"
]

repo-updater: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning repo-updater: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_short_term"
]

repo-updater: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning repo-updater: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_container_oomkill_events_total"
]

repo-updater: go_goroutines

maximum active goroutines

Descriptions

• warning repo-updater: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_go_goroutines"
]

repo-updater: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning repo-updater: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_repo-updater_go_gc_duration_seconds"
]

repo-updater: pods_available_percentage

percentage pods available

Descriptions

• critical repo-updater: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod repo-updater (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p repo-updater.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "critical_repo-updater_pods_available_percentage"
]

searcher: replica_traffic

requests per second per replica over 10m

Descriptions

• warning searcher: 5+ requests per second per replica over 10m

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

JSON
"observability.silenceAlerts": [
  "warning_searcher_replica_traffic"
]

searcher: unindexed_search_request_errors