Grafana Cloud Metrics - Intermittent Write Latency in prod-us-central, prod-us-central-5, and prod-eu-west-0

What to do

• Check whether you use Grafana Cloud GCP Belgium - prod-eu-west-0: Ingestion, GCP US Central - prod-us-central-0: Ingestion, GCP Belgium - prod-eu-west-0: Ingestion, GCP US Central - prod-us-central-5: Ingestion in eu, eu-west-0, us, us-central-0. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 19d 22h it lasted.

  Anything that failed between 2026-02-25 19:54 UTC and 2026-03-17 18:22 UTC may need a replay.

• Subscribe to Grafana Cloud's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=grafana-cloud works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Grafana Cloud 8

1. investigatingFeb 25, 2026, 7:54 PM UTC

   Since February 19, we have been investigating an intermittent issue causing increased write latency in the prod-us-central-0 and prod-us-central-5 regions. The issue does not affect all traffic but may result in delayed write operations for some customers. Our engineering team is actively working to identify the root cause and stabilize performance. We will share additional updates as progress is made.

   provider

2. investigatingFeb 26, 2026, 4:23 PM UTC

   We have begun rolling out mitigation steps to reduce write latency in the prod-us-central-0 and prod-us-central-5 regions. While these measures are expected to improve performance, we are continuing to investigate the underlying root cause of the issue. We will provide additional updates as more information becomes available.

   provider

3. monitoringFeb 27, 2026, 2:55 PM UTC

   We've implemented mitigation in place and are continuing to monitoring and investigating this issue.

   provider

4. monitoringFeb 27, 2026, 10:05 PM UTC

   We are continuing to investigate this issue alongside the CSP. Any notable updates will continue to be shared here for tracking.

   provider

5. monitoringMar 5, 2026, 10:22 PM UTC

   We are continuing to investigate this issue alongside the CSP, and have taken steps to escalate through the appropriate channels. The mitigation in place continues to work as expected, and any notable updates will continue to be shared here for tracking.

   provider

6. monitoringMar 6, 2026, 8:53 PM UTC

   We have seen an increase in latency in our cloud providers services, and are rolling out a change to mitigate the issue. We are monitoring.

   provider

7. monitoringMar 6, 2026, 9:44 PM UTC

   We are rolling out a mitigation across the environments in these regions, and preemptively where possible to ensure it doesn’t spread elsewhere.

   provider

8. resolvedMar 17, 2026, 6:22 PM UTC

   This incident is now resolved. During the incident the Cloud Metrics platform experienced intermittent latency spikes communicating with a backend cloud service in the prod-us-central-0 and prod-us-central-5 regions. During the incident the internal CSP-facing issue was escalated to a P1. After determining the scope of the latency spikes was limited to only one availability zone, the team mitigated the situation by migrating all write traffic from to the single nearly unaffected availability zone. As the CSP service team attempted to remedy the situation, the situation became worse and began affecting the previously unaffected zone. Given this, another mitigation path was needed. Changing the connection strategy employed by Cloud Metrics to a different method was deployed to all environments, stabilizing the write path once again as we found the different connection method was more reliable and not affected by these increases in latency. We have migrated all tenants back to multi-zone write paths and are happy with and confident in the current method of connectivity to the backend cloud service, which is the one we migrated to during the course of the incident. We have no immediate plans to use the previous problematic connectivity method for the foreseeable future.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.