# Degraded query performance in GCP us-central1

- Provider: [InfluxDB Cloud](https://down.guide/providers/influxdata)
- Confidence: Confirmed
- Impact: Partial outage
- lasted 2h 48m
- started 2025-11-10 21:04 UTC
- API Queries, Tasks
- us, us-central-1, us-central1
- id: DWN-73f7d8467c
- Official page: https://stspg.io/j31d7rqd8h8k

## Root cause
> Timeline Time \\(UTC\\) What happened 8:45pm The engineer-on-call was paged due to a high rate of query errors in the cluster.

[Postmortem](https://stspg.io/j31d7rqd8h8k)

## Updates from InfluxDB Cloud
### 2025-11-10 21:04 UTC: investigating
We are investigating increased query errors within the region

### 2025-11-10 22:17 UTC: investigating
We are continuing to investigate this issue.

### 2025-11-10 22:48 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2025-11-10 23:52 UTC: resolved
This incident has been resolved.

### 2025-11-25 23:19 UTC: postmortem
RCA for Cloud 2 prod01-us-central-1 query outage on Nov 10, 2025
Summary
The SRE team has been working on a long-term project to review and rebalance the storage workloads, to provide better separation of workloads within nodepools, which will improve performance, reliability and ease of maintenance. 
The Cloud 2 service is designed with a set of primary and secondary storage pods. During normal deployments, whenever a primary pod is restarted, the secondary pod takes over, so that the service is not interrupted. The change that was made in this cluster was a change to the primary and the secondary pods. When the configuration change was made in prod01-us-central-1, it caused some of the primary storage pods and their corresponding secondary pods to be restarted at the same time. While the pods were unavailable, queries failed. Once the pods restarted, the query error rate went back to normal levels and the cluster recovered. 
Timeline 
Time \\(UTC\\) What happened 
8:45pm The engineer-on-call was paged due to a high rate of query errors in the cluster. Upon investigation, the primary and secondary pods for the same storage slice are unavailable. 
8:50pm Engineer-on-call identified that the problem was due to a misconfiguration that caused the primary and secondary pods to be restarted at the same time. 
9:00pm The system will recover by itself when the pods restart with the new configuration, so no intervention is required.
9:20pm Queriers start recovering, and error rate is returning to pre-incident level. Some lingering errors in nginx. 
10:00pm Query backlog is fully cleared. The cluster has fully recovered.
Future Mitigations 
1. Reviewing procedures relating to storage pod reconfigurations 
Going forward we will ensure that configuration changes for primary pods and secondary pods are never applied at the same time.

From the builder: does your app depend on InfluxDB Cloud? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=influxdata) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention InfluxDB Cloud.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-73f7d8467c), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.