Slowness in US region

What to do

• Check whether you use Honeycomb ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 App Interface, ui.honeycomb.io - US1 Trigger & SLO Alerting in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 21m it lasted.

  Anything that failed between 2026-10-20 14:04 UTC and 2026-10-20 16:26 UTC may need a replay.

• Subscribe to Honeycomb's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=honeycomb works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Honeycomb 4

1. investigatingOct 20, 2026, 2:04 PM UTC

   We are currently investigating the degradation of various services in our US region.

   provider

2. investigatingOct 20, 2026, 3:18 PM UTC

   We've identified and isolated the impacted service and are seeing recovery in query execution. SLO processing remains degraded.

   provider

3. resolvedOct 20, 2026, 4:26 PM UTC

   SLO processing has fully recovered. All services are now functional.

   provider

4. postmortemOct 21, 2026, 11:26 PM UTC

   On October 20, we experienced roughly three hours of degraded service in the US region. * From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed. * From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time. * From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC \(1% error rate\) and for approximately a minute at 14:15 UTC \(1.9% error rate\). Telemetry rejected during these periods was not ingested unless the sender successfully retried it. Customers may see gaps in their data for those periods, including in triggers and SLOs that cover it. Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed. Our investigation found that the primary database’s CPU was saturated by a large number of long-running read queries. We performed a database failover to relieve the load. This briefly eased the pressure, but the same queries soon reappeared on the new primary, so it did not resolve the problem. We then reduced database load by terminating those queries and temporarily pausing SLO processing, which restored the other affected services. Once a database-side timeout was in place on the responsible query, we resumed SLO processing, which then caught up, and we repaired historical SLO data. These queries came from a check our SLO processing runs whenever a Honeycomb environment's settings change. In a vast majority of cases, it completes in a few milliseconds. For the 0.1% of our environments with an unusually large number of SLOs and calculated fields, however, the database ran it in a very inefficient way, and a routine update to one such environment's settings made the check too slow to ever complete. These queries continued running after the requesting service timed out, and repeated attempts increased the load on a database shared by several services. To prevent this from happening again, in addition to the database-side timeout, we changed how the database executes the query for environments like this one, which brought its runtime down from minutes to a few milliseconds. We have also adjusted our alerting thresholds to ensure that an on-call engineer is made aware of future database load changes quickly.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.