Querying issues

What to do

• Check whether your product depends on Honeycomb. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2026-07-31 15:42 UTC and 2026-07-31 14:30 UTC may need a replay.

• Subscribe to Honeycomb's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=honeycomb works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Honeycomb 2

1. resolvedJul 31, 2026, 2:30 PM UTC

   Between 10:30 and 11:00 AM EDT, we experienced elevated querying failures and slowness. The impact has subsided, and we are currently investigating and monitoring the situation.

   provider

2. postmortemAug 21, 2026, 1:18 PM UTC

   Starting at 9:50 AM Central Time on July 31st we received alerts around querying health and latency. The cause was eventually tracked down to a clash of two different factors. First, we have been introducing Lambda Managed Instances \(LMI\) into our query architecture, which behave similar to classic lambdas in many ways and had been introduced as part of work around querying performance improvements with the goal of a better and faster user experience. Unfortunately, a difference between classic lambdas and LMIs we discovered is the 15 minute hard stop of classic lambdas does not exist on LMIs, which means that certain assumptions built into our querying system no longer held. At the time of the incident, our newer Canvas feature was running batches of queries for a customer which should have a self imposed one minute time out. This timeout was not properly communicated back to the lambda, and while with classic lambdas the task would automatically cut off at 15 minutes, the new LMIs were not enforcing this sort of safety mechanism. As a result, some large queries were running for up to 30 minutes, driving up query latency across the board and resulting in errors for users trying to run fresh queries. The system recovered by 10:10 AM CT, 20 minutes later, when the large queries cleared out, but we continued to investigate the cause of the incident. Once we identified this weakness in our querying architecture we began work to fix it and prevent the same incident from occurring again. We have shipped all associated incident follow ups.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.