Application and API errors

What to do

• Check whether you use Asana App, App, App, App and 16 more. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 28m it lasted.

  Anything that failed between 2024-08-01 15:21 UTC and 2024-08-01 19:49 UTC may need a replay.

• Subscribe to Asana's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=asana works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Asana 12

1. identifiedAug 1, 2024, 3:21 PM UTC

   We are investigating application issues related to overload, and will provide updates.

   provider

2. identifiedAug 1, 2024, 4:04 PM UTC

   We are continuing to work on a fix for this issue.

   provider

3. identifiedAug 1, 2024, 4:34 PM UTC

   We are continuing to work on a fix for this issue.

   provider

4. identifiedAug 1, 2024, 4:37 PM UTC

   We are continuing to work on a fix for this issue.

   provider

5. identifiedAug 1, 2024, 4:58 PM UTC

   We are continuing to work on a fix for this issue.

   provider

6. identifiedAug 1, 2024, 5:03 PM UTC

   We are continuing to work on a fix for this issue.

   provider

7. identifiedAug 1, 2024, 5:05 PM UTC

   We are continuing to work on a fix for this issue.

   provider

8. identifiedAug 1, 2024, 5:33 PM UTC

   We are continuing to work on a fix for this issue.

   provider

9. identifiedAug 1, 2024, 6:13 PM UTC

   We are continuing to work on a fix for this issue.

   provider

10. monitoringAug 1, 2024, 6:35 PM UTC

    We've identified the root cause and are monitoring our systems

    provider

11. resolvedAug 1, 2024, 7:49 PM UTC

    This incident has been resolved.

    provider

12. postmortemAug 2, 2024, 12:16 AM UTC

    **Incident:** On 2024-08-01 at 14:09 UTC, an increased connection rate on a central database saturated the database’s ability to accept new connections. As a result, Asana became unavailable to users in all regions. In response, Asana engineers throttled traffic for automations, webhooks, and API, and were able to restore primary functionality of the Asana application at 15:01 UTC. The application continued to run with sporadic unavailability until 18:22 UTC, when the underlying cause of the increased connections was determined and disabled. At this point the application was fully operational, but asynchronous workloads such as automations and notifications were delayed due to the work enqueued while traffic was throttled. The incident was resolved when asynchronous work queues fully caught up at 19:39 UTC. **Impact:** From 14:09 UTC to 15:01 UTC, users were unable to access Asana through the web, mobile, or API. From 15:01 UTC to 18:22 UTC users could access Asana through the web and mobile, but experienced sporadic unavailability. Access to the API was restored gradually between 15:15 UTC and 15:58 UTC. From 15:01 UTC to 19:39 UTC users experienced delayed automations, background actions, webhooks, and event streams. No customer data was lost. **Moving forward:** We have reverted the change that caused the increased connections. We are adding monitoring to detect changes in database connection rates, so we can identify potential problems earlier. Beyond this, we will be replicating data from the central database to other databases to reduce the impact of any single database being overloaded.

    provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.