- Level
- confirmed
- Worst impact
- Major outage
- Components
- Trello.com, API
- Down Index
- 49of 100
What to do
Check whether you use Trello Trello.com, API. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 2h 16m it lasted.
Anything that failed between 2024-02-13 09:30 UTC and 2024-02-13 11:46 UTC may need a replay.Subscribe to Trello's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=trello works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Trello 7
- investigating
We've noticed that Trello is responding slowly. This will be present in both the web and mobile apps. Our engineering team is actively investigating this incident and working to bring Trello back up to speed as quickly as possible. We'll keep you posted with further updates on this page.
provider
- investigating
Our team is still investigating the issue that's causing Trello to be slow or unavailable, and working to bring Trello back up to speed as quickly as possible. Thanks for your patience and understanding!
provider
- investigating
We're still investigating the root cause of this problem, and a new update will be shared soon! Thanks for your patience!
provider
- investigating
Our team is still looking into the issue causing Trello to be slow or unavailable. Thank you for your understanding, we're working to have Trello back up and running as quickly as possible.
provider
- monitoring
We've identified an issue that was causing slowness in Trello and put a fix in place. We're seeing things improving for all our customers and we'll keep monitoring this latest fix for now. Once again, we appreciate your understanding!
provider
- resolved
Trello is now available. Thank you for your patience.
provider
- postmortem
### Summary On Feb. 13, 2024, between 8:00 AM and 11:34 AM UTC, Trello experienced severely degraded performance appearing as a full or partial outage to Atlassian customers. The event was triggered by a buildup of long-running queries against our database, leading to slowed API response times and causing Trello to be degraded or unavailable for users. The root cause of the incident was identified as a compression change in our database deployed approximately 11 hours earlier during a low-traffic period. As European customers came online, traffic started increasing, resulting in a buildup of queries and the subsequent incident. The incident was detected by our monitoring system at 8:07 AM UTC and was mitigated by reverting the compression change and restarting components of our database system. The total time to resolution was 3 hours and 34 minutes. ### **IMPACT** The overall impact was between 8:00 AM and 11:34 AM UTC on Feb. 13, 2024. The incident caused Trello to be fully or partially unavailable for customers using or attempting to access the site during this period. ### **ROOT CAUSE** The issue was caused by a compression change in our database, which resulted in the build up of queries in the system. This build up then caused API response times to increase to critical levels. During the incident many users received HTTP 429 errors as the system began rate-limiting in an attempt to recover. Users that did not receive errors experienced API response times 10-100x slower than our standard response times. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. We are prioritizing the following actions to avoid repeating this incident and reduce time to resolution: * Improve our process for releasing incremental configuration changes which would have allowed the team to identify the root cause before a peak load period and prevent similar incidents. * Adjust priority level of alerts related to this class of incident to improve signal to noise ratio and drive faster time to resolution. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve Trello’s performance and availability. Thanks, Atlassian Customer Support
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-6b87698c74
Is Trello still down?
No, this incident was resolved 2024-02-13 11:46 UTC. For Trello's live state, see https://down.guide/trello.
How long did it last?
2h 16m, from 2024-02-13 09:30 UTC to 2024-02-13 11:46 UTC, by the times Trello posted.
What was affected?
Trello named Trello.com, API.
What caused it?
Trello hasn't published a root cause. If it does, it shows here.
Where does this come from?
Trello's official status page, https://trello.status.atlassian.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.