- Level
- confirmed
- Worst impact
- Degraded
- Components
- Incident Flow, Incident Flow, Alert Flow, Alert Flow, Email Notification Delivery, Email Notification Delivery, Public Website, SMS Notification Delivery, SMS Notification Delivery, Voice Notification Delivery, Voice Notification Delivery, Mobile Notification Delivery, Heartbeat Monitoring, Heartbeat Monitoring, Incident REST API, Incident REST API, Alert REST API, Alert REST API, Heartbeat REST API, Heartbeat REST API, Incoming Email Service, Incoming Email Service, Incoming Integration Flow, Incoming Integration Flow, Outgoing Integration Flow, Outgoing Integration Flow, Signup, Login & Authorization, Signup, Login & Authorization, Opsgenie Actions, Opsgenie Actions, Web Application, Web Application, Mobile Application, Mobile Application, Configuration REST APIs, Configuration REST APIs, Reporting & Analytics, Reporting & Analytics, Pricing & Billing, Pricing & Billing, Logs, Logs, Incoming Call Routing
- Down Index
- 37of 100
What to do
Check whether you use Opsgenie Incident Flow, Incident Flow, Alert Flow, Alert Flow and 39 more. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 1h 35m it lasted.
Anything that failed between 2024-02-14 21:57 UTC and 2024-02-14 23:32 UTC may need a replay.Subscribe to Opsgenie's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=opsgenie works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Opsgenie 5
- investigating
We are investigating reports of intermittent errors for all Cloud Customers across all Atlassian products. We will provide more details once we identify the root cause.
provider
- identified
We have identified the root cause of the increased errors and have mitigated the problem. We continue to work on resolving the issue and monitoring this closely.
provider
- monitoring
We have identified the root cause of the Service Disruptions affecting all Atlassian products and have mitigated the problem. We are now monitoring this closely.
provider
- resolved
We experienced increased errors on Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, Compass, and Atlassian Analytics. The issue has been resolved and the services are operating normally.
provider
- postmortem
### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-3b0e824a81
Is Opsgenie still down?
No, this incident was resolved 2024-02-14 23:32 UTC. For Opsgenie's live state, see https://down.guide/opsgenie.
How long did it last?
1h 35m, from 2024-02-14 21:57 UTC to 2024-02-14 23:32 UTC, by the times Opsgenie posted.
What was affected?
Opsgenie named Incident Flow, Incident Flow, Alert Flow, Alert Flow and 39 more.
What caused it?
Opsgenie hasn't published a root cause. If it does, it shows here.
Where does this come from?
Opsgenie's official status page, https://opsgenie.status.atlassian.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.