# US Customers MQTT Service Disruption

- Provider: [Spacelift](https://down.guide/spacelift)
- Level: confirmed
- Impact: Partial outage
- lasted 1h 22m
- started 2026-10-21 00:03 UTC
- Public workers
- eu, us
- id: DWN-4c9b6f4e9a
- Official page: https://stspg.io/h6k0j0f0s71n

## Root cause
> \*\*Root Cause\*\*

[Postmortem](https://stspg.io/h6k0j0f0s71n)

## Updates from Spacelift
### 2026-10-21 00:03 UTC: investigating
We’re seeing an increased number of errors affecting the worker polling API for customers in our US region. Our team is actively working on mitigation and will share updates as they become available.

### 2026-10-21 00:17 UTC: identified
We have the fix implemented and we will release it shortly.

### 2026-10-21 01:10 UTC: monitoring
We deployed the fix and monitoring.

### 2026-10-21 01:26 UTC: resolved
This incident has been resolved.

### 2026-10-28 14:04 UTC: postmortem
Root Cause 
A change in a new launcher version caused it to connect by default to the EU endpoint if the region was not explicitly configured. Workers registered in other regions were then rejected at authentication. This was a human error.
Detection was delayed for three reasons:
\* Our pre-release testing did not cover workers outside the EU region well enough.
\* Launcher authentication errors usually come from individual customer misconfiguration, so the increase was not flagged as a platform issue.
\* MQTT disconnections coincided with our planned MQTT-to-polling migration, which made the drop look expected.
Mitigations
\* Immediate: We rolled back to a working version of the launcher
\* Before every deployment, we will now boot fresh private workers in other regions and confirm that they connect and complete a run. A failure like this should be caught before a release.
\* We are improving alerting so that authentication failures across multiple customers are flagged automatically as a potential platform issue.
Conclusion 
A broken binary caused launchers without a configured region to target the wrong endpoint. Gaps in regional test coverage and alerting meant we did not catch it before customers did. Additional automated checks are being put in place so that this failure would be detected before release along with improved alerting so that we are notified sooner for issues that make it past our automated checks.

From the builder: does your app depend on Spacelift? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=spacelift) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Spacelift.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-4c9b6f4e9a), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.