# CloudBees Rollout service incident

- Provider: [CloudBees](https://down.guide/jenkins)
- Level: confirmed
- Impact: Degraded
- lasted 12h 29m
- started 2020-07-15 13:11 UTC
- us
- id: DWN-e7fedd9dfa
- Official page: http://stspg.io/l5m8k8htjjlx

## Updates from CloudBees
### 2020-07-15 11:41 UTC: identified
There is an outage with a 3rd party service. We're contacting them to see what's the situation.

### 2020-07-15 12:38 UTC: identified
We have confirmed the database issue with our service provider and they are working to restore service. 
Service impact is that experiments can’t be updated, but existing flags are unaffected.

### 2020-07-15 13:11 UTC: identified
Our service provider has updated us about the situation and the rough estimate for recovery is 2 to 3 hours.

### 2020-07-15 14:08 UTC: identified
Our service provider announced is going to take longer to recover the system. The current rough estimate for recovery is from 5 to 9 hours.

### 2020-07-15 16:02 UTC: identified
Monitoring https://status.compose.com/ for further updates

### 2020-07-15 21:07 UTC: identified
We've recieved this message from our service provider: “At this point we are cautiously optimistic. Our engineers are close to having virtual networking up across all hosts in the cluster. So far so good. Once stable we will start bringing capsules back up.”
More updates to follow.

### 2020-07-15 22:42 UTC: identified
IBM Compose update - "Virtual networking is up across all hosts in the cluster and the situation appears to be stable. We are slowly starting data/member capsules. Once those are up, we will start portals which will restore customer access"
In parallel - CloudBees engineering teams are now working to restore the database service to our own infrastructure - with a view to failing over if Compose is not able to restore access in a timely manner.

### 2020-07-16 00:05 UTC: identified
Our Engineering Team has been successfully restored the database but still not 100% operational. 
More updates to follow.

### 2020-07-16 00:40 UTC: monitoring
Apart from Impression analytics - which is currently not working - the service is back to operational. We're monitoring the situation.

### 2020-07-16 01:18 UTC: monitoring
The Rollout core service (API/login/web) outage has been resolved and these services are now fully operational.
However, Impression Analytics are not currently available, the engineering team are working to resolve this issue.
We will continue to provide service updates on the status of Impression Analytics until the issue is resolved.

### 2020-07-16 01:39 UTC: resolved
All CloudBees Rollout (Feature Flags) services are now fully operational. We have not identified any data-loss or security impact from this outage.
An outage post-mortem and corrective actions will be performed in due course.
Thank you for your patience.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).