Skip to main content
CloudBees

CloudBees Rollout service incident

Degraded Lasted 12h 29m · started · CloudBees's page

Level
confirmed
Worst impact
Degraded
Regions
us
Down Index
27of 100

What to do

  • Check whether you use CloudBees in us. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 12h 29m it lasted.

    Anything that failed between 2020-07-15 13:11 UTC and 2020-07-16 01:39 UTC may need a replay.
  • Subscribe to CloudBees's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=jenkins works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from CloudBees 11

  1. identified

    There is an outage with a 3rd party service. We're contacting them to see what's the situation.

    provider

  2. identified

    We have confirmed the database issue with our service provider and they are working to restore service. Service impact is that experiments can’t be updated, but existing flags are unaffected.

    provider

  3. identified

    Our service provider has updated us about the situation and the rough estimate for recovery is 2 to 3 hours.

    provider

  4. identified

    Our service provider announced is going to take longer to recover the system. The current rough estimate for recovery is from 5 to 9 hours.

    provider

  5. identified

    Monitoring https://status.compose.com/ for further updates

    provider

  6. identified

    We've recieved this message from our service provider: “At this point we are cautiously optimistic. Our engineers are close to having virtual networking up across all hosts in the cluster. So far so good. Once stable we will start bringing capsules back up.” More updates to follow.

    provider

  7. identified

    IBM Compose update - "Virtual networking is up across all hosts in the cluster and the situation appears to be stable. We are slowly starting data/member capsules. Once those are up, we will start portals which will restore customer access" In parallel - CloudBees engineering teams are now working to restore the database service to our own infrastructure - with a view to failing over if Compose is not able to restore access in a timely manner.

    provider

  8. identified

    Our Engineering Team has been successfully restored the database but still not 100% operational. More updates to follow.

    provider

  9. monitoring

    Apart from Impression analytics - which is currently not working - the service is back to operational. We're monitoring the situation.

    provider

  10. monitoring

    The Rollout core service (API/login/web) outage has been resolved and these services are now fully operational. However, Impression Analytics are not currently available, the engineering team are working to resolve this issue. We will continue to provide service updates on the status of Impression Analytics until the issue is resolved.

    provider

  11. resolved

    All CloudBees Rollout (Feature Flags) services are now fully operational. We have not identified any data-loss or security impact from this outage. An outage post-mortem and corrective actions will be performed in due course. Thank you for your patience.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-e7fedd9dfa

Is CloudBees still down?

No, this incident was resolved 2020-07-16 01:39 UTC. For CloudBees's live state, see https://down.guide/jenkins.

How long did it last?

12h 29m, from 2020-07-15 13:11 UTC to 2020-07-16 01:39 UTC, by the times CloudBees posted.

What was affected?

CloudBees named in us.

What caused it?

CloudBees hasn't published a root cause. If it does, it shows here.

Where does this come from?

CloudBees's official status page, https://www.cloudbeesstatus.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.