Elevated Error Rate

What to do

• Check whether your product depends on Close. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2023-07-28 15:45 UTC and 2023-07-26 16:30 UTC may need a replay.

• Subscribe to Close's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=close works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Close 2

1. resolvedJul 26, 2023, 4:30 PM UTC

   Between 16:47 UTC and 17:18 UTC on July 26, 2023 Close users may have experienced degraded performance and an elevated error rate when using the Close application or API.

   provider

2. postmortemJul 28, 2023, 3:45 PM UTC

   Close sincerely apologizes for the interruption of our service. We take the stability of a platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.  ## Impact Between 16:47 UTC and 17:18 UTC on July 26, 2023 Close users may have experienced degraded performance and an elevated error rate when using the Close application or API. ## Root Cause and Resolution This incident began when our internal metrics service became degraded at 16:47 UTC on July 26, 2023. Our internal metrics service is used by our compute platform to automatically determine the appropriate amount of resources needed to run the Close application. When the metrics service became degraded our compute platform was unable to determine the correct amount of resources needed to run the Close application. Over the next several minutes our compute platform de-provisioned resources causing the system to become overloaded and perform poorly. When the internal metrics service was restored at 17:18 UTC our compute platform provisioned the correct amount of resources and performance returned to normal. Our compute platform has a fail safe built in to avoid this exact situation that did not function. We are investigating why this fail safe did not function. We are also deploying improvements to our internal metrics service to make it more stable and alert sooner if it becomes unstable.  ## Timeline * 16:47 UTC - Our internal metrics service becomes degraded. * 17:00 UTC - Without metrics available our system assumed it was not under load and deprovisioned resources. This caused performance to degrade. * 17:09 UTC - The Engineering Team becomes aware of an issue with application performance. * 17:18 UTC - Our internal metrics system is restored. This causes our system to provision the appropriate amount of resources. Application performance recovers

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.