Elevated Error Rate

What to do

• Check whether you use Close Application UI, API. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 1m it lasted.

  Anything that failed between 2023-08-01 16:22 UTC and 2023-08-01 17:22 UTC may need a replay.

• Subscribe to Close's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=close works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Close 3

1. monitoringAug 1, 2023, 4:22 PM UTC

   Close has become aware of degraded performance of our API. This resulted in a brief window between 15:55 and 16:05 UTC when users may have seen elevated error rates when using the Close App and API. Performance has recovered and users should no longer be impacted. Close Engineering will continue to monitor and investigate the root cause.

   provider

2. resolvedAug 1, 2023, 5:22 PM UTC

   This incident has been resolved.

   provider

3. postmortemAug 1, 2023, 10:38 PM UTC

   Close sincerely apologizes for the interruption of our service. We take the stability of a platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.  ## Impact Between 15:55 UTC and 16:05 UTC on August 1, 2023 Close users may have experienced degraded performance and an elevated error rate when using the Close application or API. ## Root Cause and Resolution This incident began after a routine deployment of changes to the Close backend. There was a subtle error in a query to the internal metrics service that caused our compute platform to intermittently and erroneously underestimate the true demand for resources after a deployment.  This query to the metrics service has been corrected. Additionally Close has implemented automatic safeguards that will prevent our compute platform from deprovisioning too many resources during peak usage periods.  ## Timeline * 15:24 UTC - A routine deployment of changes to the Close application occurs.  * 15:31 UTC - Our compute platform begins deprovisioning resources from the Close application. * 15:55 UTC - The Close application becomes starved of resources. Customers begin to notice elevated error rates. * 16:01 UTC - Another routine deployment of changes to the Close application occurs. * 16:05 UTC - Our compute platform begins provisioning the correct amount of resource for the Close application. Performance recovers.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.