High response times on CMA and CDA - Admin interface not loading

What to do

• Check whether your product depends on DatoCMS. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2023-11-06 13:56 UTC and 2023-11-06 13:56 UTC may need a replay.

• Subscribe to DatoCMS's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=datocms works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from DatoCMS 1

1. resolvedNov 6, 2023, 1:56 PM UTC

   Resolved — This incident was caused by a customer with a large user base using a private plugin that went rogue. The plugin started making a lot of API calls and since the user base of this customer is very large the entire platform was hit by a DDoS. This caused generic slow downs as the requests were authenticated as users and were hitting directly our web servers. To avoid this from happening again we have added rate-limitations to JWT tokens, that are used by plugins acting as users, something that we didn't anticipate before. Resolved — We can close the issue now Monitoring — We noticed another batch of unusual traffic at reported time. We will deploy some countermisures soon. Monitoring — We experienced about 10 minutes of high response times on our CMA and CDA endpoints. This means that requests sent to those endpoints (site-api.datocms.com, graphql.datocms.com) started to have high latencies or timeout errors (503). The admin interface was affected too, since it uses the CMA endpoint itself. The cause was as unusual amount of traffic that we are going to investigate further. We are monitoring the situation right now.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.