Lokalise main application service degradation

What to do

• Check whether you use Lokalise in apac, us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-12-20 09:15 UTC and 2024-12-20 09:15 UTC may need a replay.

• Subscribe to Lokalise's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=lokalise works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Lokalise 2

1. resolvedDec 20, 2024, 9:15 AM UTC

   Several content integrations were affected including Contentful, Storyblok, Zendesk Guide, HubSpot, and ContentStack.

   provider

2. postmortemDec 23, 2024, 12:32 PM UTC

   On December 19, 2024, we experienced a significant service degradation affecting several content integrations, including Contentful, Storyblok, Zendesk Guide, HubSpot, and ContentStack. The incident began at 22:32 UTC and lasted approximately 9 hours, primarily impacting our customers in the US and APAC regions during their business hours. The issue was first reported by one of our customers. The incident was triggered by a late release that introduced a misconfiguration on our production environment. Upon receiving the customer complaint at 01:00 UTC on December 20, our team promptly initiated a rollback of the faulty release. By 08:15 UTC, we successfully restored services to a stable state. If you had scheduled imports or exports during the affected hours, it is likely that they failed due to this incident. We understand how important these operations are for your business, and we apologize for any disruption this may have caused. To prevent similar incidents in the future, we are taking the following steps: * We will ensure consistent configurations across all environments to catch potential issues early. * We are enhancing our testing protocols to include scenarios that replicate customer-reported issues. * We will reassess our release schedule, particularly regarding significant changes, to avoid late releases before critical periods. We sincerely apologize for the inconvenience this incident caused to our customers. We appreciate your understanding and patience as we work to improve our systems and ensure reliable service in the future. Your trust is important to us, and we are committed to enhancing our processes to prevent such occurrences.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.