Heroku sync integrations are failing

What to do

• Check whether you use Doppler Heroku. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 17h 34m it lasted.

  Anything that failed between 2023-01-05 00:48 UTC and 2023-01-05 18:23 UTC may need a replay.

• Subscribe to Doppler's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=doppler works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Doppler 4

1. identifiedJan 5, 2023, 12:48 AM UTC

   New and existing Heroku sync integrations are currently not functioning. We have identified the issue and are working on resolving it. Secrets previously synced to Heroku apps will remain available but new secret changes in Doppler will not be synced until this is resolved.

   provider

2. identifiedJan 5, 2023, 1:06 AM UTC

   This incident may persist for up to 24 hours. In the meantime, if you have an urgent secret change that needs to be made, please update the secret in your Doppler dashboard and then manually update the secret in Heroku directly via `heroku config:set`. Updating the secret in Doppler will ensure that the value in Heroku is not overwritten once the incident is resolved.

   provider

3. resolvedJan 5, 2023, 6:23 PM UTC

   This issue is now resolved. Additional action is required to re-enable existing Heroku syncs. All workplaces will need to reconnect their Heroku sync integrations from the [workplace Settings page](https://dashboard.doppler.com/workplace/settings). Once integrations are reconnected, all syncs will automatically be triggered to sync any pending changes in Doppler. A postmortem will be available soon with more details regarding what happened.

   provider

4. postmortemJan 5, 2023, 9:30 PM UTC

   **Root Cause** Doppler syncs secrets to Heroku via a Heroku OAuth application. This application is created in the Heroku dashboard and must be owned by a single Heroku user account. Doppler’s previous Heroku OAuth application was owned by a specific Doppler employee’s Heroku account without access to any additional resources. During a routine external account audit, this account was mistakenly identified as unused and manually deleted by our security team. This irrecoverably deleted Doppler’s existing Heroku OAuth application, thereby breaking any existing syncs and requiring the creation of a new OAuth application in a new account. **Resolution** Because users had authorized our previous Heroku OAuth application to their Heroku account\(s\), users need to authorize the new Heroku OAuth application. This involves reconnecting the integration from the [Doppler dashboard](https://dashboard.doppler.com/workplace/settings). Once the integration is reconnected, Doppler will re-enable all associated syncs that have been disabled and perform a fresh sync. Note that the previous OAuth application was deleted and therefore no action is required to remove its access to your Heroku account. **Next Steps** Internally, we’re reorganizing how shared accounts used for critical functionality are stored in 1Password. This new 1Password organization should help prevent this kind of accidental deletion in the future. We avoid shared accounts whenever possible, but this isn’t always feasible given third party implementations. We'll also be adding our individual integrations to our status page. This will allow customers to more easily see which integrations, if any, are currently experiencing issues.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.