Skip to main content
down.guideby Sazabi
TaxJar
DWN-632e0936e8TaxJarResolved

TaxJar App, API Outage

Major outage Lasted 1h 31m · started · TaxJar's page

Level
confirmed
Worst impact
Major outage
Components
TaxJar Reporting, Tax Calculations API, Tax Rates API, Transaction Push API, Other API Services
Regions
us
Down Index
57of 100

What to do

  • Check whether you use TaxJar TaxJar Reporting, Tax Calculations API, Tax Rates API, Transaction Push API and 1 more in us. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

  • Confirm your retries, queues and scheduled jobs caught up after the 1h 31m it lasted.

    Anything that failed between 2021-01-11 18:30 UTC and 2021-01-11 20:00 UTC may need a replay.

  • Subscribe to TaxJar's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=taxjar works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from TaxJar 9

  1. investigating

    We are currently investigating this issue.

    provider

  2. investigating

    We are continuing to investigate this issue.

    provider

  3. investigating

    We are continuing to investigate this issue.

    provider

  4. identified

    The issue has been identified and a fix is being implemented.

    provider

  5. monitoring

    A fix has been implemented and we are monitoring the results.

    provider

  6. monitoring

    We are continuing to monitor for any further issues.

    provider

  7. monitoring

    We are continuing to monitor for any further issues.

    provider

  8. resolved

    This incident has been resolved.

    provider

  9. postmortem

    During this incident, TaxJar customers were not able to access the TaxJar App or use the TaxJar API. We know this was impactful, and we are truly sorry it happened.  We have already implemented the following operational changes to ensure this type of failure does not happen again: * We updated our deployment pattern to a blue-green deployment pattern to allow us to better verify changes to production environments. * We are conducting a full audit of our vendor provided managed services that lack the acceptable level of rollback capabilities **Incident Root Cause Analysis**  * The incident started with a routine Kubernetes minor version upgrade using our vendor’s managed kubernetes service * This is a routine upgrade operation that we’ve completed 15 times in the past across 3 accounts and 2 regions. We perform this upgrade quarterly in order to keep pace with Kubernetes releases. * Immediately following completion of the upgrade of our production cluster, Kubernetes workers began reporting “Not Ready” status. * Within a few minutes all nodes were now in a state of “Not Ready” which caused all workloads to be marked as offline by our load balancers. * Kubernetes upgrades on our vendor’s managed Kubernetes service are not able to be rolled back. Furthermore new deployments and upgrades to the managed Kubernetes service can take 30-50 minutes to complete, leaving us forced to resolve the immediate issue rather than rolling back. * The vendor’s support team was able to identify the issue: * Clusters, starting with Kubernetes version 1.14 create a cluster security group when they are created.  * This security group is designed to allow all traffic from the control plane and[ ](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)managed node groups to flow freely between each other.  * After the upgrade was completed, the vendor identified that the security group no longer had the required rules configured to allow this traffic to pass, even though this has always happened in prior instances of minor version upgrades. * We manually added the missing rule, which restored connectivity to our managed Kubernetes cluster. * At this point our services started coming back online. * Several other security groups, managed with cloudformation, which had utilized this rule for connectivity between our K8s workloads to other services provided by the vendor \(such as memory caches and databases\) were identified as being unexpectedly altered after this upgrade and also had to be repaired before all services could be restored. * We continue to work with the vendor to understand the root cause for the failure of the managed service to not operate as documented.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-632e0936e8

Is TaxJar still down?

No, this incident was resolved 2021-01-11 20:00 UTC. For TaxJar's live state, see https://down.guide/taxjar.

How long did it last?

1h 31m, from 2021-01-11 18:30 UTC to 2021-01-11 20:00 UTC, by the times TaxJar posted.

What was affected?

TaxJar named TaxJar Reporting, Tax Calculations API, Tax Rates API, Transaction Push API and 1 more in us.

What caused it?

TaxJar hasn't published a root cause. If it does, it shows here.

Where does this come from?

TaxJar's official status page, https://status.taxjar.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.

Sources

Feed