[Resolved] Network outage

What to do

• Check whether you use SourceHut in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-01-10 08:50 UTC and 2024-01-10 08:50 UTC may need a replay.

• Subscribe to SourceHut's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=sourcehut works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from SourceHut 1

1. resolvedJan 10, 2024, 8:50 AM UTC

   Update 2024-01-17 10:30 UTC : full service has been restored. Update 2024-01-16 19:45 UTC : builds.sr.ht is now running at full capacity. Update 2024-01-16 18:00 UTC : builds.sr.ht is now available for full service. We are running it at half of the planned capacity, with the remainder coming online shortly, so you may experience longer queue times while waiting for builds to be processed. Update 2024-01-16 10:30 UTC : Our priority for today is to bring builds.sr.ht and chat.sr.ht online, as well as address a number of less user-impacting tasks we have on our todo list. Update 2024-01-15 22:15 UTC : Object storage for git.sr.ht (e.g. releases attached to git tags) is now available for read and write operations. Update 2024-01-15 15:50 UTC : pages.sr.ht is now available for full service, albeit with degraded performance for publish operations. Update 2024-01-15 08:20 UTC : hg.sr.ht is now available for full service. We are working on builds availability today. Update 2024-01-15 07:45 UTC : We got in touch with hg.sr.ht’s community maintainer and put the finishing touches on it. It is now available in read-only mode. Update 2024-01-14 14:45 UTC : The mail system is coming back online, and with it lists and todo are entering full service. We do not believe any emails were lost, queues should process normally and catch up on emails sent during outage intervals. For users with questions about billing: the billing system has been shut off for the duration of the outage. Before we turn it on again, we are going to credit all paying users with free service for the duration of the outage period. Update 2024-01-14 14:00 UTC : We have have restored git.sr.ht and man.sr.ht to full service. Note that we are still working with our existing transit provider and are experimenting with a new solution for mitigating the DDoS, but we are not certain that this approach is reliable and we are still working on setting up a more permanent transit solution in the background. Update 2024-01-14 08:50 UTC : We have partially restored service once again, and are working on restoring more services. lists and todo are operational from the web, but we have disabled the mail system for the time being. Update 2024-01-13 14:17 UTC : We will need to find a new transit provider to mitigate the problem. We are in talks with a provider to address the problem and we will begin the engineering work shortly. Further updates to come as we have them. Thank you again for your patience. Update 2024-01-13 13:00 UTC : Our new transit provider has notified us that the DDoS has followed us to our new network. They have deployed mitigations and we do not expect an interruption in service. Update : service is impacted, we are investigating further. Note that our new transit solution utilizes end-to-end encryption such that traffic between you and SourceHut is received and processed directly by our colocated servers and is not handled in plaintext by third-parties. Update 2024-01-13 12:40 UTC : Mail service to lists and todo has been restored, subject to DNS propagation delays. Emails queued during the outage should resume processing now – we do not believe any emails have been lost. Update 2024-01-13 11:42 UTC : We have brought pages.sr.ht into service for read-only operations. All custom domains should be working soon, subject to DNS propagation delays, with the exception of custom domains using apex records (i.e. top level domains such as example.org rather than subdomain.example.org). Manual intervention is required for affected users . We have established a temporary IP address for serving custom domains using apex records. Users can change their apex record to the following IP address to restore read-only pages service: @ IN A 141.95.4.185 Note that we may change this IP address in the future. You will be notified by email if later changes are required for your domain. Update 2024-01-13 11:00 UTC : We have enabled read/write access to most

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.