# \[Resolved\] Network outage

- Provider: [SourceHut](https://down.guide/sourcehut)
- Level: confirmed
- Impact: Partial outage
- lasted 0m
- started 2024-01-10 08:50 UTC
- us
- id: DWN-f04ebde53a
- Official page: https://status.sr.ht/issues/2024-01-10-network-outage/

## Updates from SourceHut
### 2024-01-10 08:50 UTC: resolved
Update 2024-01-17 10:30 UTC : full service has been restored.
 Update 2024-01-16 19:45 UTC : builds.sr.ht is now running at full capacity.
 Update 2024-01-16 18:00 UTC : builds.sr.ht is now available for full service.
We are running it at half of the planned capacity, with the remainder coming
online shortly, so you may experience longer queue times while waiting for
builds to be processed.
 Update 2024-01-16 10:30 UTC : Our priority for today is to bring builds.sr.ht
and chat.sr.ht online, as well as address a number of less user-impacting tasks
we have on our todo list.
 Update 2024-01-15 22:15 UTC : Object storage for git.sr.ht (e.g. releases
attached to git tags) is now available for read and write operations.
 Update 2024-01-15 15:50 UTC : pages.sr.ht is now available for full service,
albeit with degraded performance for publish operations.
 Update 2024-01-15 08:20 UTC : hg.sr.ht is now available for full service. We
are working on builds availability today.
 Update 2024-01-15 07:45 UTC : We got in touch with hg.sr.ht&rsquo;s community
maintainer and put the finishing touches on it. It is now available in read-only
mode.
 Update 2024-01-14 14:45 UTC : The mail system is coming back online, and with
it lists and todo are entering full service. We do not believe any emails were
lost, queues should process normally and catch up on emails sent during outage
intervals.
 For users with questions about billing: the billing system has been shut off for
the duration of the outage. Before we turn it on again, we are going to credit
all paying users with free service for the duration of the outage period.
 Update 2024-01-14 14:00 UTC : We have have restored git.sr.ht and man.sr.ht
to full service.
 Note that we are still working with our existing transit provider and are
experimenting with a new solution for mitigating the DDoS, but we are not
certain that this approach is reliable and we are still working on setting up a
more permanent transit solution in the background.
 Update 2024-01-14 08:50 UTC : We have partially restored service once again,
and are working on restoring more services. lists and todo are operational from
the web, but we have disabled the mail system for the time being.
 Update 2024-01-13 14:17 UTC : We will need to find a new transit provider to
mitigate the problem. We are in talks with a provider to address the problem and
we will begin the engineering work shortly. Further updates to come as we have
them. Thank you again for your patience.
 Update 2024-01-13 13:00 UTC : Our new transit provider has notified us that
the DDoS has followed us to our new network. They have deployed mitigations and
we do not expect an interruption in service. Update : service is impacted, we
are investigating further.
 Note that our new transit solution utilizes end-to-end encryption such that
traffic between you and SourceHut is received and processed directly by our
colocated servers and is not handled in plaintext by third-parties.
 Update 2024-01-13 12:40 UTC : Mail service to lists and todo has been
restored, subject to DNS propagation delays. Emails queued during the outage
should resume processing now &ndash; we do not believe any emails have been lost.
 Update 2024-01-13 11:42 UTC : We have brought pages.sr.ht into service for
read-only operations. All custom domains should be working soon, subject to DNS
propagation delays, with the exception of custom domains using apex records
(i.e. top level domains such as example.org rather than subdomain.example.org).
 Manual intervention is required for affected users .
 We have established a temporary IP address for serving custom domains using apex
records. Users can change their apex record to the following IP address to
restore read-only pages service:
 @ IN A 141.95.4.185
 Note that we may change this IP address in the future. You will be notified by
email if later changes are required for your domain.
 Update 2024-01-13 11:00 UTC : We have enabled read/write access to most

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).
