GitHub CTO Promises Overhaul after Outages: “We Let You Down”

Following two severe outages, GitHub promises improvement. CTO Fedorov announces an architectural overhaul due to massive overload.

GitHub CTO Vladimir Fedorov commented in a statement on the severe system outage from August 17. The nearly eight-hour outage brought core services such as GitHub Actions, Pull Requests, Copilot, and APIs to a standstill worldwide.

It was already the second major disruption within a few weeks (following an incident on August 6). The reason for this was insufficient scaling behavior during sudden surges in load.

“If you were trying to ship software that day, we let you down.”

Vladimir Fedorov, GitHub CTO

Ad

Explosive traffic growth overruns platform

The root cause lies in the enormous growth of the developer platform, which the existing infrastructure could not keep pace with:

• Doubling of load: In April, GitHub processed around 1.4 billion commits per month—now it is 2.9 billion, accompanied by 24 million new repositories and 130 million merges monthly.
• Azure migration: Currently, around 58 percent of the platform load and half of all Git operations run on Microsoft Azure. The migration of further workloads to the parent company’s cloud is now being accelerated.
• No release error: According to Fedorov, the outages were not triggered by faulty code or configuration changes. Instead, traffic spikes and so-called “retry storms” (automated retry requests) brought pre-existing architectural flaws to light.

Architectural overhaul and limiting the blast radius

To restore reliability, GitHub announced a phased overhaul of its platform architecture. The primary goal is linear scaling of read capacity to enable virtually unlimited read operations, starting with the largest monorepositories.

In addition, critical systems are to be isolated more strongly from one another in order to minimize the blast radius of future disruptions. Tightened limits for automated retry requests and an early warning logic for traffic spikes are intended to prevent future overload spirals.

(Editorial Team)