On October 21, 2018, GitHub lost a network connection for 43 seconds. The resulting service degradation lasted 24 hours and 11 minutes: the connection returned, but databases on opposite sides of the United States now held different histories of accepted writes.
Restoring the link could not settle which records belonged in the recovered system. GitHub’s engineers had to rebuild a usable database arrangement while preserving changes already received from users.
The uncomfortable part was that the automatic failover system had behaved as configured. It had moved database leadership to a location the application could not comfortably use.
A replacement database was only part of the answer
GitHub stored more than Git repositories. Its MySQL databases supported issues, pull requests, authentication and background work. Losing reliable access to that metadata could disrupt the website even when the underlying problem was separate from raw Git object storage.
Database clusters had a primary server that accepted writes and replicas that copied changes and handled much of the read traffic. Orchestrator, the tool managing those arrangements, could promote a replacement primary when the existing arrangement failed.
GitHub engineer Shlomi Noach had explained the operational need in a 2016 article: waiting for someone to wake up and repair a failed primary would take too long. Automation also had to distinguish a failed server from an observer that merely could not reach it.
That distinction became consequential during routine maintenance on October 21. At 22:52 Coordinated Universal Time (UTC), work to replace failing optical equipment interrupted connectivity between GitHub’s East Coast network hub and its main East Coast data center.
During the interruption, Orchestrator nodes on the West Coast and in an East Coast public cloud could establish the agreement needed to act. They began promoting West Coast databases. Once connectivity returned, the application started sending writes to those new primaries.
The two sides had accepted different changes
The East Coast databases contained a short period of writes that had not reached the West Coast. The newly promoted West Coast databases then accepted further writes. Each side held changes missing from the other.
Simply switching back would therefore leave accepted work outside the active history. The incident report describes engineers choosing to continue recovery from the West Coast state while preserving the outstanding East Coast records for reconciliation.
That decision carried an immediate cost. Applications running on the East Coast could not cope with the additional delay of repeatedly accessing a primary across the country. A database could be available, yet be too far away for the application’s expectations.
At 23:19 UTC, engineers paused webhook delivery—the notifications sent to connected services—and GitHub Pages builds. Their stated priority was data integrity over site usability and recovery time. Continuing background work would have introduced further writes while they were trying to stabilize the databases.
Backups still needed time to become serving databases
The recovery plan called for restoring backups, synchronizing replicas at both sites, returning to a stable arrangement and then processing queued jobs.
Having backups did not make that sequence instantaneous. Restoring multiple terabytes involved transferring data from remote storage, decompressing it, checking it, preparing it and loading it into newly provisioned MySQL servers. GitHub said the restore procedure was tested daily, but the company had not previously needed to rebuild an entire cluster from backup during an incident.
By 11:12 UTC on October 22, database primaries were back on the East Coast. Writes became more responsive because the application and its primary databases were again nearby. Some read replicas, however, remained hours behind. Requests reaching those replicas could still show old information.
The team’s earlier recovery estimate had assumed replication would catch up along a linear trajectory. It did not. Increasing daytime traffic added load, and replication delays grew. Additional read replicas helped spread that load so synchronization could progress.
The backlog had its own failure mode
At 16:24 UTC, synchronized replicas allowed a return to the original topology. GitHub kept its service status red while engineers worked through accumulated jobs.
There were more than five million hook events and 80,000 Pages builds queued. Restarting processing meant balancing recovery speed against load on GitHub and the external services receiving notifications.
About 200,000 webhook payloads had exceeded an internal expiry period and were dropped during processing. Engineers paused that work and increased the expiry period temporarily. A queue could retain work through an outage and still discard it when recovery finally reached it.
The site returned to green at 23:03 UTC on October 22. The postmortem nevertheless described reconciliation of outstanding East Coast writes as still in progress when the report was published. That qualification matters alongside the company’s statement that no user data was lost; the report also explicitly records the dropped webhook payloads.
Failover had to match the application
GitHub identified a concrete configuration change: prevent Orchestrator from promoting database primaries across regional boundaries. The tool’s action had followed its configuration, while the resulting cross-country database access exceeded what the application could support.
The company also committed to testing failure scenarios more systematically. The useful question extended beyond whether a replacement primary could be elected: could the application use it, could replicas catch up under traffic, and could queued work survive until processing resumed?
The network interruption lasted 43 seconds. Recovery required bringing the databases, the application and the waiting work back into agreement.
Related stories
Sources
- Jason Warner — October 21 post-incident analysis
- Shlomi Noach — Orchestrator at GitHub
- Shlomi Noach — MySQL High Availability at GitHub
- Chris Williams, The Register — GitHub.com freezes up as techies race to fix dead data storage gear
- Featured image: Derrick Coetzee from Berkeley, CA, USA; Wikimedia Commons; CC0 (adapted from the original)
SIGNAL TITANS
Leave a Reply