On June 8, 2021, a customer’s valid configuration change triggered a dormant bug that caused 85% of content delivery network (CDN) provider Fastly’s network to return errors, according to Fastly. Websites including The New York Times, The Guardian, and Reddit were affected.
The disruption began at 09:47 Coordinated Universal Time (UTC). Fastly’s automated monitors detected the failure within one minute.
The customer had not submitted a malformed configuration. Fastly’s engineering leadership documented a valid change that activated a software bug introduced in a deployment nearly four weeks earlier.
The Dormant Flaw in Production
On May 12, 2021, Fastly had begun deploying an edge software release that contained an undiscovered defect. Fastly said the bug could be triggered by a specific customer configuration under specific circumstances. It did not disclose the configuration or those circumstances. The deployment began 27 days before the outage.
Then, early on June 8, a customer pushed a valid configuration change that triggered the bug, according to Fastly. As Fastly executive Nick Rockwell later reported, the resulting failure caused 85% of Fastly’s network to return errors. Fastly did not define the denominator behind that network percentage; it was not a figure for global internet traffic. Because visitors reached these sites through Fastly, a failure in its delivery layer could prevent access even when a site’s origin servers remained available.
Minutes to Isolate, Hours to Patch
Fastly’s response unfolded quickly. At 09:48 UTC, one minute after disruption began, automated monitoring identified the failure. Ten minutes later, at 09:58 UTC, Fastly published its first public incident notification. At 10:27 UTC, Fastly Engineering identified the customer configuration that had triggered the disruption.
Fastly isolated the cause and disabled the specific customer configuration. With the configuration disabled, impacted services began to recover at 10:36 UTC. Fastly reported that within 49 minutes of onset, 95% of its network was operating normally, without defining the denominator behind that percentage. By 11:00 UTC, the majority of customer services had recovered, and at 12:35 UTC, Fastly officially marked the incident mitigated.
That quick recovery was not a software fix. The network recovered because engineers had deactivated the valid input that provoked the failure. Fastly said that after mitigating the immediate effects, it turned to fixing the bug and communicating with customers. Fastly did not begin deploying its permanent bug fix until 17:25 UTC that evening.
The Architecture of Shared Edge Dependencies
A single customer’s update could spread across unrelated sites because of how content delivery networks work. CDNs operate networks of proxy servers that cache content close to end users. TechCrunch explained that fetching a page from the other side of the world can take hundreds of milliseconds, while a nearby CDN with a cached page can usually start sending it in less than 25 milliseconds. Fastly’s documentation says origin shielding can potentially protect an origin server from unexpected spikes in requests.
However, this shared reverse-proxy model means many unrelated websites depend on the same provider’s edge. When an edge proxy fails, requests routed through it can fail even if the origin infrastructure remains available. Redundant origin data centers alone do not ensure that the delivery path remains available.
Engineering Lessons from Valid Failures
The Fastly outage illustrates that a valid input can trigger a latent software defect and disrupt a production service. Fastly said it would investigate why its software quality assurance and testing processes had not detected the bug.
For engineers, the incident suggests separating three questions: What software defect caused the failure? What input triggered the defect? What can operators disable while engineers prepare a patch?
Fastly’s network largely recovered once the triggering configuration was disabled, hours before the software fix began rolling out. Teams can plan how to remove a trigger while the defect still exists.
Related stories
Sources
- Nick Rockwell (2021), Summary of June 8 outage, Fastly Blog, June 8, 2021.
- Fastly Documentation, Getting started with Full-Site Delivery, Fastly Network Services Guides.
- Manish Singh and Romain Dillet (2021), Twitch, Pinterest, Reddit and more go down in Fastly CDN outage, TechCrunch, June 8, 2021.
- David Meyer (2021), From WhiteHouse.gov to Reddit: Fastly CDN outage knocks out sites and services around the world, Fortune, June 8, 2021.
- Featured image: Derrick Coetzee from Berkeley, CA, USA; Wikimedia Commons; CC0 (adapted from the original)
SIGNAL TITANS
Leave a Reply