A Maintenance Command Cut Facebook Off the Internet in 2021. Its DNS Servers Were Still Running.

On October 4, 2021, a command meant to assess the available capacity of Facebook’s global backbone network disconnected the company’s data centers around the world instead. Facebook, Instagram, WhatsApp and Messenger vanished from the internet. They stayed unreachable for about six hours.

The strangest detail is in Facebook’s own explanation. Its DNS servers — the Domain Name System machines that tell browsers and apps where to find facebook.com — were still running. They had taken themselves off the internet on purpose, as they were designed to do when they lost contact with the data centers.

From the outside, the outage looked like a DNS failure. Cloudflare, which runs the public 1.1.1.1 DNS resolver, saw a burst of routing changes from Facebook’s network at about 15:40 Coordinated Universal Time (UTC). At 15:51 it opened an internal incident titled “Facebook DNS lookup returning SERVFAIL,” the error a resolver returns when it cannot get an answer from a domain’s name servers. At 15:58, Cloudflare noticed that Facebook had stopped announcing the routes to the addresses of its DNS servers.

What the maintenance command did

Facebook’s backbone is the network of fiber-optic cables and routers that links its data centers. Engineers take parts of it offline for maintenance. Santosh Janardhan, Facebook’s vice president of infrastructure, wrote that during one of these routine jobs, a command was issued “with the intention to assess the availability of global backbone capacity.” Instead, it “unintentionally took down all the connections in our backbone network.”

A safeguard should have stopped it. “Our systems are designed to audit commands like these to prevent mistakes like this,” Janardhan wrote, “but a bug in that audit tool prevented it from properly stopping the command.”

Why working DNS servers disappeared

Two pieces of internet plumbing explain what happened next. DNS turns a name such as facebook.com into the numeric address of a server. BGP, the Border Gateway Protocol, is how networks tell one another which addresses they can deliver traffic to. Facebook’s smaller facilities host its DNS servers, also called name servers, and BGP advertisements tell the rest of the internet how to reach those name servers.

The name servers follow a rule. If they cannot talk to the data centers, Janardhan explained, they disable their BGP advertisements, “since this is an indication of an unhealthy network connection.” On October 4, the entire backbone was out of operation, so those facilities declared themselves unhealthy and withdrew their advertisements. The servers, in his words, “became unreachable even though they were still operational.”

The rest of the internet kept asking. People and apps retried again and again, and Cloudflare reported that DNS resolvers worldwide were “handling 30x more queries than usual.”

Engineers had to get into the data centers

Inside Facebook, the engineers trying to fix the outage faced two obstacles. They could not reach the data centers “through our normal means because their networks were down,” Janardhan wrote, and losing DNS broke many of the internal tools they would normally use to investigate and resolve outages.

So Facebook sent engineers to the data centers to debug the problem and restart systems on site. That took time, because the facilities are “designed with high levels of physical and system security in mind.” They are hard to get into, and the hardware and routers inside are designed to be difficult to modify even by someone with physical access. “So it took extra time,” Janardhan wrote, “to activate the secure access protocols needed to get people onsite and able to work on the servers.”

Janardhan defended that friction. Hardening against unauthorized access, he wrote, had slowed the recovery from an outage caused “not by malicious activity, but an error of our own making.” He still judged the trade worth it: “greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this.”

Turning Facebook back on was a risk of its own

Cloudflare saw renewed BGP activity from Facebook’s network at around 21:00 UTC. At 21:20, facebook.com resolved again on Cloudflare’s resolver, and by 21:28 UTC Facebook appeared reconnected to the global internet.

Restoring the backbone did not end the danger. Switching every service back on at once could have caused a new round of crashes from the surge in traffic. Individual data centers were reporting dips in power usage “in the range of tens of megawatts,” and suddenly reversing a dip like that could put everything from electrical systems to caches at risk.

Facebook had rehearsed for that part. In exercises it calls “storms,” it simulates a major system failure by taking a service, a data center or an entire region offline. But, Janardhan wrote, “we’ve never previously run a storm that simulated our global backbone being taken offline.” The experience from the drills still helped the team bring services back while carefully managing the rising load.

Not a hack, and not quite a DNS outage

Rumors followed the outage. Social media posts claimed that a 13-year-old Chinese hacker had taken Facebook down; fact-checkers at Agence France-Presse (AFP) rated the claim false. Facebook said there was “no malicious activity behind this outage.”

Calling it a DNS outage is not quite right either. DNS was the failure most outsiders could see, and Cloudflare’s internal incident title named it. But the name servers were working. The backbone command cut them off from the data centers, and their own rule then took them off the internet.

The lesson: safety rules that fire everywhere at once

Three safeguards shaped that day. The audit tool meant to block a dangerous command had a bug. The DNS rule meant to take an unhealthy location out of service fired in every location at once. And the security that keeps intruders out of the data centers also slowed down the engineers who needed to get in.

None of those designs was foolish on its own. The DNS rule made sense when one location lost contact with the data centers. Facebook’s storm drills had been scoped the same way: they had taken out a service, a data center or a region, never the whole backbone. The backbone command produced a total failure instead. The useful question about any safety rule is what it does when every part of a system trips it at the same moment.

On October 4, 2021, Facebook got its answer: name servers that were up and running, and an internet that could not find them for about six hours.

Explore

Stay connected