At midnight UTC on January 1, 2017, official timekeepers inserted a leap second to keep coordinated universal time aligned with the Earth’s slowing rotation. On affected servers, operating systems adjusted their clocks back by one second. Inside Cloudflare’s custom authoritative Domain Name System (DNS) software, known as RRDNS, that one-second backward step caused an elapsed-time measurement to produce a negative number. (DNS translates domain names into the addresses computers use to reach servers.) Minutes later, the negative value propagated into server selection routines, triggering software panics that caused DNS resolution failures for some Cloudflare-managed web properties.
The incident did not take down the entire Internet, nor did it disable all of Cloudflare’s systems. According to the company’s postmortem published by John Graham-Cumming, the failure specifically affected lookups sent to other DNS servers for customer domains using canonical name (CNAME) records, which point one domain name to another. Across Cloudflare’s 102 data centers at the time, approximately 0.2 percent of DNS queries were affected at peak, and fewer than 1 percent of Hypertext Transfer Protocol (HTTP) requests—the requests used to fetch web content—encountered an error. Yet the incident revealed a fundamental flaw in how software systems measured elapsed time: relying on wall-clock timestamps to compute durations.
A Measurement That Traveled Back in Time
To resolve CNAME records that pointed to external origins, RRDNS queried internal recursive DNS resolvers running inside each data center. To optimize routing, RRDNS tracked the round-trip response time of each resolver and used a weighted selection algorithm to send incoming queries to the most responsive servers. The measurement logic was straightforward: before sending a query over the User Datagram Protocol (UDP), the program recorded a start timestamp using Go’s standard library, and upon receiving the reply, it computed the difference: rtt := time.Now().Sub(start).
In normal operation, internal resolvers responded within a few milliseconds, producing a small positive duration. But when the host operating system rewound its clock by one second to accommodate the leap second, time.Now() returned a timestamp earlier than start. The subtraction yielded a value of roughly negative 990 milliseconds. RRDNS did not act on a single isolated reading; it maintained a smoothed average of resolver performance. As negative duration measurements were factored into the resolver’s smoothed performance metric, the metric itself eventually crossed below zero.
From Smoothed Averages to Random Panic
The failure remained dormant until RRDNS executed its weighted server selection algorithm. To pick an upstream resolver proportionally based on performance weights, the selection code passed the calculated values into Go’s pseudo-random number generator, calling rand.Int63n(). By specification, rand.Int63n() panics immediately if its argument is less than or equal to zero.
Because the smoothed performance number had become negative, the function panicked. Although Cloudflare’s engineers had wrapped request handlers with Go’s recover() mechanism to catch runtime panics and keep server processes from terminating abruptly, recovering from the panic aborted the resolution process for that request. The client received a DNS lookup failure. As more resolvers accumulated negative weights, lookups across affected data centers failed repeatedly.
Containment and the Immediate Patch
Cloudflare’s engineering team was alerted to rising error rates at 00:10 UTC on January 1, ten minutes after the clock adjustment. By 00:34 UTC, engineers confirmed that RRDNS was failing inside the weighted server selection code. Because the clock had already stopped rewinding, the immediate operational priority was preventing negative weights from reaching rand.Int63n().
At 00:55 UTC, engineers deployed a mitigation to a canary node. The patch checked whether the measured duration was negative; if backward time was detected, the code discarded the current measurement, prevented negative values from entering upstream server weights, and allowed the smoothed averages to normalize. The fix was applied to the most heavily affected data center by 01:23 UTC—approximately 90 minutes after the start of the incident—and rolled out across all data centers by 06:45 UTC. Restarting the RRDNS instances cleared the corrupted in-memory state and restored normal DNS operations.
Fixing the Clock at the Language Level
The Cloudflare incident exposed a widespread misconception among software engineers: assuming that reading the system clock twice and subtracting the two values is a safe way to measure elapsed time. At the time, Go’s time.Now() returned only wall-clock time, which is subject to adjustments through the Network Time Protocol (NTP), used to synchronize computer clocks, and to leap seconds. Operating system kernels had long maintained separate monotonic clocks—counters that only advance forward at a steady rate—specifically for interval measurement, but Go had not exposed monotonic time through its main functions for measuring durations.
Russ Cox’s design proposal for Go issue 12914, last updated on January 26, 2017, addressed the problem through Go’s time package. Cox noted that wall clocks are meant for telling calendar time, while monotonic clocks are meant for measuring elapsed intervals. Rather than creating a separate API that developers would have to remember to use, the Go team chose to embed an optional monotonic clock reading directly inside the existing time.Time struct.
When Go 1.9 was released on August 24, 2017, it included transparent monotonic time support. Under the new implementation, calling time.Now() captured both the current wall clock and the monotonic clock. When two time.Time values containing monotonic readings are subtracted using t.Sub(u) or time.Since(t), Go automatically computes the duration using the monotonic counter, insulating interval measurements from wall-clock shifts. The standard library documentation notes that if a timestamp is serialized, parsed from a string, or stripped of its monotonic reading, subtraction falls back to the wall clock.
The 2017 leap second outage demonstrated how a subtle divergence between calendar time and physical elapsed time could propagate through mathematical smoothing into an operational failure. Go 1.9 put the distinction into its existing time type: calendar timestamps tell software when an event happened; monotonic readings measure how long the event took.
Related stories
Sources
- How and Why the Leap Second Affected Cloudflare DNS, John Graham-Cumming
- Proposal: Monotonic Elapsed Time Measurements in Go, Russ Cox
- Go 1.9 is Released, Francesc Campoy
- Go 1.9 Release Notes
- Package time Documentation, Go Programming Language
- Featured image: Raysonho @ Open Grid Scheduler / Grid Engine; Wikimedia Commons; CC0 (adapted from the original)
SIGNAL TITANS
Leave a Reply