One Missing Length Check Exposed Half a Million Sites. OpenSSL Got About $2,000 a Year in Donations.

Imagine asking a server a trivial question: “Are you still there? Please repeat this one-letter word back to me.” Now add a lie. You say the word is 64,000 characters long.

In April 2014, thousands of servers believed you. They sent back the letter, followed by whatever happened to sit next to it in memory: usernames, passwords, session cookies, sometimes the secret keys that prove a website is who it claims to be. The bug got a name with a logo, Heartbleed, and it turned a quiet piece of open-source plumbing into front-page news.

One missing comparison

The code lived in OpenSSL, the library that handles encryption for a huge share of the web. TLS has an optional “heartbeat” feature, a way to check that the other end of a connection is still alive. A client sends a small payload and says how long it is; the server echoes it back.

The flaw was that the server trusted the stated length. OpenSSL’s own advisory describes it as a missing bounds check: a request could make the server return up to 64 KB of its memory at a time, over and over. The Heartbleed site added the worst part for defenders: exploiting it “does not leave any trace of anything abnormal happening to the logs.”

It was an implementation bug, not a flaw in TLS or HTTPS themselves. The protocol was fine. One check was missing.

A contribution, a review, two years

The heartbeat code was written by Robin Seggelmann, a researcher who was contributing a feature. By the commonly reported timeline it was committed on 31 December 2011 and shipped in OpenSSL 1.0.1 on 14 March 2012. Versions 1.0.1 through 1.0.1f were vulnerable.

When the bug became public, Seggelmann said he had “missed validating a variable containing a length.” A reviewer missed it too. This is the uncomfortable part of the story: nobody was careless in an obvious way. A feature was proposed, looked at and merged, as open source is supposed to work.

The discovery was nearly simultaneous. Neel Mehta of Google reported it to OpenSSL on 1 April 2014. A team at Codenomicon found it independently and logged it on 3 April. The fix, version 1.0.1g, shipped with the public disclosure on 7 April. Nobody can say whether anyone else found it earlier. Netcraft’s survey, as cited by Wikipedia, estimated that about 17 percent of web servers with trusted certificates, roughly half a million, were vulnerable at disclosure.

The funding question

The more interesting question came next: who was looking after this code? In April 2014 the OpenSSL Software Foundation’s Steve Marquess wrote that donations to the project were about $2,000 a year, according to ITPro’s summary of his post. He said the project had one full-time developer and would need at least six. The numbers come through that secondary report rather than the post itself.

Careful, because the popular version is a little off. OpenSSL was not “run by unpaid volunteers.” It had a foundation, support contracts that supplied most of its revenue, and one paid full-time developer. The $2,000 was the donation figure, not the whole budget. BuzzFeed’s profile of the team put it more modestly: “only really one full-time developer.”

The money arrived after the crisis. Donations rose to about $9,000, mostly from individuals. On 24 April 2014 the Linux Foundation announced the Core Infrastructure Initiative: 13 founding companies pledging $100,000 a year each for three years, nearly $4 million in total, which funded two full-time OpenSSL core developers.

What it was not

Heartbleed attracted conspiracy talk. Bloomberg cited two unnamed people saying the NSA had known about it. The NSA denied it, and former White House adviser Richard Clarke told Reuters it did not know. Seggelmann called the bug an honest mistake. There is no evidence of intent, and the NSA claim remains unproven.

It also did not end in April 2014. On 11 July 2019 Shodan still counted 91,063 vulnerable devices.

Meanwhile, OpenBSD’s developers forked OpenSSL into LibreSSL, announced on 22 April 2014. Wikipedia reports that more than 90,000 lines of C were removed in the first week.

The mental model

The lesson is not “open source is unsafe.” The code was public and reviewed; the review is what failed to catch one line. The lesson is about attention: critical dependencies get attention in proportion to funding and staffing, not in proportion to how much we rely on them.

A fair question for your own stack: which library would take your system down if it had one missing check, and how many people are actually paid to look at it?

Explore

Stay connected