OpenSSL after Heartbleed

Did you know...? LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net.

Rich Salz and Tim Hudson started off their LinuxCon Europe 2016 talk by stating that April 3, 2014 shall forever be known as the "re-key the Internet date." That, of course, was the day that the Heartbleed vulnerability in the OpenSSL library was disclosed. A lot has happened with OpenSSL since that day, to the point that, Salz said, this should be the last talk he gives that ever mentions that particular vulnerability. In the last two years, the project has recovered from Heartbleed and is now more vital than ever before.

Ever since Heartbleed, every severe vulnerability has had to feature its own catchy name, web site, and logo. It is good, in that it has forced security researchers to develop some artistic sense. Seriously, though, short names for vulnerabilities are useful. Saying that a system "is susceptible to Heartbleed" makes sense to almost everybody. That is much less true if one talks about "susceptibility to CVE-whatever" instead. Heartbleed was also the first general defect that made it onto the front pages of the mainstream press. Salz suggested that it may end up outlasting the Kardashians.

At its core, Heartbleed was a simple bug, a missing buffer-length check. That resulted in the disclosure of some arbitrary memory that could be used to extract session and key information from a system running OpenSSL. This bug had been in place for three years when it was disclosed; nobody had managed to see it. The OpenSSL developers missed it, as did external security reviewers and code analysis tools (though, Salz noted, those tools all caught it three days after its disclosure). It was an ordinary bug, not as bad as some of its contemporaries ("goto fail" for example), but it got a lot of attention, perhaps because it, unlike many bugs, affected both server and client systems.

OpenSSL asleep

At the time of Heartbleed, the OpenSSL project had gone into a nearly moribund state. There were no policies for dealing with issues, and releases were not announced. There was no recognition that users might have to respond to releases containing security fixes. The source was complex and arcane; it "made procmail look pretty." There were almost no comments; indeed, there are still almost no comments, though it is starting to get better. The code was hard to maintain and hard to contribute to, especially for developers in the US due to crypto export issues. There were two overworked developers doing almost all of the work and making little money in the process; donations to the project were less than $2,000/year.

How did things get to this point? The project had put almost no time into building its community. It used mailing lists based on an old Majordomo server; they were unsearchable among other problems. Maintainership was static, with a great deal of wariness around the idea of letting anybody else in. The existing developers were strongly driven by the need to obtain consulting money, which caused them to focus on FIPS 140 certification. The project had no ability to make, much less adhere to, plans. The developers had fallen into a mindset that the easiest way to avoid breaking things was to not change them in the first place, so changes of any type were discouraged.

The advent of Heartbleed led immediately to a lot of negative feedback and hard questions. How can the project be trusted after such a vulnerability? When will the next one hit? These are, Salz said, good questions to ask.

After Heartbleed

One of the first after-effects of the Heartbleed disclosure was a wider recognition of the existence of critical, but underfunded projects. In response, the Linux Foundation launched the Core Infrastructure Initiative (CII) as a way of getting resources to those projects. That has helped OpenSSL, among others, to get back into a healthy state.

Before April 2014, OpenSSL had two primary developers, both of whom were volunteers, and no decision-making process. As of December of that year, the project had 15 members, two of whom are paid full-time by CII and two others who are paid from donations made directly to the project. There is a set of "very formal" decision processes in place now.

The group had its first in-person meeting that year in Düsseldorf, allowing them to get to know each other — a critical part of making a group functional. They drafted a set of major policies, covering release strategies, security, coding style, and more. The project has become much more transparent, in contrast to the insularity that prevailed before. Now, Salz said, only the OpenSSL data structures are opaque. Much of the project's work runs on GitHub now; email traffic on the lists has increased and has also become much more useful.

In 2016 (so far), the project has seen 3,246 commits. It has put out one major release, 15 bug-fixing releases, and responded to 29 CVE numbers. In general, it shows a lot more activity from many more people and has a far better sense of community. Developers have gotten serious about closing issues reported against the project which, Salz said, is an important step toward building a community. If issues go without responses, users will eventually stop reporting them. The goal is to respond to all reports within four days; the actual response time remains about three times that number, but it's getting closer.

The support policies for OpenSSL release are now well defined. The 1.1.0 release will receive support through September 30, 2018, while 1.0.2, the project's first long-term supported release, will be maintained through the end of 2019. Support for 1.0.1 is limited to security fixes now, and will end entirely at the end of 2016. All older versions are unsupported now. It is important, they said, to drop support for older releases, giving the project the resources to move forward.

Along with more development, the project has more people who are actively looking for problems. There is more fuzz testing going on, and the static analysis tools have been updated; indeed, there is more automated testing in general. A recently added tool will modify the code to invert the sense of the conditions in if statements to see whether the test suite catches the resulting bug; if it doesn't, there is a coverage gap in the test suite. All code must be formally reviewed before being committed.

Future plans and lessons learned

There is now a posted project roadmap to work from. Currently, some 1,100 forks of the project exist on GitHub. At the top of the list for future development is support for TLS 1.3. The project plans to move to the Apache2 license, though they couldn't say just when that will happen. More testing is in the plans.

Then there is the ongoing issue of FIPS 140 certification. This certification is mandatory for most vendors selling software to the United States government; meeting this need essentially funded the project for five years. The validation process is, they said, a time-consuming and irritating thing to have to go through; it takes a couple of years and is expensive. It is an ongoing process as well, in order to keep up with changes in the software; FIPS 140 certification is not something that can be done once and forgotten about.

The OpenSSL FIPS 2.0 certification module applies to the 1.0.x release series. There is a new FIPS module coming, supported by SafeLogic; it will work with the 1.1.0 release. In general, the project is working to make the FIPS-related changes less intrusive in the code base as a whole.

What was learned from the whole Heartbleed experience? No project can rely on any one individual to continually perform superhuman feats; that will always lead to disappointment in the end. Code reviewers have to actually look at the code; they can't just hope the the community will somehow perform a proper review. A project also cannot rely just on tools, they will never do the full job. Proper code review takes a lot of time, and it needs to be done by experienced developers.

Needless to say, OpenSSL is looking for contributors. Beyond contributing patches, interested developers can test the pre-releases, report bugs, and help to close bugs. The presenters concluded by saying that they would like users to get in touch, especially those who are distributing OpenSSL further downstream.

---

(Log in to post comments)