Debian Bug Screws us All

This morning, I spotted this nasty tidbit on Slashdot: Debian Bug Leaves Private SSL/SSH Keys Guessable

It turns out a maintainer of the OpenSSL package on Debian removed the "seeding" of the random number generator that is used to generate, among other things, SSH keys. For those unfamiliar with random number generators, they work by generating a sequence of pseudo-random numbers based on some initial seed. The default value most programmers use when seeding their random number generators is simply the time, because it changes quickly and ensures a great deal of variability in what the generated random sequence of numbers will look like. If you seed your random number generator with the same number every time, you'll end up with the same sequence of numbers being generated over and over again - It won't be random at all!

What this means to Debian users is that your SSH keys are not random, and they're much easier to crack/guess because of this. Because Ubuntu is based on Debian, and the OpenSSL packages are relatively untouched by the Ubuntu maintainers, this bug also affects all Ubuntu users.

Both Debian and Ubuntu have released security updates which fix the problem and ensure that any future keys that are generated have the expected level of security. However, keys that have already been generated need to be expired and replaced.

Fortunately, the Ubuntu update that you will receive through update-manager takes care of this for you. For a desktop user, this is sufficient. For system administrators who might use SSH keys widely, it's a massive pain in the ass.

There's much more to this story though - A Slashdot user dug up the original Debian bug report that lead to the "fix" that removed the seeding. The OpenSSL developers used uninitialized memory to seed their random number generator, which caused a warning in Valgrind that someone playing with the code noticed. Valgrind is a tool to help find memory leaks and memory corruption (ie. programmers' mistakes). However, the original code wasn't erroneous at all - The programmer that wrote that code must have asserted that uninitialized memory has more "randomness" than the time, which may or may not have been a good assumption. Regardless, it's clear that the real mistake was "fixing" this code without fully understanding the consequences, and the Debian package maintainers (and the developer that submitted the patch) are at fault.

What happened here is a nightmare scenario for an open source software developer like myself. When my development team makes a release, we do some distribution packaging ourselves, but we also count on other people to make packages for other distributions. If any of those package maintainers modify our software in any way, we can no longer guarantee the quality of our software. If this sounds familiar, you might remember that this is exactly why Mozilla wanted Debian to stop using the Firefox name back in 2006. Mozilla wanted to ensure that all of their users got "Firefox", not "Firefox plus Joe Blow's crappy tweaks", and I completely agree with them. As more open source projects grow and become professionally run, I can see this becoming a more common issue in the future.

Finally, there is the question of whether or not the OpenSSL vulnerability was introduced intentionally. To give the poor guy the benefit of the doubt, I think it was an honest mistake. He fixed something that he thought was broken, and it turns out he was wrong - an understandable, human mistake. The uploader that approved his change probably should have caught the mistake, but again, he too was also only human. Is it possible that this was intentional? Sure. Is it possible that this could be used as a blueprint for future open source sabotage? Absolutely.

Are we any less likely to see security flaws introduced like this again? Absolutely not. It's the process through which packages are maintained and updated that is broken here.

The solution? Discuss.


Disclaimer: Don't go on a witch-hunt for the Debian guys who made mistakes here. Stuff like this happens, and if it were you or I in their shoes, we may have made the exact same mistake. I stress once again that it's the process of distribution package management that is flawed, not the people involved.