Wednesday, October 29, 2008

TCP Timestamps & 2.6.27 -- Why Ubuntu Put Out a Day Zero Security Update.

As the Ubuntu Intrepid release came down to the wire we ended having a serious bug (LP #264019). This bug was very difficult due to the way it manifested itself. First some background.

For a number of years the Linux Kernel had something called TCP Timestamping in the kernel. In 2.6.27 in the rc1 timeframe upstream did some TCP stack fixes and one of these broke some very old consumer grade DSL modems and routers. Keep in mind the fixes in question are technically correct, they follow the requisite IETF RFCs. It was this old consumer grade equipment that was at fault. All this is documented in kernel bugzilla in bug #11721. In the end a patch was developed that reset the TCP ordering.

Ok, after all this why is this such a big deal? Timing and the nature of the bug. A user reported that without this patch they could not connect to our archive servers over the Internet. This posed a problem for any user that had the old hardware. They would be unable to get the fix via the normal update method. Not a good thing.

So the next question would be why not just add the kernel patch? Thats where the timing issue comes in. We were at a point in the release cycle where to spin, test and validate a new kernel would have delayed the release up to a week.

We decided to go with a temporary workaround. The workaround would all the affected users the ability to get the fixed kernel. In parallel we prepared a security kernel that was ready in the archive by the time the Intrepid images hit the mirrors. The security kernel turns off the workaround we put into the procps package, and contains only the patch to fix this issue.

Decisions like this are made all the time by Distribution vendors. Its walking a fine line between whats best for the users and the amount of work, cost and end user expectation. We don't take issues like this lightly, all parts of the Ubuntu team and the highest levels of Canonical management are involved.

I hope this helps clairify things for people.

P.S. Its currently 14:43 London time as I write this and the security kernel has not yet hit the archive. Don't worry its making in the process of being published. It hould the archive shortly.