Wednesday, October 29, 2008

TCP Timestamps & 2.6.27 -- Why Ubuntu Put Out a Day Zero Security Update.

As the Ubuntu Intrepid release came down to the wire we ended having a serious bug (LP #264019). This bug was very difficult due to the way it manifested itself. First some background.

For a number of years the Linux Kernel had something called TCP Timestamping in the kernel. In 2.6.27 in the rc1 timeframe upstream did some TCP stack fixes and one of these broke some very old consumer grade DSL modems and routers. Keep in mind the fixes in question are technically correct, they follow the requisite IETF RFCs. It was this old consumer grade equipment that was at fault. All this is documented in kernel bugzilla in bug #11721. In the end a patch was developed that reset the TCP ordering.

Ok, after all this why is this such a big deal? Timing and the nature of the bug. A user reported that without this patch they could not connect to our archive servers over the Internet. This posed a problem for any user that had the old hardware. They would be unable to get the fix via the normal update method. Not a good thing.

So the next question would be why not just add the kernel patch? Thats where the timing issue comes in. We were at a point in the release cycle where to spin, test and validate a new kernel would have delayed the release up to a week.

We decided to go with a temporary workaround. The workaround would all the affected users the ability to get the fixed kernel. In parallel we prepared a security kernel that was ready in the archive by the time the Intrepid images hit the mirrors. The security kernel turns off the workaround we put into the procps package, and contains only the patch to fix this issue.

Decisions like this are made all the time by Distribution vendors. Its walking a fine line between whats best for the users and the amount of work, cost and end user expectation. We don't take issues like this lightly, all parts of the Ubuntu team and the highest levels of Canonical management are involved.

I hope this helps clairify things for people.

P.S. Its currently 14:43 London time as I write this and the security kernel has not yet hit the archive. Don't worry its making in the process of being published. It hould the archive shortly.


~pete

1 comment:

Marques Johansson said...

I reported LP Bug 264019 so it's only fitting that I get first post!

My out-of-state family are guinea pigs to my household Linux sociotechnological experiment. Sure my younger brothers hate that some of their Windows games don't work out of the box, but they don't mind so long as I can get it working in Wine by the end of the day. For comparison, the same games wouldn't work in Vista without additional effort (late 90s games with CD copy protection - LoTR2, RA, etc).

When they told me that Youtube wouldn't work I chalked it up as their typical sky-is-falling wolf crying. It turned out to be a serious issue. One that would have turned many people away from Ubuntu.

I think the resolution taken with this bug was the correct one for the reason stated: that updating would be difficult without it. But also because this bug would prevent millions(?) of Live CD users from considering Ubuntu long enough to enjoy it.

I am worried about the ramifications for business users, as TCP time-stamping has importance for high-bandwidth applications. The IT managers and admins I have palled around with in the past are very wary of kernel upgrades and (as history shows) rightly so. I hope this doesn't prevent any high-end adoption of 8.10.