Backing out the procps change... how it works

I've had some questions on how we backed out the procps change, specifically the mechanism that was used. Here is how it works;

Both the kernel package and the procps package are security updates. The procps package removed the file /etc/sysctl.d/10-tcp-timestamps-workaround.conf (at least as long as the user had not modified it; it takes care in its preinst to check for this, since clobbering explicitly modified configuration files would be a clear policy violation). However it does not "undo" the setting on the fly in the running kernel. The new kernel is installed and upon rebooting you will get tcp timestamps turned back on, since the configuration file that turned them off is no longer present.

It's true that it's technically possible for a user to upgrade procps but not the kernel, or vice versa. However, we felt that it wasn't worth the gymnastics required to cope with this, bearing in mind that there are many different kernel package names in 8.10, many of which never suffered from this problem in the first place, and that the TCP timestamps setting is not undone until you reboot. Particularly given that the kernel update doesn't change ABI and therefore doesn't change package name, the overwhelming majority of users will just apply both security updates at once. We made sure that both the kernel and procps were published with all their binaries at the same time.

Thanks to Colin Watson for the technial explination.

~pete

TCP Timestamps & 2.6.27 -- Why Ubuntu Put Out a Day Zero Security Update.

As the Ubuntu Intrepid release came down to the wire we ended having a serious bug (LP #264019). This bug was very difficult due to the way it manifested itself. First some background.

For a number of years the Linux Kernel had something called TCP Timestamping in the kernel. In 2.6.27 in the rc1 timeframe upstream did some TCP stack fixes and one of these broke some very old consumer grade DSL modems and routers. Keep in mind the fixes in question are technically correct, they follow the requisite IETF RFCs. It was this old consumer grade equipment that was at fault. All this is documented in kernel bugzilla in bug #11721. In the end a patch was developed that reset the TCP ordering.

Ok, after all this why is this such a big deal? Timing and the nature of the bug. A user reported that without this patch they could not connect to our archive servers over the Internet. This posed a problem for any user that had the old hardware. They would be unable to get the fix via the normal update method. Not a good thing.

So the next question would be why not just add the kernel patch? Thats where the timing issue comes in. We were at a point in the release cycle where to spin, test and validate a new kernel would have delayed the release up to a week.

We decided to go with a temporary workaround. The workaround would all the affected users the ability to get the fixed kernel. In parallel we prepared a security kernel that was ready in the archive by the time the Intrepid images hit the mirrors. The security kernel turns off the workaround we put into the procps package, and contains only the patch to fix this issue.

Decisions like this are made all the time by Distribution vendors. Its walking a fine line between whats best for the users and the amount of work, cost and end user expectation. We don't take issues like this lightly, all parts of the Ubuntu team and the highest levels of Canonical management are involved.

I hope this helps clairify things for people.

P.S. Its currently 14:43 London time as I write this and the security kernel has not yet hit the archive. Don't worry its making in the process of being published. It hould the archive shortly.


~pete

Bureaucracy

I received a comment on my blog, referencing the Kernel Team requiring a LP bug prior to committing a patch to the tree. The person commenting called it "Bureaucracy". I thought this would be the common reaction so I wanted to raise it here. My response is below...

Its not about bureaucracy, its about accountability. We add numerous patches during a cycle. While most do have a LP entry there are quite a few that don't. The problem manifests itself when someone can't remember why they added the patch. Obviously it was intended to fix a problem. There have been occasions where the patch, while fixing one bug introduced a much bigger one. When going back through the history trying to figure out why we would add it, the usual answer is "it seemed like a good idea at the time".

We are striving to stick as closely as possible to upstream, and every patch we add, whether a backport from a newer kernel or a sauce patch needs to have a bug attached. This is common a common "Change Control Measure". If it worth adding it should have a valid bug attached.

~pete

Linux Plumbers Conference Recap.

Seems like these days I get most of my blogging done on the plane going from here to there. This entry is no different.

I spent the last week with the kernel team in Portland. We held a mini-sprint on Monday and Tuesday, and Wednesday thru Friday were consumed with the Linux Plumbers Conference.

Some developments out of the mini-sprint:

* We will be adding git commit hooks for all check-ins. If a LP bug number is not referenced in the commit template, the commit won't happen. This will prevent stray patches from going to the tree with out a bug, explanation/rationale & justification. Thanks to Launchpad's new API we can flip the state from "In Progress" to "Fixed Released" and drop the commits SHA1 id into a comment in the bug.

* More kernel QA. We will be hooking the build into Kees's automatic security testing framework. By having every commit needing a LP bug we can attach a test case to the bug and have it automatically inserted into the test harness. This will keep regressions from sneaking into the build and give us a higher QA on the kernel.

* The kernel team will be packaging up the latest bleeding edge upstream "Vanilla" kernel. There are some details still to be worked out, like where it will live. Will go in main or universe, a PPA... The point with this is so that users will have a pristine upstream kernel to test against. This is where the kernel team is doing new development anyway. If the issue is fixed we can quickly do a git bisect and figure out what patch is needed and backport it to the stable kernel if it makes sense. I'll write more about this one in a future blog posting.

The Plumbers Conference was outstanding given this was it's first year. The talks were good, focused and useful. For example Arjan's "Booting in 5 sec" was right on time. Some of the low hanging fruit faster boot stuff we can prob squeeze into Intrepid.

Canonical took a beating from Greg Kroah-Hartman of Novell during his Keynote Speech. He has had a long running issue with Canonical not "giving back" and "leeching". Both Matt Zimmerman and I took this opportunity to talk with Greg.

It was a productive conversation and I think we have come to some common ground. Our plan as we hire more kernel developers is to work in upstream head and work on kernel bits that are of interest to Ubuntu and Canonical. We will be pushing patches and fixes to the upstream sub-system maintainers. We talked briefly on what he considers "good upstream" citizenry. Greg offered his help and advice going forward. We will continue the discussion in email.

We'll the battery is about dead so I need to save and shutdown.

~pete

Thoughts about growing up.

This was a seriously busy weekend for work. We were trying to debug hardware and software problems on a particular model of computer. We were running into some late breaking issues and for one reason or another, all the test units were at my house.

After hours of debugging and replication we finally found the issues & created fixes. Now comes the fun part. Testing to validate that we really did fix the issue. I had all the hardware, four of the units to be exact. One of the issue only manifested itself during a cold boot. To test it we had to boot the machine, wait till it came up to the GUI and shut it back down and start all over. No big deal right? Sounds easy huh? Try it across four computers 100 times per computer. Not hard, but time consuming.

That why I was blessed with children. I guess I got that from my parents (I was the family remote control until I moved out!) My son Matt (he's 11) was instrumental to the replication process. He dutifully sat behind two of the computers trying to get it to break while I had the other two and tried to replicate the other problem. And sure enough after a few hours we replicated it and worked with the rest of the team to develop the fixes.

Once we had the fixes he led the charge on testing, he rounded up his sister and they sat in font of the computers, flashing the BIOS, installing Ubuntu from USB flash drives and started testing. Six hours later the testing was complete.

The events of this past weekend got me thinking about when I was a kid. I remember working with my dad on the car, fixing lawnmower engines, hunting, fishing. Thats where my dad taught me about being a man. One of his favorite sayings was "Don't do anything half-assed, your as good as your word and the job you do". Its a different time and place today from when I grew up. My dad worked in factories and was a manual laborer. I work with computers, not much manual labor these days and I don't work on my car, I pay someone to mow my lawn. I remember being about 12 or 13 in the middle of the winter under our family car, a 1972 Ford LTD in the driveway with snow all around. We had to rebuild the transmission. Does anyone do that anymore?

As I thought about all this, I realized while I do spend time with my kids, its not the same. For example Dad taught me bar room etiquette, how to shoot pool, throw darts and in general to be a good citizen in the rough and tumble world. I grew up in Prescott Arizona and there is one of the oldest saloons in the town called the Palace Bar. Dad and I would meet some of his buddies on Saturday afternoons and he would bring me along. I was all of about 10 years old. He would prop me up on the bar stool order me a Coke and meet with his friends. He would teach me the finer points of how to behave. Today you can't even bring a kid into a bar. No wonder people don't know how to act in that type of setting. How many time have you seen a young kid acting the fool?

I guess in a different way I'm passing the mantle to my son. He can do things with computers and Linux that most grown adults can't do. We don't work on cars, but we do work on computers together. We do fish and hunt (not enough as we would like) so some things don't change.

I'm off this week to the Linux Plumbers conference in Portland, in fact I'm writing this on the plane on the way there. When I get back I think I take my son fishing and start tell him all the things his Grandfather taught me and more. I know times have changed and its not quite the same but hopefully as he gets older he'll remember one day and say "Wow dad was right". I know I say it all the the time...

~pete

New Kernel Upload 2.6.27-3.4

rtg (Tim Gardner) just uploaded Intrepid 2.6.27-3.4 rebased on 2.6.27-rc6 look for it to hit the repos soon.

Changelog:

Changes:
linux (2.6.27-3.4) intrepid; urgency=low

  [ Colin Ian King ]

  * SAUCE: fix kernel oops in VirtualBox during paravirt patching
    - LP: #246067
  * SAUCE: qc-usb: Enable Logitech QuickCam Messenger
    - LP: #209901
  * SAUCE: appleir: Enable driver for new MacBook Pro
    - LP: #157919

  [ Tim Gardner ]

  * Enabled CONFIG_DEBUG_RODATA=y

  [ Upstream Kernel Changes ]

  * Revert "ALSA: hda - Added model selection for iMac 24""
  * Revert "x86: fix HPET regression in 2.6.26 versus 2.6.25, check hpet
    against BAR, v3"
  * Revert "[ARM] use the new byteorder headers"
  * Revert "mac80211: Use IWEVASSOCREQIE instead of IWEVCUSTOM"
  * Revert "crypto: camellia - Use kernel-provided bitops, unaligned access
    helpers"
  * svcrdma: Fix race between svc_rdma_recvfrom thread and the dto_tasklet
  * sched, cpuset: rework sched domains and CPU hotplug handling (v4)
  * ACPI: Fix now signed module parameter.
  * ACPI: Change package length error to warning
  * ACPI: Fix now signed module parameter.
  * ACPI: Fix typo in "Disable MWAIT via DMI on broken Compal board"
  * acpi: add checking for NULL early param
  * UBIFS: fix zero-length truncations
  * Input: bcm5974 - add maintainer entry
  * sh64: re-add the __strnlen_user() prototype
  * sh: fix ptrace_64.c:user_disable_single_step()
  * PNPACPI: ignore the producer/consumer bit for extended IRQ descriptors
  * UBIFS: always read hashed-key nodes under TNC mutex
  * UBIFS: allow for racing between GC and TNC
  * [CIFS] Fix plaintext authentication
  * sparc32: Implement smp_call_function_single().
  * sh: crash kernel resource fix
  * sh: fix kexec entry point for crash kernels
  * sh: fix platform_resource_setup_memory() section mismatch
  * sh: update Migo-R defconfig
  * sh: update AP325RXA defconfig
  * sh: fix semtimedop syscall
  * cifs: fix O_APPEND on directio mounts
  * [CIFS] update cifs change log
  * [CIFS] Turn off Unicode during session establishment for plaintext
    authentication
  * ACPI: thinkpad-acpi: wan radio control is not experimental
  * sparc: Fix resource flags for PCI children in OF device tree.
  * remove blk_register_filter and blk_unregister_filter in gendisk
  * ALSA: oxygen: fix distorted output on AK4396-based cards
  * ipv6: When we droped a packet, we should return NET_RX_DROP instead of
    0
  * pkt_sched: Fix locking of qdisc_root with qdisc_root_sleeping_lock()
  * net: Unbreak userspace usage of linux/mroute.h
  * Don't trigger softlockup detector on network fs blocked tasks
  * Resource handling: add 'insert_resource_expand_to_fit()' function
  * sparc64: setup_valid_addr_bitmap_from_pavail() should be __init
  * UBIFS: do not update min_idx_lebs in stafs
  * UBIFS: push empty flash hack down
  * UBIFS: remove incorrect index space check
  * UBIFS: improve statfs reporting
  * UBIFS: fix assertion
  * UBIFS: add forgotten gc_idx_lebs component
  * UBIFS: introduce LEB overhead
  * UBIFS: improve statfs reporting even more
  * UBIFS: fill f_fsid
  * drm/radeon: downgrade debug message from info to debug.
  * Remove invalidate_partition call from do_md_stop.
  * Fix problem with waiting while holding rcu read lock in md/bitmap.c
  * ALSA: hda: Distortion fix for dell_m6_core_init
  * ALSA: ASoC: fix pxa2xx-i2s clk_get call
  * block: restore original behavior of /proc/partition when there's no
    partition
  * debugobjects: fix lockdep warning
  * avr32: Fix lockup after Java stack underflow in user mode
  * avr32: pm_standby low-power ram bug fix
  * nfsd: fix compound state allocation error handling
  * sunrpc: fix possible overrun on read of /proc/sys/sunrpc/transports
  * nfsd: fix buffer overrun decoding NFSv4 acl
  * audit: Moved variable declaration to beginning of function
  * Fix modules_install on RO nfs-exported trees.
  * Remove '#include ' from mm/page_isolation.c
  * dabusb_fpga_download(): fix a memory leak
  * [MTD] mtdchar.c: Fix regression in MEMGETREGIONINFO ioctl()
  * ALSA: hda - Fix ALC663 auto-probe
  * ALSA: hda - Add mic-boost controls to ALC662/663 auto configuration
  * Un-break printk strings in x86 PCI probing code
  * kernel/resource.c: fix new kernel-doc warning
  * softlockup: minor cleanup, don't check task->state twice
  * fix typo in arch/parisc/hpux/fs.c
  * m68k: atari_keyb_init operator precedence fix
  * ACPI: Fix typo in "Disable MWAIT via DMI on broken Compal board"
  * don't diff generated firmware files
  * IDE: compile fix for sff_dma_ops
  * IDE: palm_bk3710: fix compile warning for unused variable
  * ide: fix hwif_to_node()
  * palm_bk3710: improve IDE registration
  * ide-disk: remove stale init_idedisk_capacity() documentation
  * ide/Kconfig: mark ide-scsi as deprecated
  * net/wireless/Kconfig: clarify the description for
    CONFIG_WIRELESS_EXT_SYSFS
  * iwlwifi: do not use GFP_DMA in iwl_tx_queue_init
  * iwlwifi: workaround interrupt handling no some platforms
  * iwlwifi: fix apm_stop (wrong bit polarity for FLAG_INIT_DONE)
  * iwlwifi: fix 64bit platform firmware loading
  * orinoco: Multicast to the specified addresses
  * wireless/libertas/if_cs.c: fix memory leaks
  * mac80211: Fix debugfs union misuse and pointer corruption
  * rt2x00: Compiler warning unmasked by fix of BUILD_BUG_ON
  * ath9k: Incorrect key used when group and pairwise ciphers are
    different.
  * ath9: Fix ath_rx_flush_tid() for IRQs disabled kernel warning message.
  * net/xfrm: Use an IS_ERR test rather than a NULL test
  * ipv: Re-enable IP when MTU > 68
  * NTFS: update homepage
  * mm: make setup_zone_migrate_reserve() aware of overlapping nodes
  * VFS: fix dio write returning EIO when try_to_release_page fails
  * acer-wmi: remove debugfs entries upon unloading
  * mm/bootmem: silence section mismatch warning -
    contig_page_data/bootmem_node_data
  * MAINTAINERS: add a maintainer for the BCM5974 multitouch driver
  * 8250: improve workaround for UARTs that don't re-assert THRE correctly
  * mmc: at91_mci: don't use coherent dma buffers
  * pid_ns: zap_pid_ns_processes: fix the ->child_reaper changing
  * pid_ns: (BUG 11391) change ->child_reaper when init->group_leader exits
  * cirrusfb: check_par fixes
  * devcgroup: fix race against rmdir()
  * mm: show quicklist usage in /proc/meminfo
  * mm: size of quicklists shouldn't be proportional to the number of CPUs
  * ipc: document the new auto_msgmni proc file
  * hp-wmi: update to match current rfkill semantics
  * hp-wmi: add proper hotkey support
  * tdfxfb: fix SDRAM memory size detection
  * tdfxfb: fix frame buffer name overrun
  * rtc_time_to_tm: fix signed/unsigned arithmetic
  * ibft: fix target info parsing in ibft module
  * sysfs: document files in /sys/firmware/sgi_uv/
  * rtc-cmos: wake again from S5
  * pm_qos_requirement might sleep
  * drivers/char/random.c: fix a race which can lead to a bogus BUG()
  * ipsec: Fix deadlock in xfrm_state management.
  * [x86] Fix TSC calibration issues
  * tipc: Don't use structure names which easily globally conflict.
  * sparc64: Fix IPI call locking.
  * [ARM] omap: fix gpio.c build error
  * sparc64: Prevent sparc64 from invoking irq handlers on offline CPUs
  * powerpc: Fix uninitialised variable in VSX alignment code
  * powerpc: Only make kernel text pages of linear mapping executable
  * powerpc: Make sure _etext is after all kernel text
  * powerpc: Work around gcc's -fno-omit-frame-pointer bug
  * powerpc: Fix build error with 64K pages and !hugetlbfs
  * powerpc: Fix for getting CPU number in power_save_ppc32_restore()
  * UBIFS: amend f_fsid
  * net/usb/pegasus: avoid hundreds of diagnostics
  * ixgbe: initialize interrupt throttle rate
  * pcnet-cs, axnet_cs: add new IDs, remove dup ID with less info
  * netxen: Remove workaround for chipset quirk
  * Split up PIT part of TSC calibration from native_calibrate_tsc
  * iwlwifi: W/A for the TSF correction in IBSS
  * iwlwifi: fix hidden ssid discovery in passive channels
  * iwlwifi: remove false rxon if rx chain changes
  * iwlwifi: fix station mimo power save values
  * iwlwifi: fix rx_chain computation
  * iwlwifi: fix Tx cmd memory allocation failure handling
  * iwlwifi: call apm stop on exit
  * iwlwifi: fix STATUS_EXIT_PENDING is not set on pci_remove
  * ath9k: Fix TX status reporting
  * ath9k: Fix TX control flag use for no ACK and RTS/CTS
  * V4L/DVB (8555): au8522: add mechanism to configure IF frequency for vsb
    and qam
  * V4L/DVB (8556): au0828: add support for Hauppauge Woodbury
  * V4L/DVB (8598): au8522: clean up function au8522_set_if
  * V4L/DVB (8599): au8522: remove if frequency settings from vsb/qam
    modulation tables
  * V4L/DVB (8600): au0828: explicitly set 6 MHz IF frequency in
    hauppauge_hvr950q_config
  * V4L/DVB (8629): v4l2-ioctl: do not try to handle private V4L1 ioctls
  * V4L/DVB (8633): ivtv: update ivtv version number
  * V4L/DVB (8648): ivtv: improve CC support
  * V4L/DVB (8660): gspca: Simplify the scan of URB packets in pac7311.
  * V4L/DVB (8661): gspca: Bug in the previous changeset about pac7311.
  * V4L/DVB (8663): gspca: Webcam 0c45:6128 added in sonixj.
  * V4L/DVB (8664): gspca: The bridge/sensor of the webcam 093a:2621 is a
    PAC 7302.
  * V4L/DVB (8665): gspca: Fix the 640x480 resolution of the webcam
    093a:2621.
  * V4L/DVB (8666): gspca: Bad scanning of frames in pac7311.
  * V4L/DVB (8667): gspca: Bad probe of Z-Star/Vimicro webcams with pas106
    sensor.
  * V4L/DVB (8668): gspca: Conflict GSPCA / ET61X251 for the webcam
    102c:6251.
  * V4L/DVB (8669): gspca: Add white balance control for spca561 rev 012A.
  * V4L/DVB (8671): gspca: Remove the unused field 'dev_name' of the device
    structure.
  * V4L/DVB (8672): gspca: Big rewrite of spca561.
  * V4L/DVB (8673): gspca: Bad frame scanning again and bad init in
    pac7311.
  * V4L/DVB (8674): gspca: Webcam 0c45:612e added in sonixj.
  * V4L/DVB (8675): gspca: Pixmap PJPG (Pixart 73xx JPEG) added, generated
    by pac7311.
  * V4L/DVB (8678): Remove the dead CONFIG_RADIO_MIROPCM20{,_RDS} code
  * V4L/DVB (8681): v4l2-ioctl.c: fix warning
  * V4L/DVB (8682): V4L: fix return value of register video func
  * V4L/DVB (8701): cx18: Add missing lock for when the irq handler
    manipulates the queues
  * V4L/DVB (8703): gspca: Do controls work for spca561 revision 12a.
  * V4L/DVB (8705): gspca: Adjust some control limits in spca561.
  * V4L/DVB (8706): Make contrast and brightness work for pac7302.
  * V4L/DVB (8707): gspca: Colors, hflip and vflip controls added for
    pac7302.
  * V4L/DVB (8709): gspca: Fix initialization and controls of sn9x110 -
    ov7630.
  * V4L/DVB (8710): gspca: Bad color control in sonixj.
  * V4L/DVB (8711): gspca: Bad controls and quantization table of pac7311.
  * V4L/DVB (8712): gspca: Bad start of sonixj webcams since changeset
    a8779025e7e8.
  * V4L/DVB (8713): gspca: Bad color control again in sonixj.
  * V4L/DVB (8714): gspca: Bad start of sn9c110 and sensor om6802.
  * V4L/DVB (8715): gspca: Change the name of some webcam in the gspca doc.
  * V4L/DVB (8716): gspca: Bad start of sn9c110 and sensor ov7630.
  * V4L/DVB (8717): gspca: Frame buffer too small for small resolutions
    (sonixj and t613).
  * V4L/DVB (8718): gspca: suspend/resume added.
  * V4L/DVB (8719): gspca: Have VIDIOC_QUERYCTRL more compliant to the
    spec.
  * V4L/DVB (8720): gspca: V4L2_CAP_SENSOR_UPSIDE_DOWN added as a cap for
    some webcams.
  * V4L/DVB (8722): sms1xxx: fix typo in license header
  * V4L/DVB (8726): link tuner before saa7134
  * V4L/DVB (8727): V4L1: make PMS not autoprobe when builtin.
  * V4L/DVB (8728): 1-make-pms-not-autoprobe-when-builtin update
  * V4L/DVB (8749): Fix error code, when camera is not turned on by sonypi
  * V4L/DVB (8750): V4L: check inval in video_register_device_index()
  * V4L/DVB (8751): vivi: Fix some issues at vivi register routine
  * V4L/DVB (8757): v4l-dvb: fix a bunch of sparse warnings
  * V4L/DVB (8769): cx18: Simplify queue flush logic to prevent oops in
    cx18_flush_queues()
  * V4L/DVB (8778): radio: fix incorrect video_register_device result check
  * V4L/DVB (8779): v4l: fix more incorrect video_register_device result
    checks
  * V4L/DVB (8790): saa7115: call i2c_set_clientdata only when state !=
    NULL
  * V4L/DVB (8803): s5h1409: Enable QAM_AUTO mode
  * V4L/DVB (8804): s5h1411: Enable QAM_AUTO mode
  * V4L/DVB (8805): Steven Toth email address change
  * V4L/DVB (8809): gspca: Revert commit
    9a9335776548d01525141c6e8f0c12e86bbde982
  * V4L/DVB (8810): gspca: Compile error when CONFIG_PM not defined.
  * V4L/DVB (8812): gspca: Do pac73xx webcams work.
  * V4L/DVB (8813): gspca: Adjust SOF detection for pac73xx.
  * V4L/DVB (8814): gspca: Set DISABLED the disabled controls at query
    control time.
  * V4L/DVB (8815): gspca: Fix problems with disabled controls.
  * V4L/DVB (8816): gspca: Set disabled ctrls and fix a register pb with
    ovxxxx in sonixb.
  * V4L/DVB (8817): gspca: LED and proble changes in sonixb.
  * V4L/DVB (8818): gspca: Reinitialize the device on resume.
  * V4L/DVB (8819): gspca: Initialize the ov519 at open time and source
    cleanup.
  * V4L/DVB (8820): gspca: Change initialization and gamma of zc3xx -
    pas106.
  * V4L/DVB (8822): gspca: Change some subdriver functions for
    suspend/resume.
  * V4L/DVB (8823): gspca: H and V flips work for ov7670 only in ov519.
  * V4L/DVB (8824): gspca: Too much code removed in the suspend/resume
    changeset.
  * V4L/DVB (8825): gspca: More controls for pac73xx and new webcam
    093a:2624.
  * V4L/DVB (8826): gspca: Webcam Labtec 2200 (093a:2626) added in pac7311.
  * V4L/DVB (8827): gspca: Stop pac7302 autogain oscillation.
  * V4L/DVB (8828): gspca: Set the clock at the end of initialization in
    sonixj.
  * V4L/DVB (8829): gspca: Have a clean kmalloc-ated buffer for USB
    exchanges.
  * V4L/DVB (8830): gspca: Move some probe code to the new init function.
  * V4L/DVB (8831): gspca: Resolve webcam conflicts between some drivers.
  * V4L/DVB (8832): gspca: Bad pixelformat of vc0321 webcams.
  * V4L/DVB (8833): gspca: Cleanup the sonixb code.
  * V4L/DVB (8834): gspca: Have a bigger buffer for sn9c10x compressed
    images.
  * V4L/DVB (8835): gspca: Same pixfmt as the sn9c102 driver and raw Bayer
    added in sonixb.
  * V4L/DVB (8837): dvb: fix I2C adapters name size
  * V4L/DVB (8839): dib0700: add comment to identify 35th USB id pair
  * V4L/DVB (8840): dib0700: add basic support for Hauppauge Nova-TD-500
    (84xxx)
  * V4L/DVB (8842): vivi_release(): fix use-after-free
  * V4L/DVB (8843): tda10048_firmware_upload(): fix a memory leak
  * V4L/DVB (8844): dabusb_fpga_download(): fix a memory leak
  * bnx2x: Accessing un-mapped page
  * SELinux: memory leak in security_context_to_sid_core
  * x86: add io delay quirk for Presario F700
  * mmap: fix petty bug in anonymous shared mmap offset handling
  * x86: Change warning message in TSC calibration.
  * PCI: fix pbus_size_mem() resource alignment for CardBus controllers
  * [ARM] omap: fix build error in ohci-omap.c
  * [ARM] remove unused #include 
  * ACPI: Make Len Brown the ACPI maintainer again
  * fujitsu-laptop: fix regression for P8010 in 2.6.27-rc
  * ACPI: Avoid bogus timeout about SMbus check
  * acer-wmi: remove debugfs entries upon unloading
  * forgotten refcount on sysctl root table
  * V4L/DVB (8868): gspca: Support for vga modes with sif sensors in
    sonixb.
  * V4L/DVB (8869): gspca: Move the Sonix webcams with TAS5110C1B from
    sn9c102 to gspca.
  * V4L/DVB (8870): gspca: Fix dark room problem with sonixb.
  * V4L/DVB (8872): gspca: Bad image format and offset with rev072a of
    spca561.
  * V4L/DVB (8873): gspca: Bad image offset with rev012a of spca561 and
    adjust exposure.
  * V4L/DVB (8874): gspca: Adjust hstart for sn9c103/ov7630 and update
    usb-id's.
  * [ARM] omap: fix virtual vs physical address space confusions
  * V4L/DVB (8876): budget: udelay changed to mdelay
  * V4L/DVB (8877): b2c2 and bt8xx: udelay to mdelay
  * V4L/DVB (8880): PATCH: Fix parents on some webcam drivers
  * V4L/DVB (8881): gspca: After 'while (retry--) {...}', retry will be -1
    but not 0.
  * powerpc/spufs: Fix multiple get_spu_context()
  * powerpc/spufs: Fix race for a free SPU
  * Input: bcm5974 - small formatting cleanup
  * Input: bcm5974 - improve finger tracking and counting
  * Input: bcm5974 - add BTN_TOUCH event for mousedev benefit
  * Input: i8042 - make Lenovo 3000 N100 blacklist entry more specific
  * sh: resume_kernel fix for kernel oops built with CONFIG_BKL_PREEMPT=y.
  * sh64: resume_kernel fix for kernel oops built with
    CONFIG_BKL_PREEMPT=y.
  * i2c: fix i2c-sh_mobile timing issues
  * clockevents: prevent clockevent event_handler ending up handler_noop
  * clockevents: prevent endless loop in periodic broadcast handler
  * clockevents: enforce reprogram in oneshot setup
  * clockevents: prevent multiple init/shutdown
  * clockevents: prevent endless loop lockup
  * HPET: make minimum reprogramming delta useful
  * [MTD] [NAND] tmio_nand: fix base address programming
  * Fix conditional export of kvh.h and a.out.h to userspace.
  * async_tx: fix the bug in async_tx_run_dependencies
  * sched_clock: fix NOHZ interaction
  * sched: fix process time monotonicity
  * UBIFS: fix division by zero
  * UBIFS: make minimum fanout 3
  * [MIPS] Fix data bus error recovery
  * [MIPS] Fix WARNING: at kernel/smp.c:290
  * [MIPS] TXx9: Fix txx9_pcode initialization
  * [MIPS] TX39xx: Add missing local_flush_icache_range initialization
  * [MIPS] Probe initrd header only if explicitly specified
  * res_counter: fix off-by-one bug in setting limit
  * forcedeth: fix kexec regression
  * atmel_lcdfb: fix oops in rmmod when framebuffer fails to register
  * tracehook: comment pasto fixes
  * drivers/mmc/card/block.c: fix refcount leak in mmc_block_open()
  * x86: boot: stub out unimplemented CPU feature words
  * x86: add NOPL as a synthetic CPU feature bit
  * x86: use X86_FEATURE_NOPL in alternatives
  * clockevents: broadcast fixup possible waiters
  * x86: HPET fix moronic 32/64bit thinko
  * x86: HPET: read back compare register before reading counter
  * Fix CONFIG_AC97_BUS dependency
  * [ARM] 5241/1: provide ioremap_wc()
  * ntp: fix calculation of the next jiffie to trigger RTC sync
  * clocksource, acpi_pm.c: use proper read function also in errata mode
  * clocksource, acpi_pm.c: check for monotonicity
  * x86: delay early cpu initialization until cpuid is done
  * x86: move mtrr cpu cap setting early in early_init_xxxx
  * sched: arch_reinit_sched_domains() must destroy domains to force
    rebuild
  * x86, xen: Use native_pte_flags instead of native_pte_val for .pte_flags
  * x86: pda_init(): fix memory leak when using CPU hotplug
  * x86: cpu_init(): fix memory leak when using CPU hotplug
  * powerpc/spufs: Fix possible scheduling of a context to multiple SPEs
  * netfilter: nf_conntrack_sip: de-static helper pointers
  * netfilter: nf_conntrack_gre: more locking around keymap list
  * netfilter: nf_conntrack_gre: nf_ct_gre_keymap_flush() fixlet
  * netfilter: nf_conntrack_irc: make sure string is terminated before
    calling simple_strtoul
  * pkt_sched: Fix qdisc state in net_tx_action()
  * powerpc: Fix rare boot build breakage
  * ahci, pata_marvell: play nicely together
  * sata_mv: add RocketRaid 1720 PCI ID to driver
  * ahci: disable PMP for marvell ahcis
  * sata_nv: disable hardreset for generic
  * libata-sff: kill spurious WARN_ON() in ata_hsm_move()
  * pata_sil680: remove duplicate pcim_enable_device
  * ahci: RAID mode SATA patch for Intel Ibex Peak DeviceIDs
  * [MIPS] IP22: Fix detection of second HPC3 on Challenge S
  * xen: fix 2.6.27-rc5 xen balloon driver warnings
  * x86: disable static NOPLs on 32 bits
  * netns : fix kernel panic in timewait socket destruction
  * bridge: don't allow setting hello time to zero
  * NFS: Restore missing hunk in NFS mount option parser
  * usb: fix null deferences in low level usb serial
  * Fix format of MAINTAINERS
  * sparc64: Disable timer interrupts in fixup_irqs().
  * [Bluetooth] Fix reference counting during ACL config stage
  * [Bluetooth] Enforce correct authentication requirements
  * [Bluetooth] Reject L2CAP connections on an insecure ACL link
  * [S390] CVE-2008-1514: prevent ptrace padding area read/write in 31-bit
    mode
  * [S390] cio: Correct cleanup on error.
  * [S390] cio: handle ssch() return codes correctly.
  * [S390] cio: allow offline processing for disconnected devices
  * ipsec: Restore larval states and socket policies in dump
  * update Documentation/filesystems/Locking for 2.6.27 changes
  * MAINTAINERS: add Atheros maintainer for atlx
  * lib: Correct printk %pF to work on all architectures
  * x86: fix memmap=exactmap boot argument
  * clockevents: remove WARN_ON which was used to gather information
  * ipv6: Fix OOPS in ip6_dst_lookup_tail().
  * Linux 2.6.27-rc6

wl wireless driver in Intrepid

The wl wireless driver is from Broadcom, it contains a binary blob that makes the driver conform to the FCC regulations in the US. Sometime between 2.6.26 & 2.6.7 they changed a setting inside the blob called set_vlanmode. The end result is that when using the driver in its default state you can't ssh or telnet into another box.

Ben Collins first noticed this behavior and filed LP Bug #259816. Broadcom has since fixed the blob turning the set_vlanmode off. That kernel should hit the repositiories this week. In the mean time if you want to workaround it you can do the following.

# sudo iwpriv eth1 set_vlanmode 0

Make sure you change eth1 in the example above to whatever you wireless interface is on your box.

That will disable the vlan setting and ssh will work again.

~pete