Ksplice in action

On July 21, 2011 Oracle announced that it has aquired Ksplice. With Ksplice users can update the Linux kernel while it is running, so without a reboot or any other disruption. As of September 15, 2011 Ksplice is available, at no additional charge, to new and existing Oracle PremierSupport customers on the Unbreakable Linux Network (ULN).

Updating the Linux kernel while it is running sounded like an impossible mission to me, and I was really keen to see this in action with my own “eyes” ;-) Yesterday I gave it a try and in this posting I will share my first exprerience with you.

The installation of Ksplice is a very easy process which took only a few minutes and can be performed while the system is up and running. It does however require an ULN account for obvious reasons ;-)

Before updating my system lets have a look when the system was booted, which kernel it is running and show you that I have an Oracle database running while the kernel is being updated to a new version:

$ who -b
         system boot  2011-09-23 18:52
$ uname -r
2.6.32-200.16.1.el5uek
$ pgrep -lf smon
6037 ora_smon_v1120
 

The above output shows that my system is running a 2.6.32-200.16.1.el5uek kernel. The “-uek” indicates an Oracle Unbreakable Enterprise Kernel which is a pre-requisite for using Ksplice on Oracle Linux.

And now, lets update the currently running Linux kernel to the latest version using Ksplice:

$ sudo uptrack-upgrade -y
The following steps will be taken:
Install [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Install [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Install [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Install [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Install [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Install [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Install [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Install [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Installing [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Installing [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Installing [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Installing [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Installing [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Installing [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Installing [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Installing [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Your kernel is fully up to date.
Effective kernel version is 2.6.32-200.19.1.el5uek

Note: Although the product is called Ksplice, the service it provides is known as uptrack.

The result of running the uptrack-upgrade command is that my system is now running kernel version 2.6.32-200.19.1.el5uek and it happened without a reboot or even stopping the running Oracle database! The output also shows that updating the running kernel occurred by installing small chunks of code corresponding to each patch that was applied to the kernel source code when the new kernel version was put together.
The output below shows that the system was not rebooted nor that the running database was restarted.

$ who -b
         system boot  2011-09-23 18:52
$ pgrep -lf smon
6037 ora_smon_v1120
$ uname -r
2.6.32-200.16.1.el5uek

It may be a bit confusing that uname –r still reports kernel version 2.6.32-200.16.1.el5uek while in reality the kernel version is 2.6.32-200.19.1.el5uek. According to the documentation this is expected behaviour and there is an uptrack-uname command available to report the kernel version that is actually running as shown below:

$ uptrack-uname -r
2.6.32-200.19.1.el5uek

In case you want to know which updates were applied to the running kernel the uptrack-show command is your friend:

$ sudo uptrack-show
Installed updates:
[694jrs5f] Clear garbage data on the kernel stack when handling signals.
[zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
[gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
[hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
[fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
[04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
[xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
[oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.

Effective kernel version is 2.6.32-200.19.1.el5uek

If, for whatever reason, you want to remove the updates that were applied to the running kernel, it is good to know that this can also be performed without a reboot or any other service disruption by running the uptrack-remove command.

$ sudo uptrack-remove -y --all
The following steps will be taken:
Remove [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Remove [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Remove [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Remove [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Remove [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Remove [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Remove [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Remove [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Removing [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Removing [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Removing [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Removing [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Removing [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Removing [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Removing [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Removing [694jrs5f] Clear garbage data on the kernel stack when handling signals.

All the previously applied updates are taken out, in reverse order, which basically reverts the system back to its original state. The output below shows that this indeed happened without a reboot or stopping the running Oracle database:

$ who -b
         system boot  2011-09-23 18:52
$ pgrep -lf smon
6037 ora_smon_v1120
$ uname -r
2.6.32-200.16.1.el5uek
$ uptrack-uname -r
2.6.32-200.16.1.el5uek
$ sudo uptrack-show
Installed updates:
None

Effective kernel version is 2.6.32-200.16.1.el5uek

Cool, isn’t it? I am impressed!

Please read this Ksplice technical paper for some background information on the Ksplice technology.

Please keep in mind that Ksplice will only update the running kernel in memory and does not install a new kernel RPM. It does re-apply the updates automatically after a system reboot and will also check for new updates on a regular basis. Ksplice can download and install new updates automatically whenever they become available ensuring your kernel is always up-to-date!
-Harald