An Introduction to Kernel Live Patching on Linux

Live patching is slowly catching on, but not for the reasons you might think.

In this article, I’ll explain what Linux kernel live patching is, what it isn’t, what services there are, who might use it and why. I’ll show the basic installation steps for Livepatch (on Ubuntu), Kpatch (on Red Hat), and one other, KernelCare, which runs on both.

What is Live Patching?

Live patching is an innocent name for a devilish operation: updating a Linux kernel without stopping it.

Live patching is a way of keeping Linux kernels updated to the latest critical security patches without affecting server downtime. In the past decade, this seductive idea surfed silently in the slipstream of Linux’s increasing popularity. Security, rather than performance or usability, has become the preoccupation of kernel developers as they seek to translate Linux’s ascendance from server to desktop markets. System administrators now see live patching as an essential part of a healthy cybersecurity strategy for their systems, and not just a convenience for themselves.

When researchers discover bugs or other issues in the Linux kernel, the first step after announcing them is to classify them. The most important kinds of kernel issues are vulnerabilities. These are discoveries and descriptions of ways in which someone might corrupt, disrupt or infiltrate a Linux system without the operator’s knowledge or consent. Many of these involve subverting the behavior of a Linux kernel by exploiting faults in its program logic.

The importance of a vulnerability relates to how many systems it affects, how easy it is to reproduce, and what happens if it is. If the answers are ‘a lot’, ‘easy’ and ‘anything’, kernel maintainers are in a frantic race to find a solution. When they do, it means an update of the kernel.

But, because of what the Linux kernel is, an update also means a reboot. For individuals and hobbyists, this is no big deal. But for enterprises, rebooting needs planning and coordination, which costs time and money. Meanwhile, their systems are vulnerable, at least in theory.

Live patching cuts the time between the release of a fix for a vulnerability and its installation on affected systems. Shrinking or eliminating this gap reduces the risk of exploitation of unpatched vulnerabilities.

What are the Pros and Cons of Live Patching?

Live patching is only for critical security problems.

When developers trace vulnerabilities to isolated pieces of kernel code, software patches can fix them without impacting other parts of the kernel. But, for issues that affect many parts, or ones that affect data structures, live patching isn’t possible.

Live patching can’t completely eliminate reboots.

Implementing a live patching policy doesn’t mean you’ll never reboot again. It does mean you can put it off until it’s more convenient, and until then, your system is safe from exploitation of the patched issues.

Not every kernel supports live patching.

All live patching approaches are different. Some use core kernel functions, while others use a mix of open- and closed-source approaches. There is no generic, multi-platform solution (although KernelCare comes close).

Live patching is easy; making patches isn’t.

Anyone with technical knowledge can build and install live patching software for their chosen Linux platform. The source code and information is available, but how easy it is depends on which platform you have.

But, for all live patching approaches, the creation of patches is a huge technical hurdle. It requires a detailed knowledge of the kernel source code, its programming models and conventions. And even if you can create a patch, it is incumbent on you to test it thoroughly before trying to install it.

Live patching isn’t free.

Cost has been the real barrier to adoption by Linux’s non-enterprise core fan base. Oracle Ksplice, Red Hat Kpatch, SUSE Kgraft and KernelCare, all ask for monthly, annual or triennial subscriptions of varying amounts. The reason for this was already mentioned: creating patches isn’t easy. To get a feel for the difficulty involved, take a look at the Kpatch patch writer’s guide?you’ll see why companies have been successful in offering live patching as a paid-for service.

Which Companies Offer Live Patching?

Ksplice was the first to provide live patching for Linux. They introduced a commercial service in 2010, supporting a range of Linux flavors. When Oracle bought them in 2011, support was withdrawn for all but Oracle’s own Linux. This was a red flag to the other big bulls of Red Hat and SUSE. Each rushed to create their own live patching solutions, which they released independently in 2014 as Kpatch and Kgraft respectively. Both became commercial services soon after. Also in that year, KernelCare entered the ring, filling the space left by Ksplice and its withdrawal of support for older kernels and other flavors.

How much is Live Patching?

As of 2019, five companies offer commercial services for live patching. Prices vary according to geographical region-those shown are for the US, and are in descending order.

• Oracle’s Ksplice for Oracle Linux 5, 6, and 7, available with a subscription to Oracle Linux Premier Support, starts at $2299 per year.
• SUSE’s Live Patching is $2198 per year, the combined cost of the live patching service($699) and SUSE Enterprise Linux with a Priority Server Subscription($1499).
• Red Hat’s kpatch for Red Hat Enterprise Linux 7, is available on a Premium support subscription for $1299 per year.
• The Canonical Livepatch Service for Ubuntu 14.04, 16.04, and 18.04 LTS, is free for 1 to 3 machines, then $225 per year for four or more.
• KernelCare is $45 per year. It runs on Ubuntu, Red Hat, Oracle Linux, Debian, CentOS, Amazon Linux, and more.

For the rest of this article, I’ll only consider the three low-to-medium cost options: Canonical, Livepatch, Red Hat Kpatch, and KernelCare.

Who Needs Live Patching?

Based on the cost of subscriptions, it seems that companies with hundreds of servers are the targets for Oracle, Red Hat and SUSE. For enterprises, the benefits of live patching are without question, so bundling the service in a support contract makes perfect sense.

For individuals, live patching remains an inessential curiosity, useful for home media servers, dorm gaming servers, cryptocurrency mining, or any situation where rebooting is verboten. A good way for them to start is with Ubuntu 18.04 LTS. It’s a pain- and cost-free introduction to live patching, for up to three machines.

There is less of a clear-cut choice for entrepreneurs, SMEs, and growing companies. They need flexibility of cost and breadth of coverage, a solution that doesn’t lock them into particular platforms, and one that scales well with growth. For them, KernelCare is a good fit.

How Do I Install Linux Kernel Live Patching?

Live Patching Ubuntu Kernels with the Canonical Livepatch Service

The Canonical Livepatch Service works on server and desktop editions of 14.04, 16.04 and 18.04 LTS. The easiest way to try it out is on the 18.04 LTS desktop edition. When you first reboot after a clean installation, the second screen of the What’s new in Ubuntu dialog window gives you an opportunity to set up Livepatch. If you miss that, open the Software Updater and go to the Updatestab of Settings, from where you can sign in to activate it.

Before installation, you must register for a key. When you do, the page shows these commands for installing via a terminal.

sudo snap install canonical-livepatch
sudo canonical-livepatch enable <your key>

From now on, a daemon process will check for critical kernel security patches and install them automatically, by default, every 60 minutes. If you like to tinker, here’s some examples of using the canonical-livepatch command line tool.

• Check the status: canonical-livepatch status
• Show all patch information: canonical-livepatch status --verbose
• Force an instant refresh: sudo canonical-livepatch refresh
• Change the check interval (for example, to 90 minutes): sudo canonical-livepatch config check-interval=90

Live Patching RHEL 7 Kernels with Kpatch

Red Hat’s Kpatch doesn’t yet have a user-friendly interface for setting it up. With some effort it works on Fedora and CentOS, and for some Debian-based systems such as Ubuntu and Gentoo (but in these cases you must write and test your own patches).

To install kpatch on Red Hat Enterprise Linux 7:

sudo yum install kpatch

There is no automation with Kpatch. Instead, you use the standard command line tool yum to check for and install patches. Critical security patches are available with the package name kpatch-patch. If you have a support subscription, you can ask Red Hat for a custom patch, which you’ll get as an .rpm file. Installing both is shown below.

sudo yum install kpatch-patch # From Red Hat's repositories
sudo yum install kpatch-patch-<version>.el7.<your arch>.rpm # Downloaded custom patch

Live Patching Ubuntu and RHEL7/CentOS 7 Linux Kernels with KernelCare

KernelCare uses the same two-line command for installation on Ubuntu 14.04, 16.04 and 18.04 LTS, and on Red Hat Enterprise Linux 7 or CentOS 7.

Before you start, get a free, 30-day trial key.

The registration page shows your registration key and these two commands needed to install KernelCare.

sudo wget -qq -O - https://repo.cloudlinux.com/kernelcare/kernelcare_install.sh | bash
sudo /usr/bin/kcarectl --register <your key>

If everything is ok, you’ll see the message Server Registered. From now on, the KernelCare daemon checks for critical kernel security patches every four hours. If any are available, it installs them without rebooting and without further intervention.

Like canonical-livepatch, the kcarectl utility lets you do things on the command line.

• Check the status: kcarectl --info
• Show all patch information: kcarectl --patch-info
• Force an instant refresh: sudo kcarectl --auto-update
• Deregister the license: sudo kcarectl --unregister

Conclusion

Live patching for Linux is at a turning point.

Although not the default choice for enterprise customers, Canonical makes great strides by making their Livepatch Service easy to use on Ubuntu 18.04 LTS. I expect its popularity will grow as users move into professional realms, taking their familiarities with them. Those that cast around for a more cost-effective live patching solution, one that runs on a variety of platforms, will find themselves gravitating toward KernelCare.

In any case, what commercial live patching offers is not so much the ability to live patch, but access to the patches themselves, hosted securely, delivered promptly, tested and verified and guaranteed to work without any harmful effects.