Faster, Stronger, and 100% More Custom: A Grand Tour of Linux Kernels

The beating heart of any Linux installation is the kernel. Most distributions that you'll come across add unique features and capabilities to this foundation, but the kernel is the common unifying piece of software across all distributions that calls themselves Linux. Unlike a commercial operating system, the kernel is both very configurable and can be completely customized by any user. Why would you want to tinker with your kernel configuration? While the stock kernel provided by your distribution is a reasonable option, there are unique sets of patches that can fundamentally change how the kernel behaves and, in some cases, may tune your system to be even more effective for your particular use case. Let's take a look at the landscape of specialized Linux kernels! Kernel Basics: The Upstream The web site is the ultimate source of truth for the mainline Linux kernel that the kernel team releases on a regular basis. If you take a look at the landing page, you may see a variety of different kernel versions listed: kernel.org If you're a Linux user, you may have never actually seen this page for the entirety of your time actually Linux. What gives? using A distribution like usually selects a kernel version that's known to be reliable and stable, applies any necessary in order to customize it as they see fit, and then provides the compiled kernel to all their users so that that actual compilation process (which can take a while!) occurs on their end instead of your desktop's. This means that two up-to-date users of Ubuntu 22.04, for example, are likely using the same kernel build, which simplifies supporting a wide variety of users across a range of different installations. It also permits the engineers developing Ubuntu to focus on providing a single kernel build that is well-tested and meets most of their user's needs. Ubuntu patch sets If you were to compare the running kernel on two up-to-date Linux distributions, though, their versions would almost certainly be different. While the kernel has an excellent reputation for stability, some distributions choose to be cautious about when to make the jump to a newer kernel. This is typically a good idea - the numbers between two kernel versions like 5.18.16 and 5.10.135 seem large, but kernel developers are conservative about adding and removing features, so there typically aren't massive practical differences between them. If one Linux distribution sticks a few versions behind the latest kernel, doing so in order to achieve more predictable releases is often worth the tradeoff. different But while the kernel doesn't make massive leaps between releases, different developers maintain collections of changes that can fundamentally alter how the kernel behaves. Some of these include: default downstream CPU scheduler changes. This is one of the core functions of the kernel and dictates how different processes on your machine are chosen to get executed on your processor. I/O scheduler changes. The kernel supports different scheduler methods like or which can make a significant impact upon the performance of disk operations, particularly for spinning disks where the physical realities of spindles and read heads impact how quickly data can be accessed. deadline cfq Security. Some patch sets make significant changes to attempt to the Linux kernel against potential exploit vectors. harden The beauty of running Linux is that all of these different options are completely open to you! Let's take a look at a few of them. Delicious, Default Vanilla The default Linux kernel doesn't have any flashy customizations, but there are still quite a few settings that are still under a user's control without any need to recompile their kernel or make changes to the source code. For example, the aforementioned I/O scheduler dictates how the kernel prioritizes pending operations to either read from or write to persistent storage. This is an important setting for a variety of reasons. Does the kernel push through a document that you're trying to save to the disk first to improve desktop latency, or should it flush the system journal to disk to ensure that its available in the event of a crash? These sort of I/O scheduler decisions factor in a large number of variables like time spent waiting on the kernel, the urgency of the request, and more. Like nearly all low-level Linux settings, you can directly inspect which I/O scheduler your system is using by using a filesystem path. This information is controlled on a per-device basis using the virtual filesystem. On my system, I can find this by inserting the device that I'm interested in, , to a path in : sys sda /sys $ cat /sys/block/sda/queue/scheduler [mq-deadline] kyber none This indicates that my drive is using the scheduler with the other methods, and , available for use but not actively chosen at this time. The scheduler is described as " ", which sounds good to me! sda mq-deadline kyber none mq-deadline [a] good all-rounder with fairly low CPU overhead You can also many of these settings using a similarly simple interface. The explains this well. In short, simply sending the desired scheduler name to the same path as before is sufficient. The following command would change the scheduler for this particular device to use the scheduler. Note that access is necessary to change these types of settings. change Linux Kernel documentation kyber root $ echo kyber > /sys/block/sda/queue/scheduler Per-device schedulers are just one of the runtime-tunable settings available on a running kernel. Check out these other settings for additional options: swappiness net.core Long Term Support Kernels: Oldies, but Goodies What if new, fancy features seem great, but you're primarily concerned about support and stability? The long-term support kernel, or , is a version of the Linux kernel specially optimized for support over a much longer period of time than a typical kernel. For systems that favor predictable, stable upgrades, using a kernel package suffixed with is often a very good choice. While no special patches have been applied, using a kernel like 5.15 means that support is guaranteed until October 2023 instead of any earlier. If you were to have adopted kernel 4.9 when it was released in December 2016, it will have been supported until January 2023 – that's a long time to not have to worry about breaking changes! LTS -lts The long-term support kernel is probably the most widely available kernel variant outside the mainline version. Some distributions like Ubuntu LTS ship with a long-term support kernel by default, but most other distributions provide the option to use an LTS kernel by installing and enabling a package with names similar to . Your chosen distribution likely has guides explaining how to install and use different kernels, so to try out the rock-solid LTS kernel, look for kernel packages ending with and follow your chosen distribution's instructions for using an alternate kernel package. linux-lts -lts Hardened Kernels: For the Security-minded Within information security, refers to the process of shoring up specific areas to mitigate potential exploits and vulnerabilities. The mainstream Linux kernel obviously tries to achieve this degree of security as much as reasonably possible, but some users may choose to constrain their kernel and its feature in order to trade them for increased security. The provides this. hardening hardened Linux kernel Typically, the hardened Linux kernel behaves very similarly to a typical kernel, though there are tradeoffs. For example, some applications may simply not work on a hardened kernel, and many pre-compiled kernel modules may need to use their equivalent in order to properly integrate with a hardened kernel. Nevertheless, for mission-critical applications where security is a priority, a hardened kernel is often the right choice. DKMS stands for "dynamic kernel module support" and is a feature that permits users to use the source code for kernel modules that do not have pre-compiled support for a desired kernel. For example, the project must be installed as a kernel module in order to use the ZFS filesystem on Linux. Many distributions provide a package that will install the pre-compiled module ready for use, which avoids lengthy compilation times and potential incompatibilities, but may not function correctly when using a custom kernel. An OpenZFS DKMS module, on the other hand, can compile itself on-the-fly against your chosen kernel in order to ensure that it functions as expected. DKMS OpenZFS on Linux Achieving Enlightenment with Zen The is one of the more widely supported community-based patch sets. While not a massive departure from the upstream Linux kernel, the zen (or Liquorix) kernel makes changes in order to supercharge performance. From the Liquorix web site: zen kernel Liquorix is a distro kernel replacement built using the best configuration and kernel sources for desktop, multimedia, and gaming workloads. One change, for example, is to alter the aforementioned I/O scheduler to select the algorithm by default, which optimizes for system responsiveness. Another set of changes compresses swap storage and uses an alternative caching algorithm to perform well under high memory load. bfq As one of the more well-established patch sets, there may already be a package in your distribution of choice ready to use – however, the also provides a tutorial explaining how to use this kernel if you need additional options. linux-zen project documentation Pushing FPS with Gaming Kernels The kernel is a specialized set of patches specifically designed to optimize gaming performance. Squeezing as many FPS as possible out of a graphics card can be a strenuous exercise, but changes in the tkg patch set can offer additional gains with focused changes to critical pieces of the Linux kernel. linux-tkg For example, additional CPU schedulers have been added, like the scheduler or the patch set. Network algorithms like are enabled by default, as are optimized compilation flags like to squeeze as much performance out of compiled code as possible. Project C -ck Cake -O2 The patch set is less ubiquitous than the LTS or Zen kernels, but the provide a comprehensive guide detailing how to compile and install the custom kernel. -tkg installation steps on GitHub The patch set provides a similar set of features targeting users that require a low-latency, high-performance kernel suitable for use cases like gaming. In either case, the tkg and XanMod kernels offer a very specialized form of Linux kernel that can still function as a day-to-day operating system with additional benefits on gaming machines. XanMod Real-time Kernels The term "real-time computing" refers to the concept of a system that aims to guarantee that certain operations run an a given window of time. For example, a system that controls the brakes of a car or the wings of an aircraft cannot afford to be interrupted by a flush of pending writes to a disk or drawing images on a screen. The project provides a set of patches that offer this functionality. While these features are often very specialized and not typically suited for desktop or server applications, they can nevertheless be critically important in the right situation. For projects that have applications in particular industries, these real-time patches are a necessity. Real Time Linux The suite of kernels have a much more targeted audience but are often more widespread than kernels optimized for gaming or low-latency due to their importance where they remain in-use. Your distribution's package manager may provide these types of kernels under names like or . -rt linux-rt linux-rt-lts Hybrid Kernels There are many more specialized kernels out there that we explore, but we'd be here all day! With that said, there's even more ways to customize a kernel using the different types we've talked about already. An interesting trait of many of these patches is that they're often into kernels that blend their features into one kernel source tree. For example, a critical piece of operational machining infrastructure may require the real-time capabilities of an kernel but also benefit from the additional security of a hardened kernel. As long as different patches can be applied cleanly together, nothing stops you from mixing and matching kernels tailored to your exact needs. combined -rt Some distributions provide this simply by combining patch sets into the kernel's package name. For example, the Arch Linux User repository provides the package, which combines the real-time and long-term support kernel features. Ubuntu provides packages like specialized with various changes to optimize the kernel for operation in AWS EC2. linux-rt-lts linux-image-*-aws Regardless of your chosen distribution, if you're interested in running a custom kernel, be sure to explore your distribution's packages for available kernels. They provide the simplest and quickest way to get started with a specialized version of the Linux kernel. Go forth and Tinker! This article has only scratched the surface of the Linux kernel landscape. While many of these kernels are widespread and commonly found in the majority of distribution repositories, some more esoteric patch sets exist that push the kernel even further. With an open-source kernel, you could even modify the kernel yourself to compile one custom to your own needs, but that's a guide for another time. Until then, check out some of these resources for more information and find a kernel that you'd like to try for yourself! ArchWiki - Kernel Ubuntu kernels from Canonical The Linux Kernel Archives