Ubuntu Natty: A custom kernel is what you want

On April 28, 2011, in HowTOz, by gtklocker

It’s really common for people to make mistakes. Some of us make just a lot of them. And when we do, the question of course is whether or not we can do something about it. Now, I don’t know about you, but I always prefer solving my problems after understanding what went wrong.

In a recent case, what happened had terrific consequences. Without knowing much about the support of Reiser4 filesystem in the kernel of Ubuntu 11.04, I stupidly went ahead and formatted a 2TB external hard drive with said filesystem from my Arch-based laptop. Then I happily put tons of data on it, since my laptop’s hard drive had already failed several times. Imagine what: Very soon after, the same hard drive failed again.

I was really excited I had a backup of my data on an external hard drive and that time around I didn’t lose a bit (yes, this has happened to me before). So the day came and I wanted to watch the fourth episode of Rozen Maiden (an epic Anime). But after plugging the drive into my Ubuntu box I discovered that it just wouldn’t mount. I tried hard with Nautilus, pmount etc. but to no avail. At some point I looked at /proc/filesystems. What I saw in there didn’t make me happy at all: Reiser4 was absent! There was also no such module in /lib/modules/`uname -r`/kernel/fs, so by that time I thought I was dead in the water. A moment later -it really seemed like an eternity- I managed to pull myself together and immediately looked up for Reiser4 support in the Linux kernel. Soon I discovered that the mainline kernel needs a patch in order to support Reiser4, so do the majority of the Linux distributions out there.

Since I was left without my Arch laptop I had to improvise in the -relatively unknown for me- land of Ubuntu. I was pretty sure I could make it compile a kernel. I mean Ubuntu is no Arch but it’s still a Linux-based OS, darn it, so in reality there was no excuse for failure. And as it turned out, failure and yours truly didn’t meet that day.

Preparing for the operation
This is the not-so-painful part of the process. You only have to install some packages that are needed in order to compile a kernel. Now, I’m not a newbie, I know my way around a Linux system. In spite of that I admit I had at first some problems with the package names – but Google really helped. So there you go:

$ sudo apt-get -y install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge git-core libncurses5 libncurses5-dev libelf-dev libdw-dev binutils-dev
...
$

If you are a developer or have messed with similar stuff, then you most probably have some or even all of those packages already installed. But it doesn’t hurt to execute this command anyways. You also have to make sure that all the dependencies for the Linux build are satisfied. To that end, just type the following:

$ sudo apt-get build-dep kernel-package linux-meta
...
$

Bringing on the patient
By now you should have everything you need to recompile your kernel. But wait. Where are you going to operate on? Why, on the Linux kernel’s source code, of course! There are many ways to get the patient on the surgery table. Personally, I prefer having a clean git clone of the latest kernel for the version of Ubuntu I’m using and keep a backup of it in case I destroy something, so I won’t have to re-download the source (believe me, it’s *huge*).

Since you installed git a bit earlier, now create a directory to have as a workplace:

$ mkdir -p ~/dev/kernel
$ cd ~/dev/kernel/
$

Then clone the repository in it:

$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-natty.git source
Cloning into source...
remote: Counting objects: 1980747, done.
remote: Compressing objects: 100% (329901/329901), done.
Receiving objects: 100% (1980747/1980747), 496.08 MiB | 1.32 MiB/s, done.
Resolving deltas: 100% (1655091/1655091), done.
$

If you’re catchy you’ll notice we just downloaded the Natty (11.04) kernel source. Needless to say that in case you’re using a different version of Ubuntu you’ll have to get the corresponding source tree.

The cloning process should take some time. When it’s done you’ll see that a new folder named source has been created. It’s contents are obvious, I believe.

The time for patching has come
First thing you should do is take a note of the exact kernel version your Ubuntu installation is currently using:

$ uname -r
2.6.38-8-generic
$

After knowing the exact version string it was much easier for me to get the right patch for proper Reiser4 support. Make sure you also select the right patch for your own kernel version. Please note that Linux stable versions appear like flies and if you’re reading this article long after it’s written you will almost surely have to make some adjustments here and there.

$ mkdir patches
$ cd patches/
$ wget http://www.kernel.org/pub/linux/kernel/people/edward/reiser4/reiser4-for-2.6/reiser4-for-2.6.38.patch.gz
--2011-04-28 12:32:23--  http://www.kernel.org/pub/linux/kernel/people/edward/reiser4/reiser4-for-2.6/reiser4-for-2.6.38.patch.gz
Resolving www.kernel.org... 149.20.4.69, 149.20.20.133
Connecting to www.kernel.org|149.20.4.69|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 609740 (595K) [application/x-gzip]
Saving to: `reiser4-for-2.6.38.patch.gz'

100%[=========================================>] 609,740      271K/s   in 2.2s    

2011-04-28 12:32:26 (271 KB/s) - `reiser4-for-2.6.38.patch.gz' saved [609740/609740]

$ gunzip reiser4-for-2.6.38.patch.gz
$

Great! The patch is where it’s supposed to be, unpacked and ready to be incorporated into the Ubuntu kernel. Now, say we want to use another patch. This time it’s gonna be the CK patch by Con Kolivas, which, if used wisely, provides a great speed boost in multitasking scenarios. Again, be sure you download the right one for your particular kernel version.

$ wget http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/2.6.38/2.6.38-ck3/patch-2.6.38-ck3.gz
--2011-04-28 12:36:33--  http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/2.6.38/2.6.38-ck3/patch-2.6.38-ck3.gz
Resolving www.kernel.org... 149.20.20.133, 149.20.4.69
Connecting to www.kernel.org|149.20.20.133|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 73398 (72K) [application/x-gzip]
Saving to: `patch-2.6.38-ck3.gz'

100%[=========================================>] 73,398      59.6K/s   in 1.2s    

2011-04-28 12:36:35 (59.6 KB/s) - `patch-2.6.38-ck3.gz' saved [73398/73398]

$ gunzip patch-2.6.38-ck3.gz
$

We’re now ready to apply the patches. Since we put them all in a single directory, it’s dead easy to do it.

$ cd ../source/
$ patch -p1 < ../patches/reiser4-for-2.6.38.patch
$ patch -p1 < ../patches/patch-2.6.38-ck3
$

Get psyched for the configuration
Terrified? You shouldn’t be, as the configuration process is really easy. Supposing you’re not a customization maniac, the default config file of the standard Ubuntu kernel, i.e. the one describing the features of the currently running kernel, will suit you well. You’re going to make a few changes here and there, but when you’re using the default config as a base the whole process is pretty easy and totally safe.

$ cp -vi /boot/config-$(uname -r) .config
$ make oldconfig
$ make oldconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/basic/docproc
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/kxgettext.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/lex.zconf.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf --oldconfig Kconfig
*
* Restart config...
*
*
* General setup
*

If everything went well you will presented with the following question. Agree by pressing the [Y] key.

BFS cpu scheduler (SCHED_BFS) [Y/n/?] (NEW)

Immediately after that you’re going to be asked about the timer frequency. Go for a low one if you have a laptop or a not-so-powerful computer. I think my dual core desktop is powerful enough so I pressed [4], which is 1000MHz.

Timer frequency
  1. 100 HZ (HZ_100)
  2. 250 HZ (HZ_250_NODEFAULT) (NEW)
  3. 300 HZ (HZ_300)
> 4. 1000 HZ (HZ_1000)
  5. 1500 HZ (HZ_1500) (NEW)
  6. 2000 HZ (HZ_2000) (NEW)
  7. 3000 HZ (HZ_3000) (NEW)
  8. 4000 HZ (HZ_4000) (NEW)
  9. 5000 HZ (HZ_5000) (NEW)
  10. 7500 HZ (HZ_7500) (NEW)
  11. 10000 HZ (HZ_10000) (NEW)
choice[1-11?]:

Next, just press [Enter] to all the other questions until you see this one:

Reiser4 (EXPERIMENTAL) (REISER4_FS) [N/m/y/?] (NEW)

Instead of incorporating Reiser4 support into our kernel we choose to build it as a dynamically loadable module instead, so we press the [M] key. We don’t want debugging support for Reiser4, so in the following question we press [N].

  Enable reiser4 debug mode (REISER4_DEBUG) [N/y/?] (NEW)

For any other question just select the default answer by pressing [Enter]. In case you’d like to configure any special options, you can also type

make menuconfig

Build it like a real builder
We’re almost done. We just have to build our new kernel, after all this hard work.

$ make-kpkg clean
exec make kpkg_version=12.036+nmu1 -f /usr/share/kernel-package/ruleset/minimal.mk clean
====== making target minimal_clean [new prereqs: ]======
This is kernel package version 12.036+nmu1.
test ! -f .config || cp -pf .config config.precious
test ! -e stamp-building || rm -f stamp-building
test ! -f Makefile || \
            make    ARCH=i386 distclean
make[1]: Entering directory `/home/gtklocker/dev/kernel/source'
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   include/config include/generated
  CLEAN   .config .config.old
make[1]: Leaving directory `/home/gtklocker/dev/kernel/source'
test ! -f config.precious || mv -f config.precious .config
rm -f modules/modversions.h modules/ksyms.ver scripts/cramfs/cramfsck scripts/cramfs/mkcramfs
$

For the next command you’ll most definitely have to make two adjustments. First, give to the –append-to-version parameter a name you’d like your new kernel to have. Second, change the value of –overlay-dir to the full path of the directory the kernel source resides.

$ nice fakeroot make-kpkg --initrd --append-to-version=-lockard --overlay-dir=/home/gtklocker/dev/kernel/source kernel-image kernel-headers
...
$

Be patient and wait for the compilation to finish. Depending on the raw processing power -or lack thereof- of your computer’s CPU, the whole process may take up to an hour or even more. In any case, a gazillion of messages will fly by your terminal window. Sit back, grab a slice of pizza or two and enjoy.

Installation
If everything went well -and I really can’t imagine a reason they don’t-, after the compilation finishes you’ll find a couple of DEB files in ~/dev/kernel.

$ cd ..
$ ls -lh
total 46M
-rw-r--r-- 1 gtklocker gtklocker 7.4M 2011-04-28 17:17 linux-headers-2.6.38.4-ck3-lockard+_2.6.38.4-ck3-lockard+-10.00.Custom_i386.deb
-rw-r--r-- 1 gtklocker gtklocker  39M 2011-04-28 17:12 linux-image-2.6.38.4-ck3-lockard+_2.6.38.4-ck3-lockard+-10.00.Custom_i386.deb
drwxr-xr-x 1 gtklocker gtklocker   80 2011-04-28 12:39 patches
drwxr-xr-x 1 gtklocker gtklocker 1.2K 2011-04-28 17:13 source
$

Go ahead and install them:

$ sudo dpkg -i linux-*.deb
[sudo] password for gtklocker:
Selecting previously deselected package linux-headers-2.6.38.4-ck3-lockard+.
(Reading database ... 133602 files and directories currently installed.)
Unpacking linux-headers-2.6.38.4-ck3-lockard+ (from linux-headers-2.6.38.4-ck3-lockard+_2.6.38.4-ck3-lockard+-10.00.Custom_i386.deb) ...
Selecting previously deselected package linux-image-2.6.38.4-ck3-lockard+.
Unpacking linux-image-2.6.38.4-ck3-lockard+ (from linux-image-2.6.38.4-ck3-lockard+_2.6.38.4-ck3-lockard+-10.00.Custom_i386.deb) ...
Done.
Setting up linux-headers-2.6.38.4-ck3-lockard+ (2.6.38.4-ck3-lockard+-10.00.Custom) ...
Examining /etc/kernel/header_postinst.d.
run-parts: executing /etc/kernel/header_postinst.d/nvidia-common 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
Setting up linux-image-2.6.38.4-ck3-lockard+ (2.6.38.4-ck3-lockard+-10.00.Custom) ...
Running depmod.
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
update-initramfs: Generating /boot/initrd.img-2.6.38.4-ck3-lockard+
run-parts: executing /etc/kernel/postinst.d/nvidia-common 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
run-parts: executing /etc/kernel/postinst.d/pm-utils 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
run-parts: executing /etc/kernel/postinst.d/update-notifier 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.38.4-ck3-lockard+ /boot/vmlinuz-2.6.38.4-ck3-lockard+
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.38.4-ck3-lockard+
Found initrd image: /boot/initrd.img-2.6.38.4-ck3-lockard+
Found linux image: /boot/vmlinuz-2.6.38-8-generic
Found initrd image: /boot/initrd.img-2.6.38-8-generic
Found memtest86+ image: /memtest86+.bin
done
$

And now it’s one of those rare times where in order for the changes to take effect you need to reboot your Linux box. Do that and after the system comes up check the version of the running kernel:

$ uname -r
2.6.38.4-ck3-lockard+
$

Great! Also take a look into the directory where the active kernel’s filesystem modules reside:

$ ls -dlh /lib/modules/2.6.38.4-ck3-lockard+/kernel/fs/reiser*
drwxr-xr-x 1 root root 20 2011-04-28 20:58 /lib/modules/2.6.38.4-ck3-lockard+/kernel/fs/reiser4
drwxr-xr-x 1 root root 22 2011-04-28 20:58 /lib/modules/2.6.38.4-ck3-lockard+/kernel/fs/reiserfs
$

Congratulations! You’re now running your own, custom-made kernel.

Conclusion
All in all, the kernel building process is easy for everyone and does not require any sophisticated or even advanced skills. So, if you want a feature and there’s a patch for it, go ahead and recompile! In the worlds of Linux in particular and Open Source / Free Software in general, nothing is impossible ;)

Tagged with:
 

25 Responses to Ubuntu Natty: A custom kernel is what you want

  1. ZomZilla says:

    this guide is fantastic! while not a total linux noob i didnt have a clue where to start and having natty and trying to follow a hardy guide wasnt going to well for me

    OVERALL a great, up-to-date, detailed guide.

    *bookmark* ;)

  2. Ant says:

    Great guide :) I’d like to ask a few things though:

    1) Where’s a good place to get a list of patches like those you mentioned and why are they in the default kernel already – especially if they improve multitasking performance?

    2) Why did you choose to build Reiser4 support into the kernel as a dynamically loadable module? That is to say, why M and not Y?

    3) 1000Hz seems to be the value most users have chosen – but what happens if you opt for 100 or even 10000? Is 1000 the ‘safe bet’ on not breaking something? What could go wrong here? lol

    Thanks again!

    • gtklocker says:

      Thanks for your comment :)

      1) I don’t think there really is a list of kernel patches. Individuals make them and reputation gets people to know them, but I’d bet there are some really good ones that even Linus Torvalds himself doesn’t know about. As for the inclusion or not of certain patches in the default kernel, there are many reasons. Reiser4, for example, is kinda unstable at this point (and I guess it will remain unmaintained for long because of its lead developer being imprisoned). As for bfs/ck, it’s highly possible to fail on older machines. Also, the distribution’s kernel maintainers can’t really know what CONFIG_HZ you want the kernel to be in, so it will either be high or low.

      2) Well, if you look at the default configuration file you’ll see that all the filesystems are built as *modules*. This happens mostly because the filesystems are loaded exactly when there’s need to mount a partition. That being said, when you try to mount your /home partition in xfs, the xfs module is loaded and *then* the partition gets mounted. This is the way things work and it would be kind of stupid to not do it the standard way.

      3) As I said, low frequencies lead to lower performance. Also 1000Hz is good enough for the most modern systems; this is why I like it. I wouldn’t advocate the use of bfs/ck on a laptop, though, as it’s not kind at all to battery. For more info read the BFS FAQ.

      Happy hacking ;)

      • Spudd86 says:

        Well lower frequencies lead to higher latency, but better throughput. So if you don’t need the low latency it’s worth it to go with lower values for HZ.

        Basically you don’t do context switches as often with lower HZ, if you’re running a webserver or something you probably don’t need low latency, but the boost in throughput might be helpful, depending on your workload.

    • Zomzilla says:

      the way i’ve been working it out (that has given me amazing performance boost already) is to do roughly 1000/no. of cores

      for example my dual core is set at 300 (thats the closest)

      actually the reason i was on this page again is for reference as i fiddle with the timings and such

  3. Ant says:

    Sorry, on 1), i meant aren’t*

  4. Spudd86 says:

    Just a question, why would you use a filesystem that is not in the kernel mainline, that has nobody working on it anymore (since Hans Reiser was arrested for murder and his company sort of went away), for your backups?

    Especially since you seem to be aware of some of the issues with reiser4…

    As you said you made a mistake, I’m just curios what lead to it…

    • subZraw says:

      Concerning your first question, my first guess is something along the lines of “because you can” :) But besides that, the Reiser4 case was just an example. The purpose of this post was to demonstrate how one can re-compile his or her Linux kernel. The discussion on the kinds of patches a user may actually incorporate to the kernel is a totally different one! (And oh BTW, thanks for the idea you just gave me :D)

  5. FireWave says:

    Il manque le paquet kernel-package, pour avoir make-kpkg.

    • subZraw says:

      Sorry but this is an English-speaking site, so you’ll have to translate your comment…

      • FireWave says:

        Sorry, I read english everyday and sometime I’m comfuse about what language I just read !

        Just told you that it looks like the package kernel-package was missing on your apt command, in order to have the make-kpkg command.

  6. FireWave says:

    Another add:
    Since I use my new custom kernel, eclipse (and some RCP tool like rssowl) didn’t work anymore.
    java.lang.IllegalArgumentException: invalid qualifier: 4-ck3-core2+
    Very nasty thing. I’m investigating…

    • FireWave says:

      Found that “Kernel release name with a ‘+’ causes Eclipse to fail to start with an “invalid qualifier” error”
      Do you know how to suppress this +?

  7. Rudiger Wolf says:

    I had a problem with this line:
    sudo apt-get -y install build-dep kernel-package linux-meta

    The following worked for me:
    sudo apt-get build-dep kernel-package linux-meta

    • subZraw says:

      You’re absolutely right, the “-y install” part has been put there by mistake. In reality, the build-dep command is all that’s needed to cause apt-get to install or remove all the packages necessary in an attempt to satisfy the build dependencies for a source package. I’m gonna correct the error right now – thanks again for pointing it out!

  8. solomy says:

    some one help me to install prism54 on natty.
    i’ve found this prism54-cvs-latest.tar.bz2 .please help me on how to install.

  9. Traveller says:

    make-kpkg has useful option ‘-j’ for compilation parallelization (for example, I used -j4 for 4-core CPU)

  10. BernardS says:

    nice guide! I was able to complete the whole building and installation of the custom kernel after two days. ;)
    my setup was ubuntu 10.04(lts) upgraded the kernel to 2.6.38-13, then patch it with ck and reiser4. the only blocker i got was the initrd.img. i wasn’t able to find it after the compilation but executing update-initramfs -c -k made it. maybe we can add it in the steps >:)
    Btw, thanks a lot!

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Subscribe to followup comments via e-mail. You can also subscribe without commenting.



Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can
take care of it!

Archives

All entries, chronologically...

Visit our friends!

A few highly recommended friends...

More in HowTOz (2 of 4 articles)