Metztli I. T.  Blog

I have been driven to experiment with Docker to create lightweight, portable, self-sufficient containers from any app. Docker relies on LXC (Linux Containers), a virtualization system, making use of the 'cgroups' feature. Although LXC and Docker are simply installed into GNU/Linux Debian from its repositories, not everything is 'peachy' when one is running a custom kernel which is built from latest sources found at the Linux Kernel Archives. This is true especially since I have been building my kernels with aufs3 (advanced multi layered unification filesystem version 3.x) support. Accordingly, below I will elaborate some necessary kernel configuration modifications to support LXC -- and thus Docker -- on a Debian 64-bit distribution.

There is much enthusiasm by publishers of cloud and virtualization articles¹ about docker. Red Hat's OpenShift platform-as-a-service (PaaS) is integrating docker into its online/hybrid cloud offering. It is natural therefore for those of us who dabble in those disciplines to explore the subject deeper to understand the technology. Docker runs on GNU/Linux but if you have a proprietary operating system, you may run docker in a virtual machine like Oracle's open source VirtualBox.

Installing Docker From the Debian APT Repositories.

At a Debian shell we install these packages²:

$ apt-get install lxc docker.io cgroupfs-mount

Indeed, Debian renames docker to docker.io to avoid conflicting with an already existing package name. After those tasks were completed, if we were to follow Rob Knight's Blog³ tutorial, we would download (pull) a debian base image which would be placed under /var/lib/docker directory. Thus (with root privilege) we type command:

# docker.io pull debian:wheezy
(sample output)

Pulling repository debian
a60f67605f28: Download complete
511136ea3c5a: Download complete
e84098d6c70f: Download complete
712526d0712e: Download complete

However, we would get as far as the command to run the already pulled debian base image into our environment, because trying to run bash inside our base image:

# docker.io run -i -t debian:wheezy /bin/bash
(showstopper output would be similar to):

[error] client.go:2298 Error getting size: bad file descriptor

Indeed, it is a maddening experience because, even after searching online, one is left at a loss to understand the issue.

Modify Configuration and Rebuild/Replace Already Nicely Running Kernel

Having tried several online suggestions, including downloading docker directly from its parent dotCloud PaaS repository ², and modifying/renaming Debian's default at /etc/init.d/docker.io -- as well as /usr/bin/docker.io -- to no avail, it gradually emerges that it must be a kernel configuration issue⁴.

Although I was currently running Linux kernel 3.13.7, its .config was based on the earlier Linux Xonecuiltzin kernel 3.12.2 with aufs3 support expounded on Debian: Patching Linux Kernel To Enable Aufs3 Module prior post. Since The Linux Kernel Archives had just released the latest stable kernel 3.14 that is what I used for the procedures that follow.

If the owner of the build directory is the system super user (or root), then the regular user who will be (re)building the kernel must be a member of group src, which in turn must be granted writing privilege on the directory where the operations on kernel 3.14 and aufs3.x source will be executed. Again, please read Debian: Patching Linux Kernel To Enable Aufs3 Module for the initial preparations -- if you are not familiar.

In this particular case I will be downloading the kernel 3.14 source at /usr/src/build/tlacauhtli-3.14.0/ and the aufs3 source at /usr/src/build/tlacauhtli-3.14.0/build/ where the kernel source will be extracted as well. Thus, in my shell I change directory:

$ cd /usr/src/build/tlacauhtli-3.14.0/

and use wget utility to download Linux kernel 3.14 source and its signature:

Shell

Indeed, you may use clamscan to verify your download is free of known malware and then we may proceed to verify the kernel 3.14 signature:

$ xz -dc linux-3.14.tar.xz | gpg --verify linux-3.14.tar.sign -
(sample output)

gpg: Signature made Sun 30 Mar 2014 08:46:44 PM PDT using RSA key ID 00411886
gpg: Can't check signature: public key not found

again, please note the RSA key ID number and use it as --recv-keys argument:

$ gpg --keyserver pgp.mit.edu --recv-keys 00411886
(sample output)

gpg: requesting key 00411886 from hkp server pgp.mit.edu
gpg: key 00411886: public key "Linus Torvalds <torvalds@linux-foundation.org>" imported

gpg: no ultimately trusted keys found
gpg: Total number processed: 1

gpg:               imported: 1  (RSA: 1)

then we run verification process again:

$ xz -dc linux-3.14.tar.xz | gpg --verify linux-3.14.tar.sign -
(sample output)

gpg: Signature made Sun 30 Mar 2014 08:46:44 PM PDT using RSA key ID 00411886
gpg: Good signature from "Linus Torvalds <torvalds@linux-foundation.org>"

gpg: WARNING: This key is not certified with a trusted signature!

gpg:    There is no indication that the signature belongs to the owner.
Primary key fingerprint: ABAF 11C6 5A29 70B1 30AB E3C4 79BE 3E43 0041 1886

Please see screenshot below (will open in a new window):

Unless I get "BAD signature"⁵, I will stop the verification process here, but you may decide to continue:

'You will now need to verify that the key used to sign the archive really does belong to the owner...'

From my current directory at /usr/src/build/tlacauhtli-3.14.0/ I change prompt into deeper build directory at /usr/src/build/tlacauhtli-3.14.0/build/, thus:

$ cd build

And I proceed to extract kernel 3.14 from previous directory:

$ tar -xvJPf ../linux-3.14.tar.xz

which will create directory ./linux-3.14 where it will extract the kernel source.

Then I proceed to git clone aufs3.x standalone to patch the kernel tree, since I want to have aufs3 as a module:

$ git clone git://git.code.sf.net/p/aufs/aufs3-standalone aufs3-standalone.git

and change into the newly git cloned aufs3 directory

$ cd aufs3-standalone.git

And referencing the minor version of the kernel we are building 3.14 (i.e. 14, number in bold font face) we enter the command:

$ git checkout origin/aufs3.14
(sample output)

error: pathspec 'origin/aufs3.14' did not match any file(s) known to git.

, ok...

UPDATE 04-09-2014: aufs3.x -- where x is the minor version -- now matches kernel 3.14 minor version. Thus, the above git directive should work and no longer should output an error. Evidently, the next git directive below is not necessary.

Rereading the aufs.sourceforge.net⁶ we realize that aufs3.x (where x is the minor version) does not necessarily follow the kernel minor version. And when that is the case, as in our current efforts, we should reference the nearest aufs.x lower version number; thus we rerun our command as:

$ git checkout origin/aufs3.13
(sample output)

Note: checking out 'origin/aufs3.13'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b new_branch_name

HEAD is now at 75dbb99... aufs3.13 20140303

Well that worked. We will continue with the application of the aufs3.x patch to the kernel 3.14 source tree slightly modified from section "'it becomes clear that 'Aufs was rejected. Let's give it up.' According to Christoph Hellwig, linux rejects all union-type filesystems but UnionMount." -- in prior relevant post -- where it starts with the text line:

"and proceed to create a couple of directories to prepare a patch for the linux kernel tree"

$mkdir --verbose ../x ../y

How you name those remains at your discretion but I named them x and y. The following sequence of commands should operate on the directory ../y/.

Code

In the above commands, the verbosty (-v) is optional.

Make sure to remove Kbuild:

$rm -v ../y/include/uapi/linux/Kbuild

else you should specify 'no' when prompted by the patch utility whether it should replace Kbuild in your linux kernel tree.

We go back to our previous directory, i.e., build:

$cd ..

And perform the following operations:

Code

We are ready to apply the patch; we change directory to our linux kernel tree source:

$cd linux-3.14

And apply the patch:

$patch -p1 < aufs.patch

You will observe a long output to your bash shell but no errors.

Yolahuialtia [Cheers!] You have just patched your latest stable linux kernel tree

As I am upgrading from my currently running linux kernel 3.13.7, I will import the relevant .config into my spanking new and Aufs-patched linux kernel tree:

$cat /boot/config-`uname -r` >.config

and do:

$make oldconfig

After the above command completes, if we do:

$ make xconfig

We want to know if the following options/resources are enabled⁴:

# CONFIG_CGROUP_DEVICE is not set
# CONFIG_BLK_CGROUP is not set
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set

Thus, we first search for the string cgroup:

we must enable (if those are not already) the shown options/resources:

subsequently searching for text string devpts we should make sure its association is enabled:

If those options/resources in the kernel .config were all that we enabled, and we proceeded to (re)build our kernel the [error] client.go:2298 Error getting size: bad file descriptor would go away and docker.io would run our command inside our container(s) -- as it should.

However, if we were to check the LXC virtualization layer with the command:

$ lxc-checkconfig

--- Namespaces ---
...
User namespace: missing
...
--- Control groups ---
...
Cgroup memory controller: missing
Cgroup cpuset: missing

In addition, getting information on docker.io with the command:

$ docker.io info

WARNING: No memory limit support
WARNING: No swap limit support

With respect to this latter WARNING issues, helpful individuals offer advice⁷: yielding root privilege open in a text editor (vi, vim, gedit, etc.) your Debian file /etc/default/grub.

# vim /etc/default/grub

Locate the following similar line:

GRUB_CMDLINE_LINUX=""

between the quotation marks insert cgroup_enable=memory swapaccount as shown next:

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

And save your modifications to your file. Please note that the 1 argument to swapaccount is optional since enable, i.e., 1 is the default, whereas argument 0 disables swapaccount. Subsequently you are advised to acquire root privilege and run command:

# update-grub

What it does is it modifies your Debian /boot/grub/grub.cfg file by adding the above argument to your existing kernel directive. Shown below is the fragment after the root directive:

... root=UUID=xyz ro cgroup_enable=memory swapaccount quiet

Notwithstanding, after rebooting our system and executing:

# docker.io info

we find that the WARNING: No memory limit support and
WARNING: No swap limit support did not go away.

We may even install additional cgroup-related packages⁸:

# apt-get install cgroup-bin

to no avail...

What we need to do is take care of the "missing" resources⁹ shown to us when we execute:

# lxc-checkconfig

so that a subsequent directive:

# docker.io info

will not output issues as shown once again in the snapshot below:

Hence when we do:

$ make xconfig

and from the ensuing graphical interface we search for text string RESOURCE and subsequently place a check mark next to:

• Cpuset support
• include legacy /proc/<pid>/cpuset file
• Simple CPU accounting cgroup subsystem
• Resource counters
• Memory Resource Controller for Control Groups

...will cascade a couple of optons that should be:

• check Memory Resource Controller Swap Extension (NEW)
• check Memory Resource Controller Swap Extension enabled by default
• check Memory Resource Controller Kernel Memory accounting (NEW)

Once the above are marked (and saved) those will take care of WARNING issue output by the docker.io info command; as well as the "missing" Cgroup memory controller and Cgroup cpuset output during lxc-checkconfig command.

Now to resolve the "missing" User namespace output during lxc-checkconfig command, we search for USER text string and place a mark next to:

• User namespace

As illustrated in the screenshot below:

Now saving our changes to our kernel 3.14 .config file, the moment to build our Debian custom linux kernel for Docker and aufs3 support has finally arrived! Do:

$fakeroot make-kpkg clean

And I proceed to build my kernel: Tezcatlipoca¹⁰ which prefixed with a leading dot, I provide as an argument to --append-to-version in the directive that follows:

$time fakeroot make-kpkg --append-to-version=.tezcatlipoca --stem aufs3+cgroup -j8 --initrd kernel_image kernel_headers

I also provide aufs3+cgroup as an argument to --stem because I want to know the main reason for my kernel customization. Additionally, I provide 8 as an argument to -j. Eight(8) represents the number of threads I wish to launch, hence I make it equal to the number of cores in my machine where I am building the kernel -- for optimum performance during compilation.

After installing my Tezcatlipoca kernel, I execute:

# lxc-checkconfig

followed by

# docker.io info

and below is a screenshot of the result:

We are ready to start playing with Docker in our Debian system

---

References:

¹Docker's red-hot application portability solution -- Infoworld

²A Brief Introduction to Using Docker

"Error resize: Error: bad file descriptor"

³Drupal on Docker

⁴[solved] Docker lxc-start "cgroup is not mounted"

⁵ "If you get "BAD signature"

If at any time you see "BAD signature" output from "gpg --verify", please first check the following first:

"Make sure that you are verifying the signature against the .tar version of the archive, not the compressed (.tar.xz) version.
Make sure the the downloaded file is correct and not truncated or otherwise corrupted.
If you repeatedly get the same "BAD signature" output, email ftpadmin[at]kernel[dot]org immediately, so we can investigate the problem."

⁶The minor version number, 'x' in '3.x', of aufs may not always
follow the minor version number of the kernel.
Because changes in the kernel that cause the use of a new
minor version number do not always require changes to aufs-util.

Since aufs-util has its own minor version number, you may not be
find a GIT branch in aufs-util for your kernel's
exact minor version number.
In this case, you should git-checkout the branch for the
nearest lower number.

⁷Testing Docker Install

⁸PDF: 3.16.1 Linux Control Groups (cgroups) in the judgedaemon

⁹Gentoo wiki: LXC

¹⁰Tezcatlipoca, i.e., Smoking Black Mirror ≈ Espejo Negro Humeante, in Nahuatl -- Mexico's language par excellence: tezcatl (mirror: espejo) + tliltic (black: negro) + pocatl (smoke: humo). Tezcatlipoca, also known as Xonecuiltzin, is depicted in the image at the very top of this post.

---

DISCLAIMER although due diligence has been applied, this resource is made available for testing/evaluation purposes on an AS IS basis. The procedure only reflects my own modifications, my limited testing, and the potential user who executes the procedures assumes all risks.

Please do not hold me or Metztli Information Technology responsible if the information provided here does not achieve the desired result. The information is provided AS IS and with the hope that it may be useful to the Internet community --especially those who need Aufs3.x and/or Docker support on Debian.

Notwithstanding, There is no implicit or explicit guarantee that the information presented here is accurate --even though due diligence was exercised during the procedure. Accordingly, if an user(s) decide to implement the procedure or shell commands described here she, he, or them, do so at her, his, or their own risk. You have been forewarned.