Building New Reiser4, Software Format Release Number 4.0.2, and Zstd Compression, Debian Stretch-Backports Kernel.
"ZSTD compression support in Reiser4 is brought for you by BratSinot. We recommend it as a replacement of gzip1. To enjoy ZSTD compression, please build and install kernel stuff for linux-4.14, reiser4progs-1.2.0, and specify mkfs option "-o compress=zstd1" fragment of Sun, Nov 26, 2017 at 3:01 PM, email sent by Edward Shishkin to reiserFS mailing list.
An open-minded developer suddenly made available a bleeding edge feature, Zstandard, is a fast lossless compression algorithm, targeting real- time compression scenarios at zlib-level and better compression ratios for Reiser4, at the time whose main developer incidentally announced a new software framework release 4.0.2 of Reiser4 kernel module and Reiser4progs. Fascinating.
Building New Reiser4 SFRN 4.0.2 Linux Kernel 4.14.x With Default Zstd Module.
Given the fact that Debian, not even in their Unstable (Sid) distribution, had implemented Zstd for any of their supported file systems, I came across a problem which probably I did not encounter before during my reiser4 -patched builds of Debian kernels: a duplicate modules error halted my UDEBs from being built.
make[2]: Entering directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
dh_testdir
dh_prep
kernel-wedge install-files 4.14.0-1+reiser4.0.2
install -D -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/boot/vmlinuz-4.14.0-1+reiser4.0.2-amd64 debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/boot/vmlinuz
install -d debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/lib/modules/4.14.0-1+reiser4.0.2-amd64
install -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/lib/modules/4.14.0-1+reiser4.0.2-amd64/modules.builtin debian/linux-image-4.14.0-1+reiser4.0.2-amd64/lib/modules/4.14.0-1+reiser4.0.2-amd64/modules.order debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/lib/modules/4.14.0-1+reiser4.0.2-amd64/
install -D -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/boot/System.map-4.14.0-1+reiser4.0.2-amd64 debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/boot/System.map
kernel-wedge copy-modules 4.14.0-1+reiser4.0.2 amd64 4.14.0-1+reiser4.0.2-amd64
kernel-wedge find-dups 4.14.0-1+reiser4.0.2-amd64
some modules are in more than one package
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/xxhash.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/xxhash.ko
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_compress.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_compress.ko
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_decompress.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_decompress.ko
command exited with status 1
make[2]: *** [install-udeb_amd64] Error 2
debian/rules.real:498: recipe for target 'install-udeb_amd64' failed
make[2]: Leaving directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
make[1]: *** [binary-arch_amd64] Error 2
debian/rules.gen:18: recipe for target 'binary-arch_amd64' failed
make[1]: Leaving directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
make: *** [binary-arch] Error 2
debian/rules:50: recipe for target 'binary-arch' failed
dpkg-buildpackage: error: fakeroot debian/rules binary-arch gave error exit status 2
Fact is developer at facebook had created Zstd patches for that other filesystem and (I speculate) that subsequent Zstd patch for reiser4 was modeled after it. Obviously, the problem I encountered only emerges building the Linux kernel 'The Debian way' because we are creating UDEBs for Debian-Installer (d-i) in addition to regular DEB packages.
So kernel.org just made available Linux kernel 4.14.6 source which minor number is higher than current latest Debian kernel source and packaging (4.14.2-2) that I intend to make available for stretch-backports. At the moment I am running Debian-packaged kernel 4.14.4-2, SFRN 4.0.2, with Zstd -- which I successfully test/Installed from Metztli Reiser4 netboot image 'burned' to USB (see the video teaser at the start of this post and Installation Reports output for a proof of concept (PoC) Successful Reiser4 SFRN 4.0.2 installation with Linux 4.14.4-1 with Zstd compression on HP Notebook )
Also installed relevant linux-headers, linux-compiler-, linux-kbuild-, linux-perf-, packages generated by my reiser4 -enabled kernel build for stretch-backports and available at SourceForge:
Shell
dpkg -i linux-headers-4.14.0-1+reiser4.0.2-amd64_4.14.4-1+reiser4.0.2_amd64.deb linux-headers-4.14.0-1+reiser4.0.2-common_4.14.4-1+reiser4.0.2_all.deb linux-kbuild-4.14_4.14.4-1+reiser4.0.2_amd64.deb linux-perf-4.14_4.14.4-1+reiser4.0.2_amd64.deb linux-compiler-gcc-6-x86_4.14.4-1+reiser4.0.2_amd64.deb |
Thus I begin the pre-build preparation procedure. Since I want to build the latest kernel available from kernel.org I need to patch the current lower minor version available from Debian. Reiterating, this is a hack we are performing on 'official' unstable (Sid) Debian kernel packaging; thus there is no guarantee of anything and we knowingly assume all responsibility for the outcome.
I get the source for Debian latest unstable kernel by adding the following directive to my Stretch distribution and using a text editor like xvi from Source Forge -- which is included in Metztli Reiser4 netboot Debian-Installer (d-i) -- and open...
Shell
xvi /etc/apt/sources.list |
add following directive:
deb-src http://ftp.us.debian.org/debian/ sid main non-free contrib
Save modifications, refresh our Debian repositories and fetch latest available Linux source from unstable (Sid).
Shell
apt-get update | |
apt-get source -d linux |
After we get our unstable Debian Linux kernel source, we may want to disable the unstable directive we added a priori by prefixing it with a couple of hashes, thus:
## deb-src http://ftp.us.debian.org/debian/ sid main non-free contrib
Again save our modifications and refresh our repositories so as not to have any references to Debian unstable sources:
Shell
apt-get update |
Depending upon our build kernel preferences we may want to download unstable (Sid) Debian kernel packaging.
Shell
git clone -b sid --single-branch https://anonscm.debian.org/git/kernel/linux.git |
OR download Debian Linux kernel master packaging:
Shell
git clone https://anonscm.debian.org/git/kernel/linux.git master |
Analyzing the linux/debian/changelog and master/debian/changelog we see Sid and master both having a lower minor number than kernel upstream at kernel.org. Thus we will patch Debian kernel with incremental patches from upstream.
We will use the Debian Linux kernel packaging for Sid that we downloaded at linux directory, hence
Shell
cd linux |
Reiser4, SFRN 4.0.2, Enhancing The Debian Unstable (Sid) Kernel Packaging.
In previous blog: Coyolxauhqui: Build Reiser4 -patched Linux Kernel, Headers, and Modules, for Stretch-Backports the 'Debian Way', we created debian-packaging_reiser4-support.patch. Let us apply it, first, assuming it is located at ../debian-packaging_reiser4-support.patch
Shell
cat ../debian-packaging_reiser4-support.patch | patch -p1 |
as explained a priori, it will reiser4-enhance the following Debian Linux kernel packaging files for AMD64:
debian/config/amd64/defines
debian/installer/amd64/modules/amd64/reiser4-modules
debian/installer/amd64/package-list
debian/installer/modules/reiser4-modules
debian/installer/package-list
Now let's add reiser4-enhanced support for i386 (have not tested but based on working AMD64 paradigm by creating file reiser4-modules in proper i386 architecture directory:
Shell
cat >>debian/installer/i386/modules/i386/reiser4-modules<< EOF | |
#include <reiser4-modules> | |
EOF |
and modifying existing debian/installer/i386/package-list by adding the following two(2) directives:
Package: reiser4-modules
Priority: standard
Thus we visualize the added i386 reiser4 -enhanced support:
We are done enhancing Debian Linux kernel packaging with reiser4, Software Format Release Number (SFRN) 4.0.2.
Stretch-Backporting The Debian Unstable (Sid) Kernel Packaging.
Starting with Linux 4.14.x, Debian Unstable kernel packaging specifies GCC-7 to build the kernel. Accordingly, first task is we must identify relevant Debian kernel packaging files to modify for GCC-6 to build our stretch-backports kernel:
debian/config/amd64/defines
debian/config/armel/defines
debian/config/armhf/defines
debian/config/hppa/defines
debian/config/i386/defines
debian/config/m68k/defines
debian/config/s390x/defines
debian/config/defines
debian/config/kernelarch-x86/config
debian/templates/control.extra.in
We can place the list above in a file, say ../chinga2GCC6.text, and modify, in place, all of them at once by replacing two(2) more precise five(5) characters gcc-7 representing GCC7 with gcc-6 to specify GCC6:
Shell
for i in $(< ../chinga2GCC6.text); do sed -i 's/gcc-7/gcc-6/g' $i; done |
Again, as in prior referenced post, we may want to create a patch so that we may apply all the above changes in our future kernel build efforts:
Shell
git add . | |
git commit -a -m "Reiser4 SFRN 4.0.2 and GCC6 stretch-backports" | |
git format-patch --stdout -1 >../reiser4-gcc6.patch |
Let's take a peek at ../reiser4-gcc6.patch (link to download) we just created:
Tell dpkg-buildpackage To Ignore Duplicate UDEBs And Finish Build Procedure
Shell
touch debian/installer/amd64/ignore-dups | |
touch debian/installer/i386/ignore-dups |
That's it! 1
Since our downloaded Debian unstable (Sid) Linux kernel and packaging is currently 4.14.2-2, we download upstream Linux kernel incremental patches from kernel.org:
Shell
cd .. | |
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/incr/patch-4.14.2-3.xz | |
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/incr/patch-4.14.3-4.xz | |
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/incr/patch-4.14.4-5.xz | |
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/incr/patch-4.14.5-6.xz | |
cd - |
Let's execute a couple of directives 'the debian way':
Shell
debian/rules orig | |
debian/rules debian/control |
Please note that in this Debian Linux kernel packaging, the developer has enabled real-time (RT) kernel build again, so the next directive will have an effect -- unlike previous instances where it had no effect:
Shell
fakeroot debian/rules source |
if desired to prevent build of real-time kernel and speed the build slightly, you may disable it in debian/config/defines
...
[featureset-rt_base]
enabled: true modify directive as below:
enabled: false
Nevertheless, in this build I will leave default, i.e., build real-time kernel, too.
Now we are ready to apply the incremental patches we downloaded from upstream kernel.org. We have to be careful as there is some overlap which we will solve by downgrading target files with those at ../orig/linux-4.14.2 untarred by Debian Linux kernel packaging a priori:
Before applying incremental patch-4.14.2-3, files below must be downgraded to allow the patch to apply smoothly upgrading them again:
drivers/media/usb/cx231xx/cx231xx-cards.c
fs/notify/mark.c
Accordingly, I copy over the two(2) files above, downgrading them in the process:
Shell
cp -vf ../orig/linux-4.14.2/drivers/media/usb/cx231xx/cx231xx-cards.c drivers/media/usb/cx231xx/. | |
cp -vf ../orig/linux-4.14.2/fs/notify/mark.c fs/notify/. |
then patch-4.14.2-3 upgrades them again while applying smoothly:
Shell
xz -dc ../patch-4.14.2-3.xz | quilt fold |
Next, prior to application of patch-4.14.3-4 we downgrade the next two files:
mm/huge_memory.c
security/apparmor/include/audit.h
Shell
cp -vf ../orig/linux-4.14.2/mm/huge_memory.c mm/. | |
cp -vf ../orig/linux-4.14.2/security/apparmor/include/audit.h security/apparmor/include/. |
then apply patch-4.14.3-4 which will apply smoothly and will upgrade the files to their previous level:
Shell
xz -dc ../patch-4.14.3-4.xz | quilt fold |
I then continue by applying incremental patch-4.14.4-5. We do not need to downgrade any files:
Shell
xz -dc ../patch-4.14.4-5.xz | quilt fold |
Finally, I apply patch-4.14.5-6. As above, there is no need to selectively downgrade any files:
Shell
xz -dc ../patch-4.14.5-6.xz | quilt fold |
Needless to say, all the incremental patches should end smoothly applying themselves -- as tail output screenshot of this last procedure shows:
Applying Reiser4, SFRN 4.0.2 to the Kernel Source Tree Itself.
I go back onto parent directory and download Reiser4, SFRN 4.0.2 developer Edward Shishkin's reiser4-for-4.14.1.patch.gz
Shell
cd .. | |
wget https://sourceforge.net/projects/reiser4/files/reiser4-for-linux-4.x/reiser4-for-4.14.1.patch.gz | |
cd - |
I return to my previous location in file system and apply reiser4-for-4.14.1.patch.gz to linux source tree:
Shell
gzip -dc ../reiser4-for-4.14.1.patch.gz | quilt fold |
We edit debian/config/defines to concatenate text string +reiser4.0.2 to its ABI:
Shell
xvi debian/config/defines |
do not forget to save changes.
Then we record our changes in debian/changelog with the dch tool, making sure to update to real kernel version and concatenate text string +reiser4.0.2, too:
Shell
dch -D metztli |
We continue with 'the debian way' directives:
Shell
debian/rules debian/control |
We should expect a couple of errors -- only after this instance.
then, the following directive should only yield an expected error -- only one:
Shell
fakeroot debian/rules debian/control-real |
We proceed with:
Shell
fakeroot make -f debian/rules.gen setup_amd64_none_amd64 |
-- which should end with no errors.
Now let's modify Linux kernel .config and opt to build reiser4 as module:
Shell
make -C debian/build/build_amd64_none_amd64 menuconfig |
I save modifications to reiser4 -enabled .config:
and exit successfully the GUI:
And finally I start the build procedure at 10:30 pm PST, on Saturday, Dec. 16, 2017:
Shell
dpkg-buildpackage -F -us -uc -j8 -T binary-arch,binary-indep |
-j8 : eight cores allocated, that's the max in my local development environment in the absence of an IBM POWER9 Talos II Debian workstation
Approximately two(2) hours later, build procedure finishes.
30 DEB packages are successfully generated:
as well as 52 UDEB packages suitable for Metztli Reiser4 Debian-Installer (d-i):
UDEB first screenshot:
UDEB second screenshot:
Shell
echo "The sublime and beautiful Russian feminine archetype can not be abstracted from my computing." |
At Metztli IT, I eat own cooking. Here is this newly-built non-rt Linux kernel 4.14.6-2 being installed over my current 4.14.4-1 in my development environment:
References:
1Packaging for kernel-wedge. Reiterating, Debian, not even in their Unstable (Sid) distribution, has implemented Zstd for any of their supported file systems.
DISCLAIMER although due diligence has been applied, this resource is made available for testing/evaluation purposes on an AS IS basis. The procedure only reflects my own modifications, my limited testing, and the potential user(s) who execute(s) the procedures assumes all risks.
Please do not hold me or Metztli Information Technology (and/or its associates) responsible if the information provided here does not achieve the desired result. The information is provided AS IS and with the hope that it may be useful to the Internet community --especially those open-minded individuals interested in Reiser4 continued development.
Notwithstanding, There is no implicit or explicit guarantee that the information presented here is accurate --even though due diligence was exercised during the procedure. Accordingly, if an user(s) decide to implement the procedure or shell commands described here she, he, or them, do so at her, his, or their own risk. You have been forewarned.
Metztli IT, but not other entities, reserves the right to modify the content -- to correct and/or elucidated procedure(s), for instance -- and/or even delete all or partial, including blog post, without previous notice.