Metztli I. T.  Blog

"ZSTD compression support in Reiser4 is brought for you by BratSinot. We recommend it as a replacement of gzip1. To enjoy ZSTD compression, please build and install kernel stuff for linux-4.14, reiser4progs-1.2.0, and specify mkfs option "-o compress=zstd1" fragment of Sun, Nov 26, 2017 at 3:01 PM, email sent by Edward Shishkin to reiserFS mailing list.

An open-minded developer suddenly made available a bleeding edge feature, Zstandard, is a fast lossless compression algorithm, targeting real- time compression scenarios at zlib-level and better compression ratios for Reiser4, at the time whose main developer incidentally announced a new software framework release 4.0.2 of Reiser4 kernel module and Reiser4progs. Fascinating.

Building New Reiser4 SFRN 4.0.2 Linux Kernel 4.14.x With Default Zstd Module.

Given the fact that Debian, not even in their Unstable (Sid) distribution, had implemented Zstd for any of their supported file systems, I came across a problem which probably I did not encounter before during my reiser4 -patched builds of Debian kernels: a duplicate modules error halted my UDEBs from being built.

make[2]: Entering directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
dh_testdir
dh_prep
kernel-wedge install-files 4.14.0-1+reiser4.0.2
install -D -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/boot/vmlinuz-4.14.0-1+reiser4.0.2-amd64 debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/boot/vmlinuz
install -d debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/lib/modules/4.14.0-1+reiser4.0.2-amd64
install -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/lib/modules/4.14.0-1+reiser4.0.2-amd64/modules.builtin debian/linux-image-4.14.0-1+reiser4.0.2-amd64/lib/modules/4.14.0-1+reiser4.0.2-amd64/modules.order debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/lib/modules/4.14.0-1+reiser4.0.2-amd64/
install -D -m 644 debian/linux-image-4.14.0-1+reiser4.0.2-amd64/boot/System.map-4.14.0-1+reiser4.0.2-amd64 debian/kernel-image-4.14.0-1+reiser4.0.2-amd64-di/boot/System.map
kernel-wedge copy-modules 4.14.0-1+reiser4.0.2 amd64 4.14.0-1+reiser4.0.2-amd64
kernel-wedge find-dups 4.14.0-1+reiser4.0.2-amd64
some modules are in more than one package
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/xxhash.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/xxhash.ko
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_compress.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_compress.ko
debian/reiser4-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_decompress.ko
debian/btrfs-modules-4.14.0-1+reiser4.0.2-amd64-di lib/modules/4.14.0-1+reiser4.0.2-amd64/kernel/lib/zstd/zstd_decompress.ko
command exited with status 1
make[2]: *** [install-udeb_amd64] Error 2
debian/rules.real:498: recipe for target 'install-udeb_amd64' failed
make[2]: Leaving directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
make[1]: *** [binary-arch_amd64] Error 2
debian/rules.gen:18: recipe for target 'binary-arch_amd64' failed
make[1]: Leaving directory '/usr/tzinti/build/kernel/tekitl-4.14.2/linux'
make: *** [binary-arch] Error 2
debian/rules:50: recipe for target 'binary-arch' failed
dpkg-buildpackage: error: fakeroot debian/rules binary-arch gave error exit status 2

Fact is developer at facebook had created Zstd patches for that other filesystem and (I speculate) that subsequent Zstd patch for reiser4 was modeled after it. Obviously, the problem I encountered only emerges building the Linux kernel 'The Debian way' because we are creating UDEBs for Debian-Installer (d-i) in addition to regular DEB packages.

So kernel.org just made available Linux kernel 4.14.6 source which minor number is higher than current latest Debian kernel source and packaging (4.14.2-2) that I intend to make available for stretch-backports. At the moment I am running Debian-packaged kernel 4.14.4-2, SFRN 4.0.2, with Zstd -- which I successfully test/Installed from Metztli Reiser4 netboot image 'burned' to USB (see the video teaser at the start of this post and Installation Reports output for a proof of concept (PoC) Successful Reiser4 SFRN 4.0.2 installation with Linux 4.14.4-1 with Zstd compression on HP Notebook )

Also installed relevant linux-headers, linux-compiler-, linux-kbuild-, linux-perf-, packages generated by my reiser4 -enabled kernel build for stretch-backports and available at SourceForge:

Shell

Thus I begin the pre-build preparation procedure. Since I want to build the latest kernel available from kernel.org I need to patch the current lower minor version available from Debian. Reiterating, this is a hack we are performing on 'official' unstable (Sid) Debian kernel packaging; thus there is no guarantee of anything and we knowingly assume all responsibility for the outcome.

I get the source for Debian latest unstable kernel by adding the following directive to my Stretch distribution and using a text editor like xvi from Source Forge -- which is included in Metztli Reiser4 netboot Debian-Installer (d-i) -- and open...

Shell

add following directive:
deb-src http://ftp.us.debian.org/debian/ sid main non-free contrib

Save modifications, refresh our Debian repositories and fetch latest available Linux source from unstable (Sid).

Shell

After we get our unstable Debian Linux kernel source, we may want to disable the unstable directive we added a priori by prefixing it with a couple of hashes, thus:

## deb-src http://ftp.us.debian.org/debian/ sid main non-free contrib

Again save our modifications and refresh our repositories so as not to have any references to Debian unstable sources:

Shell

Depending upon our build kernel preferences we may want to download unstable (Sid) Debian kernel packaging.

Shell

OR download Debian Linux kernel master packaging:

Shell

Analyzing the linux/debian/changelog and master/debian/changelog we see Sid and master both having a lower minor number than kernel upstream at kernel.org. Thus we will patch Debian kernel with incremental patches from upstream.

We will use the Debian Linux kernel packaging for Sid that we downloaded at linux directory, hence

Shell

Reiser4, SFRN 4.0.2, Enhancing The Debian Unstable (Sid) Kernel Packaging.

In previous blog: Coyolxauhqui: Build Reiser4 -patched Linux Kernel, Headers, and Modules, for Stretch-Backports the 'Debian Way', we created debian-packaging_reiser4-support.patch. Let us apply it, first, assuming it is located at ../debian-packaging_reiser4-support.patch

Shell

as explained a priori, it will reiser4-enhance the following Debian Linux kernel packaging files for AMD64:

debian/config/amd64/defines
debian/installer/amd64/modules/amd64/reiser4-modules
debian/installer/amd64/package-list
debian/installer/modules/reiser4-modules
debian/installer/package-list

Now let's add reiser4-enhanced support for i386 (have not tested but based on working AMD64 paradigm by creating file reiser4-modules in proper i386 architecture directory:

Shell

and modifying existing debian/installer/i386/package-list by adding the following two(2) directives:
Package: reiser4-modules
Priority: standard

Thus we visualize the added i386 reiser4 -enhanced support:

We are done enhancing Debian Linux kernel packaging with reiser4, Software Format Release Number (SFRN) 4.0.2.

Stretch-Backporting The Debian Unstable (Sid) Kernel Packaging.

Starting with Linux 4.14.x, Debian Unstable kernel packaging specifies GCC-7 to build the kernel. Accordingly, first task is we must identify relevant Debian kernel packaging files to modify for GCC-6 to build our stretch-backports kernel:

debian/config/amd64/defines
debian/config/armel/defines
debian/config/armhf/defines
debian/config/hppa/defines
debian/config/i386/defines
debian/config/m68k/defines
debian/config/s390x/defines
debian/config/defines
debian/config/kernelarch-x86/config
debian/templates/control.extra.in

We can place the list above in a file, say ../chinga2GCC6.text, and modify, in place, all of them at once by replacing two(2) more precise five(5) characters gcc-7 representing GCC7 with gcc-6 to specify GCC6:

Shell

Again, as in prior referenced post, we may want to create a patch so that we may apply all the above changes in our future kernel build efforts:

Shell

Let's take a peek at ../reiser4-gcc6.patch (link to download) we just created:

Tell dpkg-buildpackage To Ignore Duplicate UDEBs And Finish Build Procedure

Shell

That's it! ¹

Since our downloaded Debian unstable (Sid) Linux kernel and packaging is currently 4.14.2-2, we download upstream Linux kernel incremental patches from kernel.org:

Shell

Let's execute a couple of directives 'the debian way':

Shell

Please note that in this Debian Linux kernel packaging, the developer has enabled real-time (RT) kernel build again, so the next directive will have an effect -- unlike previous instances where it had no effect:

Shell

if desired to prevent build of real-time kernel and speed the build slightly, you may disable it in debian/config/defines
...
[featureset-rt_base]
enabled: true modify directive as below:
enabled: false

Nevertheless, in this build I will leave default, i.e., build real-time kernel, too.

Now we are ready to apply the incremental patches we downloaded from upstream kernel.org. We have to be careful as there is some overlap which we will solve by downgrading target files with those at ../orig/linux-4.14.2 untarred by Debian Linux kernel packaging a priori:

Before applying incremental patch-4.14.2-3, files below must be downgraded to allow the patch to apply smoothly upgrading them again:
drivers/media/usb/cx231xx/cx231xx-cards.c
fs/notify/mark.c

Accordingly, I copy over the two(2) files above, downgrading them in the process:

Shell

then patch-4.14.2-3 upgrades them again while applying smoothly:

Shell

Next, prior to application of patch-4.14.3-4 we downgrade the next two files:

mm/huge_memory.c
security/apparmor/include/audit.h

Shell

then apply patch-4.14.3-4 which will apply smoothly and will upgrade the files to their previous level:

Shell

I then continue by applying incremental patch-4.14.4-5. We do not need to downgrade any files:

Shell

Finally, I apply patch-4.14.5-6. As above, there is no need to selectively downgrade any files:

Shell

Needless to say, all the incremental patches should end smoothly applying themselves -- as tail output screenshot of this last procedure shows:

Applying Reiser4, SFRN 4.0.2 to the Kernel Source Tree Itself.

I go back onto parent directory and download Reiser4, SFRN 4.0.2 developer Edward Shishkin's reiser4-for-4.14.1.patch.gz

Shell

I return to my previous location in file system and apply reiser4-for-4.14.1.patch.gz to linux source tree:

Shell

We edit debian/config/defines to concatenate text string +reiser4.0.2 to its ABI:

Shell

do not forget to save changes.

Then we record our changes in debian/changelog with the dch tool, making sure to update to real kernel version and concatenate text string +reiser4.0.2, too:

Shell

We continue with 'the debian way' directives:

Shell

We should expect a couple of errors -- only after this instance.

then, the following directive should only yield an expected error -- only one:

Shell

We proceed with:

Shell

-- which should end with no errors.

Now let's modify Linux kernel .config and opt to build reiser4 as module:

Shell

I save modifications to reiser4 -enabled .config:

and exit successfully the GUI:

And finally I start the build procedure at 10:30 pm PST, on Saturday, Dec. 16, 2017:

Shell

-j8 : eight cores allocated, that's the max in my local development environment in the absence of an IBM POWER9 Talos II Debian workstation

Approximately two(2) hours later, build procedure finishes.
30 DEB packages are successfully generated:

as well as 52 UDEB packages suitable for Metztli Reiser4 Debian-Installer (d-i):
UDEB first screenshot:

UDEB second screenshot:

Shell

At Metztli IT, I eat own cooking. Here is this newly-built non-rt Linux kernel 4.14.6-2 being installed over my current 4.14.4-1 in my development environment:

References:
¹Packaging for kernel-wedge. Reiterating, Debian, not even in their Unstable (Sid) distribution, has implemented Zstd for any of their supported file systems.

DISCLAIMER although due diligence has been applied, this resource is made available for testing/evaluation purposes on an AS IS basis. The procedure only reflects my own modifications, my limited testing, and the potential user(s) who execute(s) the procedures assumes all risks.

Please do not hold me or Metztli Information Technology (and/or its associates) responsible if the information provided here does not achieve the desired result. The information is provided AS IS and with the hope that it may be useful to the Internet community --especially those open-minded individuals interested in Reiser4 continued development.

Notwithstanding, There is no implicit or explicit guarantee that the information presented here is accurate --even though due diligence was exercised during the procedure. Accordingly, if an user(s) decide to implement the procedure or shell commands described here she, he, or them, do so at her, his, or their own risk. You have been forewarned.

Metztli IT, but not other entities, reserves the right to modify the content -- to correct and/or elucidated procedure(s), for instance -- and/or even delete all or partial, including blog post, without previous notice.