Tag: "systemd"

  • In an ageing reiser4 development environment where I still build Metztli Reiser4 and, recently, Metztli Reiser5 kernels and hack the Debian Installer (d-i) minimal netboot images that I make available in SourceForge site, I began to notice systemd errors in dmesg logs.
    ...

    [ 3821.858183] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
    [ 3821.858285] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
    [ 3821.859532] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
    [ 3821.860945] systemd[1]: Stopping Flush Journal to Persistent Storage...
    [ 3821.870796] systemd[1]: systemd-journal-flush.service: Succeeded.
    [ 3821.871516] systemd[1]: Stopped Flush Journal to Persistent Storage.
    [ 3821.872296] systemd[1]: Stopped Journal Service.
    [ 3821.875407] systemd[1]: Starting Journal Service...
    [ 3912.076346] systemd[1]: systemd-journald.service: start operation timed out. Terminating.
    [ 4002.329312] systemd[1]: systemd-journald.service: State 'stop-sigterm' timed out. Killing.
    [ 4002.329476] systemd[1]: systemd-journald.service: Killing process 11100 (systemd-journal) with signal SIGKILL.
    [ 4092.582238] systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.
    [ 4182.835076] systemd[1]: systemd-journald.service: State 'final-sigterm' timed out. Killing.
    [ 4182.835156] systemd[1]: systemd-journald.service: Killing process 11100 (systemd-journal) with signal SIGKILL.
    [ 4224.746769] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
    [ 4224.746796] systemd[1]: systemd-journald.service: Failed with result 'timeout'.
    [ 4224.748891] systemd[1]: Failed to start Journal Service.
    [ 4224.749509] systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
    [ 4224.749605] systemd[1]: systemd-journal-flush.service: Job systemd-journal-flush.service/start failed with result 'dependency'.
    [ 4224.753055] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 2.
    [ 4224.754325] systemd[1]: Stopped Journal Service.
    [ 4224.756880] systemd[1]: Starting Journal Service...
    [ 4239.468189] systemd[1]: Started Journal Service.

    ...
    I was already using Debian Buster backports and there was no Debian Systemd package, nor source, newer than what I was using. Moreover, the systemd issue was introducing peculiar erratic computing behaviour in the GUI user land, Enlightenment Window Manager 0.24.2 in my case. So I had to fix the issue and headed to the GitHub Systemd development repository --where there were newer commits ahead of the latest official source found at Debian repositories. And I decided to give the systemd build a try, locally.

    After the test-acl-util stopped my build attempts I filed an issue: systemd build fails on reiser4 4.0.2 #17013. I was not sure if my issue would get noticed for this was my first incursion into building systemd and likely not many individuals build on reiser4. Luckily, on the next day I received a comment from a dedicated person, whose last comment included the phrase I used as the topic of this post. He was kind enough to modify and commit a few lines of code so that file systems which do not currently support ACLs, like reiser4, would be able to skip the test-acl-util during the build procedure. I closed the issue, subsequently.

    That made my systemd build go further; yet, my build was experiencing TIMEOUTS which failed the build procedure:

    test-journal                                              TIMEOUT 30.32 s
    ...
    463/562 test-journal-stream             TIMEOUT 30.17 s
    ...
    468/562 test-journal-interleaving    TIMEOUT 31.02 s

    I tried the systemd build in another relatively clean partition on this same ageing hardware with same fail result. After multiple failures in both local reiser4 partitions I reopened the issue erroneously believing that if the test code and headers were completely on systemd git repository probably -- just probably -- the developers never envisioned any reiser4 peculiarities. I arrived at that irrational conclusion by analyzing some of the source files involved -- where I noted some comments to accommodate a btrfs quirk.

    On the next day I saw my reopened issue closed again by a different systemd contributor. Well, I thought that was it and I began to explore the possibility of installing OpenRC, after reading an older article in Linux Mafia (link to non-SSL site). Researching further, I was pleasantly surprised that some well known Debian developers maintain OpenRC and decided to give it a try and build it as a potential replacement for my currently failing systemd. Fact is OpenRC is hosted at GitHub, too, and the source readily built under the Debian packaging framework in my reiser4 environment. OpenRC was not a clean Systemd replacement as I had to spend some time visiting Devuan (Debian without Systemd) and ripping out the dbus innards from Beowolf to use in my quest. And then stripping the Systemd references in Pulseaudio source and Debian packaging to fix an Enlightenment volume control application issue. But the replacement worked:

    'maybe reiser4 is just terribly slow for this access pattern?'
    Installing non-systemd Pulseaudio

    Notwithstanding, a day later I got another message from the first person who committed the skip fix for non-acl supporting file systems in the GitHub Systemd master code repository; the person made the suggestion, 'maybe reiser4 is just terribly slow for this access pattern?', i.e., the topic of this post. And I decided to dispel any doubts about systemd build on reiser4 once and for all.

    I have a spare ~200Gb custom Metztli Reiser4 Zstd transparent compression disk snapshot saved in the cloud, more specifically Google Cloud Platform (GCP). First, I generated a Google Compute Engine (GCE) disk with my saved disk snapshot, moiocoiatzin-epyc-zstd-cloud-kernel, in the US Central1 region where I could create an AMD Epyc -based cloud instance.

    Shell

    gcloud compute disks create moiocoiatzin --description "AMD64 Rome Epyc" --labels=amd64=epyc --source-snapshot=moiocoiatzin-epyc-zstd-cloud-kernel --zone us-central1-a

    Subsequently, I created an instance with moiocoiatzin -- an allusion to 'Entity who self-creates, self-thinks, self-invents, or wills, Himself into Being', i.e., the disk we emerged 'a priori':

    Shell

    gcloud beta compute instances create iztaccihuatl --disk name=moiocoiatzin,mode=rw,boot=yes --metadata enable-guest-attributes=TRUE --metadata-from-file startup-script=./local-directory/reghostkey.sh --tags chingon --machine-type=n2d-standard---zone us-central1-a

    And named the cloud instance as Iztaccihuatl -- an allusion to the real Mexicah's notion of a 'White woman' in reference to a mountain range / volcano in the valley of their eponymous city-state which, when covered with snow, projects the shape of a sleeping woman underneath. Other options include the disk properties read/write and must have the ability to boot. Our Debian -based Metztli Reiser4 cloud images are created and customized for the Google Compute Engine; thus, for security of whoever uses our images, I remove the /etc/ssh/ssh_host_* files as these are regenerated upon booting the instance with a script, say reghostkey.sh, which contents might be:

    #! /bin/bash
    # Regenerates deleted host keys upon GCE image first boot
    sudo su -
    apt-get -t buster-backports update
    dpkg-reconfigure openssh-server
    ##/etc/init.d/ssh restart
    systemctl try-reload-or-restart ssh

    Obviously, if our cloud image were running openrc -- and not systemd -- we would uncomment the second to last directive and comment out the last one.

    I am also specifying the machine type as an AMD Epyc (N2D) with eight(8)-cores and 32 Gb RAM, i.e., standard.

    After upgrading the older software which came from the snapshot -- including the reiser4 kernel labeled as 'reizer4' to distinguish it from tlilxochitl, i.e., vanilla, Intel reiser4 kernels, I uploaded my local systemd master source code repository which my ageing eight(8) core / 16 Gb RAM machine was unable to build.

    As can be seen in the media, the Metztli Reiser4 environment in the cloud completed the systemd build in approximately seven(7) minutes. And I can certainly assert that the systemd build issues that I experienced were not because 'reiser4 is just terribly slow for this access pattern' but, rather, those issues were due to my ageing local hardware platform.


    Image reference: Вики Одинцова : Viki Odintcova, aka, Вики Xochiquetzal

    NOTE: Although the commands and/or code discussed above generally work well, please consider these are for illustrative purposes. and are provided AS-IS without any implicit/explicit assurance as to their correctness.