Mailing List Archive

Systemd Was: Please get me straight about sysvinit vs. systemd [...]
Canek Peláez Valdés posted on Mon, 03 Mar 2014 11:58:18 -0600 as
excerpted:

> On Mon, Mar 3, 2014 at 11:44 AM, Duncan <1i5t5.duncan@cox.net> wrote:

>> Meanwhile, for the record, the systemd and now udev folks have stated
>> that they would like to eventually merge udev fully into systemd, and
>> indeed, it's already shipped as a single tarball, but that udev is
>> likely to remain a separate binary that can be run stand-alone for some
>> time, because that's necessary in ordered to be able to keep a somewhat
>> small initramfs, with udev but without all the trappings of a
>> full-fledged systemd.
>
> If you use dracut to generate your initramfs, you can get a full-fledged
> systemd inside it, so you can use the systemd in the initramfs to start
> the systemd in the real filesystem. I use it like that.

I had seen hints to that effect, but as I was still running openrc it
didn't have any immediate bearing on my dracut and initr* config, beyond
USE=-systemd for dracut.

>> (This said as an openrc user at least for the time being... even
>> apparently one of the only people actually running the live-git
>> openrc-9999, or at least all the bugs filed on it seem to be mine.
>> I've suspected for some time that I'll eventually switch to systemd,
>> but was at least originally hoping to avoid it until it quits actively
>> blackholing nearly everything it comes across and had some reasonable
>> time to stabilize without gobbling something else up. But when that'll
>> be... who knows? And I'm getting an itch to try it one of these days,
>> or at least seriously read up on it with a view to _consider_ trying
>> it, tho if I do it'll likely still be against my better judgment, since
>> I don't see it really stabilizing any time soon and I had originally
>> planned to wait for that. So I guess I sort of fall in the middle in
>> this debate.)
>
> If you run OpenRC live, I think you'll be fine running systemd, even
> 209/210, which introduced so many changes I've been waiting to upgrade
> my systems.
>
> Come to the dark side. We have cookies.

=:^)

As you /might/ have deduced by now from the hints of the past-tense "was"
and "didn't" above, as well as the fact that I'm dredging up this
slightly (10 days) stale thread to reply to...

Yeah, I'm actually replying from a systemd booted system, now.

As I was privately predicting for myself, but didn't want to promise even
to myself[1], I was getting close to the tipping point and this thread,
plus a couple extra days off, provided the trigger.

Since my experience may be of help to others considering the switch, here
we go...

In addition to unfortunately not finding a proper good detail but not
losing sight of the forest for the trees admin-level-targeted systemd
guide, a problem I overcame by simply immersing myself in the idea, full
systemd for admins series, multiple hour-ish videos, hours of thinking
about it while doing other things, sleeping over what I was learning over
several days...

There were five specific problems, three of which I've now overcome, with
two less pressing ones to go:

1) Various fstab mount configuration issues (local-fs.target).

To my very pleasant surprise, systemd has a reasonably effective nested-
mount-dependency algorithm and can often figure out on its own which
fstab configured mount-points are nested under and thus depend on
others. This actually simplified things considerably as one of my custom
openrc services was basically a localmount that had to come after
localmount but before some other service, in ordered to properly handle
nested mountpoints, and systemd's nested-mount-dependency processing very
nicely solved most of that automatically, without my having to worry
about porting a custom service.

1a) Unfortunately, systemd's nested-mountpoint detection algorithm isn't
smart enough to deal with symlinked paths, at least not in all cases.
One of my mount-related problems was thus a bind-mount configured in
fstab, that wasn't happening because the origin, on a different non-root
filesystem, wasn't mounted yet, so the attempt to bind-mount it elsewhere
failed, and local-fs.target was failing as a result of that failure.

I solved that one by changing the origin/device path for that fstab entry
to point at the canonical path instead of the symlinked path.

1b) My fstab entry for tmpfs on /run had a size=2m option, plenty for
/run on openrc, but NOT for systemd, as by the time it got to the /run
remount it already had an 8-meg early-boot journal file there, so
obviously the remount to 2-meg was failing! Of course that was failing
local-fs.target as well.

That one was hard to track down as I didn't expect systemd to be putting
a journal there and thus couldn't imagine what the problem was, but once
tracked down, given that I'm running 16-gig RAM, upping that to something
a bit more reasonable for the new usage, size=512m, was an easy fix.

With those fixes, local-fs.target finally succeeded, and I could get
farther in the boot, most services of course having an automatic
dependence on local-fs.target.

2) Unit file .include deprecated, replaced by *.d/*.conf files.

2a) systemd-random-seed.service was failing, again due to a mount-point-
detection symlink issue -- my /var is on my rootfs, which is read-only
mounted for normal operation (only rw mounted for updates and config
changes), so /var/lib/ is a symlink to /home/var/lib, /home naturally
being rw mounted in normal operation, and systemd was trying to read the
random-seed off that symlinked /home location before it was mounted,
because all it knew to check for was /var, which being on / was of course
mounted.

This one I solved initially by creating a new systemd-random-seed.service
file in /etc, .including the default service in /usr/systemd/system/, but
with an added RequiresMountsFor= entry.

That worked, but later on I found a note in the TODO to document the fact
that .include was deprecated and to point people to *.d/*.conf files
instead. So I decided to try that. The problem with /that/ was that the
TODO reference (and various other *.conf references I found) weren't
fully fleshed out, and as a systemd newbie I had some difficulty trying
to figure out what they were talking about.

But after absorbing systemd for a few days I finally realized that not
yet fully documented allusions I was seeing were to (paths like, for this
service) /etc/systemd/system/systemd-random-seed.service.d/whatever.conf.
And after adding a *.conf file there with the appropriate

[Unit]
RequiresMountsFor=

... entry, I was able to delete my custom service file and return to
using the shipped service file, appending the appropriate *.conf line to
the shipped configuration that way.

2b) I wasn't satisfied with the default gpm unit file config either, but
by the time I got to that, I knew .includes were deprecated, and was in
the process of trying to figure out what the allusion to *.d/*.conf was
all about.

But once I figured that out for random-seed, I was quickly able to add
/etc/systemd/system/gpm.service.d/execstart.conf , with a customized
ExecStart= line. =:^)


3) Systemd network config.

This one took me awhile too. After the above and a few other minor
tweaks to general systemd/journald/logind configuration[2], the big
problem was that I was still networkless. I didn't want to install
network manager or whatever for a simple single-NIC static IP config, I'm
still on systemd-208 as I've not figured out the systemd/dracut thing
yet, so I don't have the 209+ networkd yet, and the various static-ip-
configuring network.service units I could find via google (plus one on a
systemd bug on bugs.g.o) all used ip, from iproute2, which I had never
installed as the old ifconfig and route from net-tools had to that point
done all I needed it to do.

IMO this one's likely the biggest current problem for the gentoo masses
who might contemplate a systemd switch, as my other problems are due to
my decidedly custom config or are reasonably covered on the wiki.g.o
systemd page, while there's very likely quite a few gentooers still on
ifconfig/net-tools and unfortunately there's very little network
discussion on the wiki page at all, let alone a nice step-by-step in the
gentoo documentation tradition, covering ifconfig/net-tools users among
others. =:^(

I'm actually quite disappointed that the gentoo/systemd folks don't have
some sort of virtual/network-manager package or the like setup, with the
various network.service and similar files for each one, and good coverage
on the wiki. Given all the noise around systemd, and the vital role I'd
guess a working network config plays for most gentoo boxes, you'd think
that would be well covered. Oh, well... At least it appears now to be a
quite temporary problem, to hopefully be solved with the networkd in
210/211/whatever-the-next-unmasked/unblocked happens to be.

Anyway, a quick emerge --pretend iproute2 said it wasn't going to pull
any other deps I didn't already have, and I decided a single additional
reasonably core package that I really probably should have installed
years ago was after all a pretty small price to pay, so I finally just
bit the bullet and installed it, and then setup the network@.service file
using the ip command from iproute2.

Not so painful after all, but it'll still be nice when I get 210 or 211
or whatever installed, and can hopefully use the new networkd solution
instead of the still unofficial custom hack I've installed to deal with
networking for the time being.

Which brings me up to current, with those three issues behind me.

The remaining two issues are:

4) Dracut and systemd

4a) Dracut's USE=systemd, enable or disable?

What does that do and given that the current systemd-less initr* I'm
running now works well booting regardless of whether init=systemd, init
(openrc), or bash (rescue shell), will whatever changes systemd in the
initr* brings be worth it, or not?

Of course since you mentioned that you're running with systemd in your
init, I'm hoping you can be of help on this one. =:^)

4b) The ongoing systemd-209+ file "musical chairs" and the havoc that
plays with dracut, thus the current >=systemd-209 blockage in dracut.

It'd sure be nice to have this resolved one way or another, particularly
so given that 209+ brings the hopefully more final networkd solution to
#3 above, as well. I was actually hoping that the multi-device btrfs
root kernel command-line rootflags issue was resolved so I could just
drop the initr* thing now and thus wouldn't have to worry about dracut
any more, but as I posted in the earlier post, unfortunately not.

5) A couple remaining minor service failures:

5a) ntp.service is still failing, for some reason.

While I was running ntp on the old mobo before I upgraded and thus
haven't ever really run this one without it, I've no reason to think this
machine's hardware is /bad/ at keeping time, so ntp isn't critical. But
it'll be nice to find whatever's going on there and fix it.

Based on an aside-comment I read when researching something else, I
already have one lead to check on this one -- the comment said systemd's
privatetmp option doesn't play well with /var/tmp being a symlink to /
tmp, and that's what I have here (with /tmp in turn being tmpfs), and I
think the ntp unit might use privatetmp, so... But I've yet to confirm,
and then I'll have to look for alternatives, so...

5b) On openrc, I have a custom user-bootclean service that cleans up a
few stray user files at boot, that systemd created an attempted port for,
except it's failing. I've not investigated that one at all yet, but just
as I write this it occurs to me that the problem might be yet another
symlinked path mount-dependency issue, so I now have one lead to
investigate on that one, too.

5c) bind/named (not failing but reduced functionality)

There's a gentoo bug about this as well. Openrc's named/bind initscript
has an optional chroot feature. Systemd has both a legacy chroot support
feature and a newer kernel-namespaces-based chroot replacement, but the
shipped named.service file doesn't make use of either one. If nobody
else gets to that bug before I get comfortable enough with systemd to try
to address it myself, I'll try to come up with a systemd-style solution,
potentially using namespaces, to parallel the gentoo/openrc named
initscript's optional chroot feature.

https://bugs.gentoo.org/show_bug.cgi?id=496198


Other than that... a couple positives already. =:^)

1) As mentioned in problem #1 above, systemd has actually surprisingly
good nested mountpoint dependency resolution. My setup is complicated
enough I really appreciate that, and now that I'm "dual-booting" openrc
and systemd <g>, I definitely miss that feature on openrc! =:^0

2) Speed, WOW!

I've had openrc's parallel boot option enabled for years, but systemd
really does run circles around it! I'd guess a lot of that's due to the
socket-based activation along with the mount dependency resolution such
that services that are only dependent on each other due to that can still
start in parallel, and particularly with socket-based dependencies, the
otherwise depending service can often actually fully initialize if it
gets to that point before the depended service is up, as long as the
socket is already active and buffering any data the depending service
might try to write to it.

The fact that it's all C based and mostly declarative, with near-zero
scripting and far /far/ less forking, thus short-circuiting a good
portion of what even openrc's semi-C-based solution does, certainly helps
as well.

And of course the fact that I'm on SSD, where all those services starting
in parallel doesn't trigger a seek-storm on the spinning rust that's not
there to have a seek-storm on, doesn't hurt! (Systemd has a read-ahead
service to counteract part of the parallel-init seek-storm issue, but
spinning rust will still be a bit of a bottleneck. I had switched to SSD
and my boot times improved, but in hindsight on openrc, only to a point,
because the service-dependency on openrc then became the bottleneck.
With systemd's socket-based-activation and the like killing /that/
bottleneck too, the speed of SSD **REALLY** shows it's stuff! I had no
/idea/!)

I have a feeling that once I get settled in, I'll be doing far more full
shutdowns, since boottimes really /are/ trivial on systemd. And looking
to the future, if logind ends up doing for kde5 login times what systemd
did for boot times... WOW! The idea of shutdown and startup cycles
being so trivial one might as well just shut down when going to microwave
a snack, since bootup and session startup's literally mostly the time
waiting for the human to type in the password...


---
Notes:

[1] Predicting but not promising, even to myself: I've discovered that I
work best without deadlines or too specific goals, which only tend to
demotivate and drive procrastination. Left to itself, at some point the
drive to find some stable conclusion normally takes over, at which point
it's generally a done deal, concluded far more rapidly and with more
finality than it would be, had I tried to artificially drive it myself,
or let others try to drive it for me.

[2] General configuration changes, well enough documented that I didn't
have much problem with them:

(a)/etc/systemd/system/default.target symlink pointing at multi-user,
since I login at the CLI and have XSESSION set so I can startx, so systemd
would quick complaining about the missing *DM service.

(b) /etc/systemd/journald.conf: Storage=volatile at least until I figure
out whether I want to keep syslog-ng or not, appropriate RuntimeMaxUse
and RuntimeKeepFree values, ForwardToConsole=yes and TTYPath=/dev/tty12,
so I get the traditional TTY12 console log.

(c) Setting NAutoVTs and ReserveVT= in /etc/systemd/logind.conf

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd Was: Please get me straight about sysvinit vs. systemd [...] [ In reply to ]
On 03/13/2014 09:45 AM, Duncan wrote:

[trim /]

> Yeah, I'm actually replying from a systemd booted system, now.

Fabulous report!

[trim /]

> 2) Speed, WOW!

Ok, now I'm going to have to try this, too.

Thank you!

Phil
Re: Systemd Was: Please get me straight about sysvinit vs. systemd [...] [ In reply to ]
On Thu, Mar 13, 2014 at 9:45 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> 3) Systemd network config.
>
> This one took me awhile too. After the above and a few other minor
> tweaks to general systemd/journald/logind configuration[2], the big
> problem was that I was still networkless.

I ran into this issue a while ago. I ended up enabling dhcpcd so that
it was launched by my target. Since I just wanted dhcp that worked.

There is probably a better solution, and perhaps more recent versions
of systemd addresses this issue (I set it up a good year ago I think).

More recently I had issues with an NFS share needed by a service and
struggling with how to express that dependency (since the unit for the
mount is created by a generator). For whatever reason it was also
trying to mount the share before running dhcp (even though the share
was tagged with _netdev, and obviously nfs will depend on the network
in general). I'm sure that is all related to my network issues in
general.

So, overall I'd say it is a bit rough around the edges, but quite
capable. I really liked how systemd handles fileshares, at least in
principle.

Rich
Re: Systemd Was: Please get me straight about sysvinit vs. systemd [...] [ In reply to ]
Duncan posted on Thu, 13 Mar 2014 13:45:14 +0000 as excerpted:

> 2b) I wasn't satisfied with the default gpm unit file config either, but
> by the time I got to that, I knew .includes were deprecated, and was in
> the process of trying to figure out what the allusion to *.d/*.conf was
> all about.
>
> But once I figured that out for random-seed, I was quickly able to add
> /etc/systemd/system/gpm.service.d/execstart.conf , with a customized
> ExecStart= line. =:^)

Turned out that didn't work after all. I thought the modified ExecStart
line would replace the previous one and thought it was working but I
guess I hadn't tried a console mouse yet, as that didn't work. I was
getting an error due to the two execstart lines where only one was
allowed.

So I ended up dup-ing the entire gpm.service file from /usr to /etc and
editing the execstart line as necessary.

> 5) A couple remaining minor service failures:
>
> 5a) ntp.service is still failing, for some reason.

> Based on an aside-comment I read when researching something else, I
> already have one lead to check on this one -- the comment said systemd's
> privatetmp option doesn't play well with /var/tmp being a symlink to /
> tmp, and that's what I have here (with /tmp in turn being tmpfs), and I
> think the ntp unit might use privatetmp, so... But I've yet to confirm,
> and then I'll have to look for alternatives, so...

That was it. PrivateTmp doesn't like a symlinked /var/tmp.

So I reset /var/tmp to a directory and setup an fstab entry to bind-mount
the /tmp tmpfs to /var/tmp as well. Seems to be working at first blush,
tho I haven't rebooted enough times yet to be sure there's no possible
service dependency issues with it, but so far, it's working. =:^)


> 5b) On openrc, I have a custom user-bootclean service that cleans up a
> few stray user files at boot, that systemd created an attempted port
> for, except it's failing. I've not investigated that one at all yet,
> but just as I write this it occurs to me that the problem might be yet
> another symlinked path mount-dependency issue, so I now have one lead to
> investigate on that one, too.

I haven't fully figured out why it's failing yet, but I think I can
manage that with a tmpfiles.d entry. And openrc supports tmpfiles.d as
well, so if I setup such an entry, openrc will hopefully handle that too,
and I can simply delete the /etc/local.d/*.start file that's failing.
That should be the clean modern solution in any case.

But I'm too sleepy now to think straight, so I'll have to sleep on
actually setting up and testing that one, lest I make a stupid mistake
and end up deleting the entire user's homedir!

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd Was: Please get me straight about sysvinit vs. systemd [...] [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/13/2014 03:20 PM, Duncan wrote:
> Duncan posted on Thu, 13 Mar 2014 13:45:14 +0000 as excerpted:
>
>> 2b) I wasn't satisfied with the default gpm unit file config either, but by the time I got to
>> that, I knew .includes were deprecated, and was in the process of trying to figure out what
>> the allusion to *.d/*.conf was all about.
>>
>> But once I figured that out for random-seed, I was quickly able to add
>> /etc/systemd/system/gpm.service.d/execstart.conf , with a customized ExecStart= line. =:^)
>
> Turned out that didn't work after all. I thought the modified ExecStart line would replace the
> previous one and thought it was working but I guess I hadn't tried a console mouse yet, as that
> didn't work. I was getting an error due to the two execstart lines where only one was
> allowed.
>
> So I ended up dup-ing the entire gpm.service file from /usr to /etc and editing the execstart
> line as necessary.

The correct way to do this is to create the file with contents like the following:

[Service]
ExecStart=
ExecStart=/usr/bin/gpm --my --options

The first ExecStart= line is a special case that tells systemd to unset the ExecStart parameter
completely so you can override it (instead of adding additional commands to run on startup).

- --
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTI/FyAAoJELHSF2kinlg44e8QAJkLXFOr0Dm5KE2fSw01+2JA
bSY56oZXoRV1d3dlzM7bfCMDRFTPW4MDWv1+qb6AdH/7+tcb8H/pk2pEgbJtjBw3
4oK9uIJOcwYXPlOZU8bVcwQ70cXItlET7waok0h1u0o7WDDabVJi+HFKWMhpHEH1
JYM9HnTMx8qujfBxvbOHuOenBRZXpejqrVqo82UPmf2Ot8I+YS5+ACb7Qm3znqFu
02fbkb9nfbonqL3YDliqlv3MavCuHBQUCKGFtYtWoEjr0V+GGv+pqHgPzefWcT0F
EBMeCOA506bJHgHs+zam010BXlXt+9okr22S5Scvcv/vUIFu7X1AhCcgEfMkTY0o
VdFFq0uPncyH51Xnc2GD9+NzYY1ddJu6xA+ysLzJz2JMe0EXRJf67dl2Zp3i/Y48
FegAevX34VzilkkBeC4ArP1yPoOXQccy/+lyQT3Q1e5Xwp/65jX0/R8RxIycTIZz
vYEc1o84X56ZfvnfzPAr5m0dF9Rg6CUEx0RfDS9mI2VsnjsWOS6otvmfgfGqvBsq
aFnrakCh6ZEcvtd1j76yHy3U1rbhbC0kjuaEkjICvCWmNzeQJabqRN1w9Kam5dvF
Dureh8SSP3JtxpY7f945yBlmNqewYCjHrjg0BSTlaTXMMLIfrBf11QuEp7vS4s1l
7mNWKZP2+HqooMu7WCsJ
=ZfwZ
-----END PGP SIGNATURE-----