Mailing List Archive

Systemd migration: opinion and questions
Hi list,

(Normally I ask these types of questions to gentoo-user, but thought I'd try
here for once, especially given the ease with which systemd flamewars erupt on
gentoo-user, compared to the much more civil discussions I've seen here.)

So, at night on Saturday, 14th February, I decided, on a whim, to install
systemd on my Uni laptop. Everything went well, save for some remaining
questions (see below). The Gentoo systemd wiki entry was very helpful.

[. It's really strange how that works: I just got the sudden urge to try systemd
out, for no real reason at all, and couldn't stop myself. Weird... ]

Heh... I was originally going to send this *before* migrating my desktop, but
it turns out I liked systemd enough to want to switch my desktop to it before I
managed to finish writing this. And holy crap, Duncan was right: systemd is
*so* *fast* on an SSD, it's just not funny. It takes *3 seconds* after the
kernel boots for me to get a login screen (for a total of about 5-6 seconds
after the boot loader). I barely get a chance to see systemd's messages, they
just start zipping by, and then *wham*: login screen. I am *amazed*!

*ahem*

Anyway, I thought it might be interesting to report other things I did and what
I like about systemd so far. Due to the length and additional context I'll
split the rest of this email into sections for better readability. I apologise
for the length; in my defence, this is a fundamental system change, and I
wanted to offer enough context for everything.

== Additional changes on the laptop ==

I replaced laptop-mode-tools with tlp from the overlay of the same name (not
directly available via layman, but there's a bug on b.g.o that links to the
GitHub repo, which has installation instructions). Since I controlled the
backlight with lmt, and tlp doesn't support that, I had to find a replacement
for that feature. I ultimately decided on power-backlight, which I found via
the Arch Linux wiki. I wrote an ebuild and put it in my personal overlay
(http://sourceforge.net/p/mjolietoverlay/code/ci/master/tree/app-laptop/power-backlight/),
and so far everything is working fine.

This might seem like a pointless change, but Arch switched to it, too, and
doesn't even package lmt anymore (albeit only since late 2014), so it seems
like the way forward.

== Additional changes on both systems ==

I uninstalled acpid, since the Gentoo Wiki page on systemd mentions that it
might not be necessary (see the question below). Being able to do this was
another reason I switched to tlp, since it doesn't depend on acpid. So far I
have yet to notice anything amiss.

I also uninstalled tmpwatch, since systemd has a built-in service for doing the
same thing (systemd-tmpfiles).

== Things I have *not* gotten rid of (yet) ==

Fcron is still around, mainly because packages might rely on it being there
(e.g., man-db and mlocate install files there), but also because I haven't
researched systemd timers yet.

I plan on uninstalling syslog-ng, but haven't done so yet. I simply feel
better waiting a bit, even though I don't run it anymore. Man, I feel silly
after typing that...

== Network migration on my desktop ==

My desktop has a somewhat more complicated network setup than the laptop (which
uses NetworkManager). The wan0 network device is no problem, but I also have a
bridge with one physical device connected to it, and one dummy device.

Thus, I had to migrate my netifrc based configuration. Due to word of mouth, I
decided on netctl. I originally ignored systemd-networkd because I keep
hearing that it's only for simple networks, but after looking at the man page
it appears that it can do (almost) everything that I needed it to, although I'm
not sure about dummy device support. I need that for MATLAB, which stupidly
requires the presence of an interface named eth0, whose MAC address it verifies
during its licence check. This became a problem after renaming my network
devices following the news entry 2013-03-29-udev-upgrade ("Upgrading udev to
version >=200").

What surprised me was that netifrc doesn't seem to integrate dummy devices
properly, i.e., it doesn't seem to be possible to rename them in
/etc/conf.d/net directly. I implemented that via an appropriate call to ip in
an /etc/local.d file, and had "net.eth0" depend on "local". So /etc/conf.d/net
looked like this:

config_eth0="172.16.1.1/24"
mac_eth0="00:18:f3:97:17:72"
rc_net_eth0_provide="!net"
rc_net_eth0_need="local"

And the local.d script executed

ip link set dev dummy0 name eth0

In comparison, the netctl configuration has everything in one place:

# cat /etc/netctl/dummy
Description='A dummy interface (for MATLAB)'
Interface=eth0
Connection=dummy
IP=static
Address=172.16.1.1
IPCustom=( 'link set dev eth0 address 00:18:f3:97:17:72' )

It also takes care of loading the dummy kernel module, so I don't don't need an
/etc/modules-load.d/ entry, whereas with netifrc/OpenRC I needed to load it
manually via /etc/conf./modules (though maybe that has changed?).

== Stuff I liked ==

If you want to skip this, the questions are in the next section.

=== Pleasant surprises ===

Some surprising things worked out of the box, for example all /etc/local.d/
files are dynamically converted to units at run-time and executed. I hardly
had to migrate anything. The one exception was chrony, where I had to override
the service file to add extra command line flags that don't have corresponding
configuration file entries.

On my desktop I had to add network dependencies to some socket units so that
they wouldn't start before the network they're supposed to listen on is fully
configured (see the question below). Otherwise, it just had a few more things
that needed doing, but that is the nature of running services on it (dovecot,
postfix, and samba).

=== The Journal ===

I like the journal. No, sorry, I *really* *really* like the journal. I like
the filters you can apply (e.g., limiting the output by unit via the -u flag),
the priority system, the automatic use of ACLs so that you can see your own
user's logs. I also think that it's a nice detail that "systemctl status -u"
will show the last lines of log output. This actually helped me at work :-) .

=== Socket activation ===

I especially learned to like it after reading the early "systemd for admins"
articles, which made the implications of the design more clear. I definitely
like the ability to have a service start on-demand instead of spending 99.99%
of its time idle. Finally, I also like the simplicity of the C API from
systemd/sd-daemon.h.

=== Miscellanea ===

Some things work better, e.g., closing the laptop's lid suspends the laptop
automatically without me having to configure anything. My desktop is *almost*
there, see the questions below.

Systemd can restart itself, e.g., for upgrades. Nice!

Systemd-analyze is really nice.

Systemd exposes various system features that require extra work to set up
otherwise. The example that made me think of this is "systemctl kexec", but
from reading Leannart Poettering's blog series, this is one of the goals of
systemd. I can definitely sympathise with that.

== Questions ===

=== ACPID: needed or not? ===

Does acpid provide anything that systemd does not, and if so, what kind of
"conflicts" might I see? The Gentoo Wiki page says that acpid is likely not
needed.

FWIW, I already unmerged it and have not noticed any missing functionality,
even after over a week of regular usage. So I'm tending towards "no, not
needed".

=== Timers ===

Can a systemd timer depend on a mount point such that it waits until the mount
point exists before running? Or will it fail after a timeout? I want to
research this myself, but haven't gotten around to it yet.

The problem I have is that my external HDD does not come up properly on cold
boot, so that I have to unplug it and plug it back in in order for the kernel
to fully initialise it and for it to mount, which is problematic for my backup
fcron jobs, since they have &bootrun set. This means that the backup script
will fail, unless I'm a) fast enough at re-plugging the HDD, and b) fast enough
at logging in (so that my automounter mounts the HDD. I will then have to
manually re-run it (i.e., fcrondyn -x "run <ID>") or wait for the next time
it's supposed to run.

Naturally, I would like a more robust system than that, and hope that systemd
timers can make my life easier here.

=== User units ===

I would like to convert some programs I start in .xprofile to units that are
started by my users's systemd instance. I started off with mpd, but it doesn't
start automatically ("systemctl --user start mpd" works fine, though), even
though it's enabled:

% systemctl --user status mpd
● mpd.service - Music Player Daemon
Loaded: loaded (/usr/lib64/systemd/system/mpd.service; enabled)
Active: active (running) since Di 2015-02-24 19:39:46 CET; 1h 6min ago
Main PID: 1091 (mpd)
CGroup: /user.slice/user-1000.slice/user@1000.service/mpd.service
└─1091 /usr/bin/mpd --no-daemon

Feb 24 19:39:46 marcec systemd[384]: Started Music Player Daemon.
[...]

Also:

% tree .config/systemd/
.config/systemd/
└── user
├── mpd.service -> /usr/lib64/systemd/system/mpd.service
└── multi-user.target.wants
└── mpd.service -> /home/marcec/.config/systemd/user/mpd.service

Is the symlink the problem? Do I have to create an actual file? Is the
target.wants wrong?

=== Suspend on the desktop ===

Like I mentioned above, my Desktop can *almost* suspend reliably, after trying
it out once every year or two (it's over 8 years old). Mostly it would just
not wake back up. The latest status (before systemd) was that the kernel
crashed after waking up (but I think that was a known bug that was fixed in the
meantime).

Now with systemd, it wakes back up properly, but one of the soundcard drivers
(ice1724) is apparently unreliable, so that sound stops working after wakeup.
Plus, access to the soundcard was blocked by rtkit. I believe what one would
normally do is unload the module before suspending, but the only way I could
find to tell systemd to do that requires creating a file in /usr (as done here,
for example: http://forums.fedoraforum.org/showthread.php?t=294065). Is there
a better way?

=== Dovecot and socket activation ===

Dovecot gave me problems with socket activation, I kept getting errors like the
following:

Feb 20 22:58:19 marcec dovecot[6500]: master: Panic: io_add(0x1) called
twice fd=4, callback=0x40a970 -> 0x40a970 Feb 20 22:58:19 marcec
dovecot[6500]: master: Error: Raw
backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x6accf) [0x7f42b9300ccf]
-> /usr/lib64/dovecot/libdovecot.so.0(i_syslog_fatal_handler

I don't know who's at fault here, systemd or dovecot, but I had to simplify the
configuration a bit:

diff --git a/dovecot/conf.d/10-master.conf b/dovecot/conf.d/10-master.conf
index ddbde28..c0ca867 100644
--- a/dovecot/conf.d/10-master.conf
+++ b/dovecot/conf.d/10-master.conf
@@ -16,11 +16,11 @@

service imap-login {
inet_listener imap {
- address = ::1, 127.0.0.1, 172.16.0.1
+ address = ::1, 172.16.0.1
port = 10087
}
inet_listener imaps {
- address = ::1, 127.0.0.1, 172.16.0.1
+ address = ::1, 172.16.0.1
port = 10887
#ssl = yes
}
diff --git a/systemd/system/dovecot.socket.d/sockets.conf
b/systemd/system/dovecot.socket.d/sockets.conf index bcc29c6..368d461 100644
--- a/systemd/system/dovecot.socket.d/sockets.conf
+++ b/systemd/system/dovecot.socket.d/sockets.conf
@@ -1,8 +1,6 @@
[Socket]
ListenStream=
-ListenStream=127.0.0.1:10087
ListenStream=172.16.0.1:10087
ListenStream=[::1]:10087
-ListenStream=127.0.0.1:10887
ListenStream=172.16.0.1:10887
ListenStream=[::1]:10887

Note that it also worked if I removed 127.0.0.1 from imap, and 172.16.0.1 from
imaps (or, IIRC, vice versa). I'm fine with the configuration being like this,
but I'm nonetheless baffled by the necessity. I couldn't even find a bug
report or ML post.

=== Depending on a specific network interface ===

Some socket units failed to start at first, due to "resource" errors. So I
made them depend on netctl@bridge via *.d/requires.conf files like so:

[Unit]
Requires=netctl@bridge.service
After=netctl@bridge.service

That fixed the errors, but is it the correct way to depend on that interface
(ignoring the fact that I could have put symlinks at the right place instead)?

=== dmesg and syslog ===

On the laptop I didn't immediately stop running syslog-ng. Following a thread
on gentoo-user about the journal complaining about messages not being forwarded
to syslog-ng, I got curious and checked the laptop today.

What I did was run

diff -U8 <(zcat /var/log/messages-*) <(journalctl)

and look for differences in the time frame where both were running (note
that /var/log/messages is empty, so I only checked the rotated logs).

First of all, while I clearly remember seeing similar messages about "missing
messages", I couldn't find any in either the journal or syslogs output. Maybe
I mixed them up with "Forward time jump detected" messages from chrony? But
those are too rare, unless I noticed one right after resuming from suspend and
got mixed up then? I think that I would like confirmation that those messages
are actually logged properly, and where.

However, I did find something unexpected. I haven't looked deeply yet, but I
noticed some dmesg entries not ending up in the journal (i.e., a bunch of "-"
lines in-between the "+" lines in the diff output). Did I do something wrong, or
is this known behaviour when running the journal and syslog-ng simultaneously?
I'll look more closely tomorrow, but wondered if anybody here noticed anything
similar.

== The End ==

*phew*

Of course, some of the complications will go away once I've gotten around to
turning my desktop back into a pure desktop and not the desktop/server/router
hybrid it is now, but that will have to wait.

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/24/2015 03:15 PM, Marc Joliet wrote:
> === User units ===
>
> I would like to convert some programs I start in .xprofile to units
> that are started by my users's systemd instance. I started off
> with mpd, but it doesn't start automatically ("systemctl --user
> start mpd" works fine, though), even though it's enabled:
>
> % systemctl --user status mpd ● mpd.service - Music Player Daemon
> Loaded: loaded (/usr/lib64/systemd/system/mpd.service; enabled)
> Active: active (running) since Di 2015-02-24 19:39:46 CET; 1h 6min
> ago Main PID: 1091 (mpd) CGroup:
> /user.slice/user-1000.slice/user@1000.service/mpd.service └─1091
> /usr/bin/mpd --no-daemon
>
> Feb 24 19:39:46 marcec systemd[384]: Started Music Player Daemon.
> [...]
>
> Also:
>
> % tree .config/systemd/ .config/systemd/ └── user ├── mpd.service
> -> /usr/lib64/systemd/system/mpd.service └──
> multi-user.target.wants └── mpd.service ->
> /home/marcec/.config/systemd/user/mpd.service
>
> Is the symlink the problem? Do I have to create an actual file?
> Is the target.wants wrong?

I ran into this problem as well. It is indeed the symlink! There is an
RFE filed for this:

https://bugzilla.redhat.com/show_bug.cgi?id=1014311

You don't have to create the entire file either though, it turns out.
It is possible to inherit a unit in a unit, so you could create a unit
that just inherits the one from /usr/lib. I've never tried this with a
user unit, but I suspect it would work. Read the section "Example 2.
Overriding vendor settings" at the end of this page:

http://www.freedesktop.org/software/systemd/man/systemd.unit.html

- --
R
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJU7OIPAAoJEHhEzLg73SRiSCkP/jTv3EWAUTKo5xWFSknMUdUc
EfKIc5i4dVX47IKpMF2jKH3201gZ6XJD9GwJW4MFcrM6Ay4q+JiowActdwihptWW
keTkJsZ3CrKm6ulD6rkugA443cjKKdeVPdRSCz9FNEH5T5PRqMTRLwnRendDI/9+
CKD1l6WTLGaHTpsmgVrKC7HBaZd4GLzGC7izrHcVuzUHUu3vJzxR/xMfjYALeU5J
EPdyHsgydN42IMbox+mD1IlSfQXWIV1YxyOgTSNoLUlpApkDzZpFoF7SauNMHW93
DwUwSqFxfvft28gVEmykc31oILzI41aVgNd+pZLCjzl9WDEWpxsqIpXqS2jiX2iD
NheSTuloR9zbz1Cs0tsY+T9QkQv/PVuGcDfymuHbaZsUdm3HoIkPTF1DeYN2zpIP
zpsJFBKQ05lE/sXpBR229QUvQHBasmczq99TvGKFxkz4JBpQybPbEbqZ862eveg3
H+Zi8/y3vvT2ISYNIvnrLDyAgf/6nLRue82I7eWvBcvuaHHeUGvTXSnN32QJBtY3
Zk5ZlpW1t9YT84jv6hXThS948N8NKJJrxa5dFiocGuTzhBdcLtlNheizPDPGprUM
a3inD7SQxw/9U93VMyRPuA9H5gA70ssQRXCvrRHQoV7U+g5qucWV53cXo3bZpBmY
MR5hDLvnQVIr/RHzmQ1E
=hUb2
-----END PGP SIGNATURE-----
Re: Systemd migration: opinion and questions [ In reply to ]
Lots of stuff here. I'm only replying where I think I can add value.

On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@gmx.de> wrote:
>
> == Things I have *not* gotten rid of (yet) ==
>
> Fcron is still around, mainly because packages might rely on it being there
> (e.g., man-db and mlocate install files there), but also because I haven't
> researched systemd timers yet.

My fault. :) I have sys-process/systemd-cron in my rich0 overlay but
I've been too lazy to get it into the main tree (I need to check to
see what units if any I had to tweak from what gets installed and
merge those back into the package). This is a set of timer/service
units that will run all the /etc/cron.*/ jobs. They'll run in series
and not in parallel as they would if you turned them into individual
timer scripts, but it is a great way to just integrate the scripts
packages already are providing. It also has a crontab generator, but
I don't think it works with fcrontab so I have not tested this.
Generators are just one of those little things I love about systemd.

>
> I plan on uninstalling syslog-ng, but haven't done so yet. I simply feel
> better waiting a bit, even though I don't run it anymore. Man, I feel silly
> after typing that...

Yeah, took me a while to uninstall it, but I don't find much value in
running both.

>
> == Network migration on my desktop ==
>
> My desktop has a somewhat more complicated network setup than the laptop (which
> uses NetworkManager). The wan0 network device is no problem, but I also have a
> bridge with one physical device connected to it, and one dummy device. ...
> I originally ignored systemd-networkd because I keep
> hearing that it's only for simple networks, but after looking at the man page
> it appears that it can do (almost) everything that I needed it to, although I'm
> not sure about dummy device support.

Here is what I'm doing for a bridge - I can't vouch for dummy devices
but I wouldn't be surprised if it works (remember, these are the guys
doing udev too):
cd /etc/systemd/network/

cat brkvm.netdev
[NetDev]
Name=brkvm
Kind=bridge


cat brkvm.network
[Match]
Name=brkvm

[Network]
DNS=192.168.0.5
Address=192.168.0.5/24
Gateway=192.168.0.21


cat eth-bridge.network
[Match]
Name=e*

[Network]
Bridge=brkvm


If you're using persistent names you might need to tweak the Name in
the last one slightly so that it matches whatever you're using.

>
> === Timers ===
>
> Can a systemd timer depend on a mount point such that it waits until the mount
> point exists before running? Or will it fail after a timeout? I want to
> research this myself, but haven't gotten around to it yet.

So, timer units are units, and units can have dependencies, and mounts
can be dependencies since mounts are units. However, if you set the
dependency on the timer itself, then the timer won't start running
until the mount exists. You probably want the depencency to be on the
service started by the timer (so the timer is watching the clock, but
the service won't start without the mount). If you set a
Requires=foo.mount and After=foo.mount, then the service shouldn't run
unless foo.mount is available. I suspect systemd will attempt to
mount the filesystem when it runs the service, and you'll get units in
the failed state if that doesn't work.

However, I haven't tested any of this. I suspect it wouldn't take
much to work this out. I have a mount dependency in one of my
services. Just look at the mount units in /run/systemd/generator for
the name of the mount unit systemd is creating from fstab.

> === Depending on a specific network interface ===
>
> Some socket units failed to start at first, due to "resource" errors. So I
> made them depend on netctl@bridge via *.d/requires.conf files like so:
>
> [Unit]
> Requires=netctl@bridge.service
> After=netctl@bridge.service
>
> That fixed the errors, but is it the correct way to depend on that interface
> (ignoring the fact that I could have put symlinks at the right place instead)?

Following my networkd scripts above, I suspect that something like
this /might/ work but I haven't tested it:
Requires=sys-subsystem-net-devices-brkvm.device
After=sys-subsystem-net-devices-brkvm.device

Again, another example of the udev integration - you get device units
for devices. That also lets you do things like start a service when a
device is activated, have dependencies, etc. You should look at the
list of units sometime - you can also do things like run a service
when a file is modified (no need to have a helper program watching it
with fnotify, etc).

Also, I think Requires basically implies Wants as well, so if that
network device doesn't exist and the service tries to start, systemd
might try to create that device, if possible.

Now, the device existing does not necessarily mean that it has an IP,
working DNS, etc. I don't know if there is any way to have a
dependency on a specific configured network. That might be a great
question to ask the systemd devs.

--
Rich
Re: Systemd migration: opinion and questions [ In reply to ]
On Tue, 24 Feb 2015 21:15:45 +0100
Marc Joliet <marcec@gmx.de> wrote:

>
> I liked systemd enough to want to switch my desktop to it before I
> managed to finish writing this. And holy crap, Duncan was right: systemd is
> *so* *fast* on an SSD, it's just not funny. It takes *3 seconds* after the
> kernel boots for me to get a login screen
>

It requires about just as long (3 secs) on my desktop machine *without*
systemd by just using my custom bash init script (no sysvinit or openrc either).
Furthermore, I use no SSD.

But there's also no login required so it's probably even faster -- and
there's also no long list of permanently running daemons as well.
(After a boot, ps ax shows a very sparse process list.)

For me, systemd is totally unnecessary, excessively burdensome, and
highly obfuscating to a complete control and understanding of my
system.

To each his own. If you want/need it, fine. But don't expect me,
or everyone else, to slavishly follow.

As has been more or less officially stated, systemd is an attempt to
provide a uniform and monolithic kernel-user space interface FOR THE
BENEFIT OF POTENTIAL (COMMERCIAL) SOFTWARE DEVELOPERS. RedHat wants
Linux to be just like MS Windows so that *they* (RedHat) can be just
like MS Windows.

Let RedHat leave (fork) the Linux community and go their own separate
way.

Maybe it's time to begin shifting a lot more attention to Gentoo/FreeBSD.

Frank Peters
Re: Systemd migration: opinion and questions [ In reply to ]
Am Tue, 24 Feb 2015 15:41:51 -0500
schrieb Randy Barlow <randy@electronsweatshop.com>:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 02/24/2015 03:15 PM, Marc Joliet wrote:
> > === User units ===
> >
> > I would like to convert some programs I start in .xprofile to units
> > that are started by my users's systemd instance. I started off
> > with mpd, but it doesn't start automatically ("systemctl --user
> > start mpd" works fine, though), even though it's enabled:
> >
> > % systemctl --user status mpd ● mpd.service - Music Player Daemon
> > Loaded: loaded (/usr/lib64/systemd/system/mpd.service; enabled)
> > Active: active (running) since Di 2015-02-24 19:39:46 CET; 1h 6min
> > ago Main PID: 1091 (mpd) CGroup:
> > /user.slice/user-1000.slice/user@1000.service/mpd.service └─1091
> > /usr/bin/mpd --no-daemon
> >
> > Feb 24 19:39:46 marcec systemd[384]: Started Music Player Daemon.
> > [...]
> >
> > Also:
> >
> > % tree .config/systemd/ .config/systemd/ └── user ├── mpd.service
> > -> /usr/lib64/systemd/system/mpd.service └──
> > multi-user.target.wants └── mpd.service ->
> > /home/marcec/.config/systemd/user/mpd.service
> >
> > Is the symlink the problem? Do I have to create an actual file?
> > Is the target.wants wrong?
>
> I ran into this problem as well. It is indeed the symlink! There is an
> RFE filed for this:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1014311

Hmm, I don't think that's related. It's not that the service isn't enabled, it
doesn't start automatically.

However, after looking at the Arch wiki again
(https://wiki.archlinux.org/index.php/Systemd/User#How_it_works), I found the
key bit of info I needed:

"When systemd user instance starts, it brings up the target default.target."

So I need to add a symlink to *default*.target.wants/, then MPD starts when I
log in! So I ended up with this trivial user unit, which replaced the symlink
I had their experimentally:

% cat .config/systemd/user/mpd.service
.include /usr/lib/systemd/system/mpd.service

[Install]
WantedBy=default.target

What was more difficult was getting it to depend on the right network target
properly. The same "Requires" line that worked for samba and cups fails with
"failed to add dependency" type errors:

Feb 24 21:20:01 marcec systemd[540]: Cannot add dependency job for unit
mpd.service, ignoring: Unit netctl@dhcp-wan0.service failed to load: No such file or directory.

So apparently user units can't "Require" system units? However, adding a
symlink from the service unit in /etc to .../mpd.service.requires/ solved that
problem, but yielded a new one:

Feb 24 23:26:49 marcec network[2200]: netctl needs root privileges

Oh, well... Maybe it would work with systemd-networkd?

Regardless,, I solved that by changing my MPD configuration to bind to
"any" (equivalent to 0.0.0.0), since it was trying to bind to all interfaces
individually anyway. So I dropped the Requires, and everything is fine now.

But now I need to see what else might profit from being started this way :) .

> You don't have to create the entire file either though, it turns out.
> It is possible to inherit a unit in a unit, so you could create a unit
> that just inherits the one from /usr/lib. I've never tried this with a
> user unit, but I suspect it would work. Read the section "Example 2.
> Overriding vendor settings" at the end of this page:
>
> http://www.freedesktop.org/software/systemd/man/systemd.unit.html

Thanks, I do in fact know about that, but I can still use reminders now and
again :-) .

--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Am Tue, 24 Feb 2015 16:44:59 -0500
schrieb Rich Freeman <rich0@gentoo.org>:

> Lots of stuff here. I'm only replying where I think I can add value.
>
> On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@gmx.de> wrote:
> >
> > == Things I have *not* gotten rid of (yet) ==
> >
> > Fcron is still around, mainly because packages might rely on it being there
> > (e.g., man-db and mlocate install files there), but also because I haven't
> > researched systemd timers yet.
>
> My fault. :) I have sys-process/systemd-cron in my rich0 overlay but
> I've been too lazy to get it into the main tree (I need to check to
> see what units if any I had to tweak from what gets installed and
> merge those back into the package). This is a set of timer/service
> units that will run all the /etc/cron.*/ jobs. They'll run in series
> and not in parallel as they would if you turned them into individual
> timer scripts, but it is a great way to just integrate the scripts
> packages already are providing. It also has a crontab generator, but
> I don't think it works with fcrontab so I have not tested this.
> Generators are just one of those little things I love about systemd.

That sounds great! I'll be looking forward to this :) .

(Although, sadly, it does appear as if ebuilds getting stuck in overlays happens
much too often these days.)

> >
> > I plan on uninstalling syslog-ng, but haven't done so yet. I simply feel
> > better waiting a bit, even though I don't run it anymore. Man, I feel silly
> > after typing that...
>
> Yeah, took me a while to uninstall it, but I don't find much value in
> running both.

Precisely. It's more a "but what if...!" kind of worry.

> >
> > == Network migration on my desktop ==
> >
> > My desktop has a somewhat more complicated network setup than the laptop (which
> > uses NetworkManager). The wan0 network device is no problem, but I also have a
> > bridge with one physical device connected to it, and one dummy device. ...
> > I originally ignored systemd-networkd because I keep
> > hearing that it's only for simple networks, but after looking at the man page
> > it appears that it can do (almost) everything that I needed it to, although I'm
> > not sure about dummy device support.
>
> Here is what I'm doing for a bridge - I can't vouch for dummy devices
> but I wouldn't be surprised if it works (remember, these are the guys
> doing udev too):
> cd /etc/systemd/network/
>
> cat brkvm.netdev
> [NetDev]
> Name=brkvm
> Kind=bridge
>
>
> cat brkvm.network
> [Match]
> Name=brkvm
>
> [Network]
> DNS=192.168.0.5
> Address=192.168.0.5/24
> Gateway=192.168.0.21
>
>
> cat eth-bridge.network
> [Match]
> Name=e*
>
> [Network]
> Bridge=brkvm
>
>
> If you're using persistent names you might need to tweak the Name in
> the last one slightly so that it matches whatever you're using.

That looks nice, I like that the bridge device is distinct from the bridge
network, is distinct from other devices attached to it. This might help in
getting unit dependencies right. I'll try it out this weekend.

My primary concern with networkd would be that it is very new, and thus not as
well tested as other solutions. While netctl is also fairly new, and
I'm using that with no problems, it uses well-known DHCP clients, while it
appears that networkd implements its own.

But I'll give it a whirl anyway ;) .

> > === Timers ===
> >
> > Can a systemd timer depend on a mount point such that it waits until the mount
> > point exists before running? Or will it fail after a timeout? I want to
> > research this myself, but haven't gotten around to it yet.
>
> So, timer units are units, and units can have dependencies, and mounts
> can be dependencies since mounts are units. However, if you set the
> dependency on the timer itself, then the timer won't start running
> until the mount exists. You probably want the depencency to be on the
> service started by the timer (so the timer is watching the clock, but
> the service won't start without the mount).

Wait, so the timer won't start watching the clock until its dependencies are
met (i.e, the mount point appears)? Is that what you mean? Because that might
be more in line with what I want (though I'm not sure yet).

> If you set a
> Requires=foo.mount and After=foo.mount, then the service shouldn't run
> unless foo.mount is available. I suspect systemd will attempt to
> mount the filesystem when it runs the service, and you'll get units in
> the failed state if that doesn't work.
>
> However, I haven't tested any of this. I suspect it wouldn't take
> much to work this out. I have a mount dependency in one of my
> services. Just look at the mount units in /run/systemd/generator for
> the name of the mount unit systemd is creating from fstab.

Right, so IIUC, I would have a oneshot service that does the backup, and the timer
runs that, and of course the timer can depend on the mount point. And if the
mount point doesn't exist, then the service started by the timer will fail.

What I would prefer to have is a timer that only runs if *both* the time *and*
mount conditions are met. Skimming the man page, this does not seem possible.
I suppose it would be nice if timers learned "conditions" on which they should
wait in addition to the time condition, but maybe that's outside the scope of
systemd?

But my confidence that at the very least something better than what I have now
will be possible, I'll just have to take the time and read the man pages
properly.

> > === Depending on a specific network interface ===
> >
> > Some socket units failed to start at first, due to "resource" errors. So I
> > made them depend on netctl@bridge via *.d/requires.conf files like so:
> >
> > [Unit]
> > Requires=netctl@bridge.service
> > After=netctl@bridge.service
> >
> > That fixed the errors, but is it the correct way to depend on that interface
> > (ignoring the fact that I could have put symlinks at the right place instead)?
>
> Following my networkd scripts above, I suspect that something like
> this /might/ work but I haven't tested it:
> Requires=sys-subsystem-net-devices-brkvm.device
> After=sys-subsystem-net-devices-brkvm.device
>
> Again, another example of the udev integration - you get device units
> for devices. That also lets you do things like start a service when a
> device is activated, have dependencies, etc. You should look at the
> list of units sometime - you can also do things like run a service
> when a file is modified (no need to have a helper program watching it
> with fnotify, etc).

Right, I really do like that about systemd, too. It's much more thorough than
other systems (in several ways). The mount point units generated from fstab
are one example, but I also noticed that each script in /etc/local.d/ gets the
same treatment and is exposed as its own unit.

The more I use systemd, the more I realise why it made sense to merge the udev
project into systemd: they both play off each others capabilities quite
nicely (udev giving systemd extra information and events to act upon, and
systemd reacting to udev's events and doing the appropriate thing).

Diving into systemd on a whim really was the best way to see what all the fuss
was about, and so far it was worth it.

> Also, I think Requires basically implies Wants as well, so if that
> network device doesn't exist and the service tries to start, systemd
> might try to create that device, if possible.
>
> Now, the device existing does not necessarily mean that it has an IP,
> working DNS, etc. I don't know if there is any way to have a
> dependency on a specific configured network. That might be a great
> question to ask the systemd devs.

Sure, if nobody here gives an authoritative answer, I'll ask there. People:
you have until the weekend ;) .

Thanks to everyone for the responses so far!
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted:

> On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@gmx.de> wrote:
>>
>> == Things I have *not* gotten rid of (yet) ==
>>
>> I plan on uninstalling syslog-ng, but haven't done so yet. I simply
>> feel better waiting a bit, even though I don't run it anymore. Man, I
>> feel silly after typing that...
>
> Yeah, took me a while to uninstall it, but I don't find much value in
> running both.

First, welcome to systemd! =:^) I may respond to other parts too but
this one's reasonably easy, while my choice is different, so I'll explain
what and why...

Intro: I chose to keep syslog-ng running here, while I (of course) have
journald running as well. I believe the way I have configured both plays
to the advantages of each, while discounting the weak areas of each as
the other one covers that area.

What:

1) Journald is configured to use volatile storage (the /run/log/journal
subdir, with /run being tmpfs) ONLY, effectively limiting it to single-
session.

2) Syslog-ng is configured pretty much as before, except that I have its
systemd USE flag turned on, which builds it with journald support so the
two cooperate a bit better.

Without looking too much into details, I believe USE=systemd for syslog-ng
tells it to build against a library systemd/journald provides for that
purpose, such that syslog-ng's "system" source listens to journald
instead of /dev/log and the kernel (dmesg) directly.

What I can say is that while I've seen complaints from others about
loggers getting duplicate entries (from systemd and the kernel both, for
instance) in some cases, or missing entries in others (the OP, here),
I've not noticed anything like that, here, and I believe it's due to the
above -- only having the "system" and internal syslog-ng sources
configured so syslog-ng isn't getting the same message from two sources,
and building with the systemd USE flag, which basically configures syslog-
ng's system source to be lazy and let journald feed it, instead of doing
all the work of fetching and parsing the kernel and syslog socket on its
own.

Why:

I love the journald/systemd integration -- having the last few log
messages on systemctl status <service> makes troubleshooting within the
same session **MUCH** easier, and the other information journald keeps is
also useful. But, for me at least, that's most useful in the same
session. Once I reboot and the information I'm looking for is in a
previous session, the usefulness of that extra information goes down
quite a bit.

Meanwhile, for logs beyond one session, I still prefer traditional text-
based logs, tho I suppose that's partly because that's what I'm most used
to. But there's several advantages to it that I see:

a) If the system crashes, partially corrupted in-the-crash text logs can
be of at least some use after a reboot. Binary journals, not so much.

b) As far as I could find, journald has absolutely no mechanism to filter
incoming (pre-storage) log entries, while syslog-ng has had (pre-storage)
filter mechanisms for ages. Journald seems to have all sorts of outgoing
post-storage filter-on-output options, some of which are pretty nifty,
but there have been several times in the past where logs filled up with
"noise", thousands of repeat entries for something or other that occurs
regularly, say once a minute, that I simply do not care about and don't
want in my logs at all, but which I can't directly stop at the source for
whatever reason. One of them was a kernel bug that triggered for a few
kernel cycles, then eventually went away, another was a ksysguard
triggered message, with that ksysguard graph updating every second, etc.
Unfortunately, while journald makes it easy to filter these out on
output, I found no way of preventing their appearance in the journal at
all.

But syslog-ng lets me dump them without ever actually logging them, or
route them to a different log file if I prefer, keeping my primary logs
clean. =:^)

As a counter-point, however, at least journald does compression to some
degree, so tens of thousands of identical or nearly identical messages
don't take up as much room as you might expect as they compress very
well. =:^) I guess after the first instance, dups basically consist of a
timestamp and a pointer to the first message. So they don't take much
space. Which is fortunate when some "noise" message starts appearing
once a second... and you're storing it in tmpfs, thus in memory!
Obviously persistent logs would be similarly space efficient, but I still
don't like the idea of not being able to filter out the noise, BEFORE it
hits permanent storage, so I just make sure it /doesn't/ hit permanent
storage. =:^)

c) I use btrfs for my primary filesystems, and btrfs and journald's
binary-format journals don't play so well together. FWIW, the latest
systemd-219 has lots of improvements for btrfs users, including some new
features that are btrfs-specific (which I never-the-less don't expect to
use right away as they're mostly VM/container oriented and I don't do
much of that sort of thing, but I may well eventually use them), as well
as some changes that should make journaling on btrfs work somewhat
better. So this point should be to some degree eliminated after that
stuff stabilizes tho part of the fix is journald setting nocow on the
journal files and that has other implications I'll skip further
discussion of here. However, by limiting journald to tmpfs only, I still
get the best benefits of systemd/journald integration (like status
including the last few log messages as mentioned above), while
eliminating the entire class of btrfs/journald issues. Best of both
worlds. =:^)

d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng
can actually make use of some of the extra information journald passes to
it, including trusted per-app filtering and routing. One of the problems
that the old syslog approach had was that anything that could log
messages could claim to be anything it wanted, and syslog really couldn't
tell whether it was telling the truth or not, all it could do is sort and
report the messages based on what was claimed. A big bullet-point
feature of systemd/journald is that because systemd enforces service
separation via control-classes and the like, there's a much stronger
guarantee that what systemd/journald says is service X, is REALLY service
X and not service Y! But that's not limited to journald. Syslog-ng has
for several versions now had the ability to filter, route and report on
out-of-log-channel information such as the extra properties that journald
associates with each message, that make features such as systemctl status
reporting the last few log messages from that service possible, because
systemd/journald actually tracks it, AND passes it on to other loggers as
well now, providing they're built to use the API provided by the systemd
library I mentioned above. And again, for gentooers, syslog-ng is built
with that API, against that library, based on USE=systemd. So once
that's there, if desired, you can do all sorts of fancy filtering/log-
routing/etc based on information such as the application/service name,
that systemd passes along to syslog-ng.

And again, with syslog-ng, this can be done BEFORE the actual logging if
desired, instead of logging everything and filtering only on output, as
journald does. =:^)

e) I'm simply more comfortable dealing with text-based logs, since I'm
used to them and can use whatever tools I want on them, as well as being
able to split/sort/route to multiple log files in as simple or complex a
setup as I want. =:^)

Summary:

For me, splitting up logging duties so journald does the collection, but
only does session journaling to tmpfs, while syslog-ng still handles the
"persistent" logging, gives me the best of both worlds. I love systemd/
journald features such as log-messages-in-status and having seen how
useful it is, would seriously find it hard to do without. But at the
same time, I can't bring myself to accept persistent logging without the
ability to filter out the noise before it hits persistent storage, and
journal storage on btrfs is problematic anyway, with both problems nicely
solved by handing off to syslog-ng for my persistent storage needs while
keeping journald for same-session use, so that's exactly what I do. =:^)


--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
Marc Joliet posted on Tue, 24 Feb 2015 21:15:45 +0100 as excerpted:

> I uninstalled acpid, since the Gentoo Wiki page on systemd mentions that
> it might not be necessary (see the question below). Being able to do
> this was another reason I switched to tlp, since it doesn't depend on
> acpid. So far I have yet to notice anything amiss.

FWIW, I still have acpid installed here, and hadn't really thought about
it. Thanks for the heads-up. I'll have to investigate...

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
Marc Joliet posted on Tue, 24 Feb 2015 21:15:45 +0100 as excerpted:

> I would like to convert some programs I start in .xprofile to units that
> are started by my users's systemd instance. I started off with mpd,

It looks like you got this sorted, but FWIW...

I just use the standard system level mpd.service here. It drops privs
and runs as its own user, and it listens localhost for connections from
clients run as normal users, which pretty well limits any potential
security issues, and that means I can control it via either my normal X
user and GUI client, or mpc or whatever as either my normal user or my
admin user (or root, for that matter).

In fact, IMO that's mpd's biggest advantage over most other players. I
don't have to be in X to play music, and in fact, I can not only quit X,
but log out as the user I started playing the music with, and have it
continue uninterrupted, controlling it when I need to using mpc from
whatever other user (here, normally my admin user).

IOW, I really do use it as the system service/daemon that the name, mpd,
music player daemon, implies.

And as such, I don't /want/ to start it from user login or service, I
want it as a system service, which is how I run it. =:^)

But you're king of your own boxes. If you want to run it as a user-level
service and have it quit when you logout that user, go right ahead.
Don't let the fact that such usage bothers me and I don't see the point,
if you're running it as your normal user anyway, there's all those other
player apps, why use mpd in that case, bother you. (No, that isn't
sarcasm. Yes, it does bother me, but it's your box and your choice, and
presuming to take that away from you would bother me **FAR** more. So do
it the way that works best for you! =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
Marc Joliet posted on Tue, 24 Feb 2015 21:15:45 +0100 as excerpted:

> Like I mentioned above, my Desktop can *almost* suspend reliably,
> after trying it out once every year or two (it's over 8 years old).
> Mostly it would just not wake back up. The latest status (before
> systemd) was that the kernel crashed after waking up (but I think
> that was a known bug that was fixed in the meantime).

FWIW... for readers that might be considering multi-device filesystems
(like btrfs can do if so configured) or mdraid/dmraid/etc type multi-
device backed filesystems...

What I've found with suspend/hibernate over the years and on multiple
systems, is that while at least one or the other generally works,
that's on the condition of no multi-device mdraid, dmraid, btrfs, etc.
Once you get more than one physical device backing a filesystem, be
that mdraid/ dmraid/etc, or a direct multi-device filesystem such as
btrfs, suspend/ hibernate, or more precisely, the resume, becomes
problematic, because invariably, one device lags the others in
resuming, and the kernel apparently doesn't know how to properly wait
for multiple devices to all resume and stabilize at once.

The result is that the system resumes, but one or more physical devices
underlying that multi-device filesystem often stabilizes slower than
the others and gets dropped from the raid or whatever.

If it's a raid0, without redundancy, that can mean you just lost it and
everything on it, period. With other raid types it's not so bad, but
it does often mean either manual missing-device delete and re-add
(mdraid), or (on btrfs) a quick reboot as btrfs becomes unstable when a
device drops and the system will often freeze or livelock if you don't,
and then a scrub after the reboot brings the device back in, to sync
the updates back to the device that was dropped and brought back in.

So... after finding that about half the time after a suspend or
hibernate and resume cycle you have to either reboot or do manual
system maintenance anyway, pretty soon you learn not to bother with
suspend/ hibernate in the first place, and simply shutdown, and restart
from power- off when you'd otherwise resume.

Back on spinning rust, this used to annoy me greatly, as the boot time
wasn't bad, but it did mean starting with an empty cache, and losing
that several gigs of cache of much slower spinning rust at the reboot
was / painful/.

While reasonably fast SSDs are still slower than cache, the difference
is 2-3 orders of magnitude smaller, and losing the cache on reboot
isn't the big deal it once was. Between that and the fact that systemd
bootup is so fast on ssd, full shutdown and restart isn't such a big
deal these days.

But it sure would be nice if the kernel could learn to handle resume
from suspend or hibernate much like it does bootup, using bootwait or
similar kernel commandline option not just at boot, but at resume as
well, so systems that can and do /boot/ multiple devices just fine,
can /resume/ them just fine as well. Then we'd not have to worry about
such problems. Oh, well... maybe someday...

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman


--
Duncan - No HTML messages please; they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
On Wed, Feb 25, 2015 at 2:50 AM, Marc Joliet <marcec@gmx.de> wrote:
> Am Tue, 24 Feb 2015 16:44:59 -0500
> schrieb Rich Freeman <rich0@gentoo.org>:
>
>> > === Timers ===
>> >
>> > Can a systemd timer depend on a mount point such that it waits until the mount
>> > point exists before running? Or will it fail after a timeout? I want to
>> > research this myself, but haven't gotten around to it yet.
>>
>> So, timer units are units, and units can have dependencies, and mounts
>> can be dependencies since mounts are units. However, if you set the
>> dependency on the timer itself, then the timer won't start running
>> until the mount exists. You probably want the depencency to be on the
>> service started by the timer (so the timer is watching the clock, but
>> the service won't start without the mount).
>
> Wait, so the timer won't start watching the clock until its dependencies are
> met (i.e, the mount point appears)? Is that what you mean? Because that might
> be more in line with what I want (though I'm not sure yet).

If you set the dependency on the timer, then the timer doesn't start
watching the clock until they're met. If you set the dependency on
the service started by the timer then it will watch the clock but not
launch the service if the dependency isn't met. You can set the
dependency in either or both places. The timer and the service are
both units.

>
>> If you set a
>> Requires=foo.mount and After=foo.mount, then the service shouldn't run
>> unless foo.mount is available. I suspect systemd will attempt to
>> mount the filesystem when it runs the service, and you'll get units in
>> the failed state if that doesn't work.
>>
>> However, I haven't tested any of this. I suspect it wouldn't take
>> much to work this out. I have a mount dependency in one of my
>> services. Just look at the mount units in /run/systemd/generator for
>> the name of the mount unit systemd is creating from fstab.
>
> Right, so IIUC, I would have a oneshot service that does the backup, and the timer
> runs that, and of course the timer can depend on the mount point. And if the
> mount point doesn't exist, then the service started by the timer will fail.
>
> What I would prefer to have is a timer that only runs if *both* the time *and*
> mount conditions are met. Skimming the man page, this does not seem possible.
> I suppose it would be nice if timers learned "conditions" on which they should
> wait in addition to the time condition, but maybe that's outside the scope of
> systemd?

I think if you just set the dependency on the service you'll get the
behavior you desire. Systemd will try to mount the backup filesystem,
and if that fails it won't run the backup.

You can set conditions on units as well, like only running if they're
on AC power or on amd64 or to run one unit the first time you start a
service and a different unit every other time. Some of that was
designed to implement some of the stateless system features they're
adding to systemd.

--
Rich
Re: Re: Systemd migration: opinion and questions [ In reply to ]
On Wed, Feb 25, 2015 at 5:13 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>
> a) If the system crashes, partially corrupted in-the-crash text logs can
> be of at least some use after a reboot. Binary journals, not so much.

Have you tested this, or found some other data to support this? I
think that journalctl does parse truncated files.

>
> But syslog-ng lets me dump them without ever actually logging them, or
> route them to a different log file if I prefer, keeping my primary logs
> clean. =:^)

I was thinking about this and another advantage of split log files is
that you could have different rotation/retention policies for each. I
believe journald's log rotation is one-size-fits-all.

--
Rich
Re: Systemd migration: opinion and questions [ In reply to ]
Am 24.02.2015 22:51, schrieb Frank Peters:

>
> It requires about just as long (3 secs) on my desktop machine *without*
> systemd by just using my custom bash init script (no sysvinit or openrc
> either).
> Furthermore, I use no SSD.
>
> But there's also no login required so it's probably even faster -- and
> there's also no long list of permanently running daemons as well.
> (After a boot, ps ax shows a very sparse process list.)
>
> For me, systemd is totally unnecessary, excessively burdensome, and
> highly obfuscating to a complete control and understanding of my
> system.
>
> To each his own. If you want/need it, fine. But don't expect me,
> or everyone else, to slavishly follow.
>
> As has been more or less officially stated, systemd is an attempt to
> provide a uniform and monolithic kernel-user space interface FOR THE
> BENEFIT OF POTENTIAL (COMMERCIAL) SOFTWARE DEVELOPERS. RedHat wants
> Linux to be just like MS Windows so that *they* (RedHat) can be just
> like MS Windows.
>
> Let RedHat leave (fork) the Linux community and go their own separate
> way.
>
> Maybe it's time to begin shifting a lot more attention to
> Gentoo/FreeBSD.
>
> Frank Peters

- ditto! -

Very well spoken.

Michael
Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 07:01:59 -0500
schrieb Rich Freeman <rich0@gentoo.org>:

> On Wed, Feb 25, 2015 at 2:50 AM, Marc Joliet <marcec@gmx.de> wrote:
> > Am Tue, 24 Feb 2015 16:44:59 -0500
> > schrieb Rich Freeman <rich0@gentoo.org>:
> >
> >> > === Timers ===
> >> >
> >> > Can a systemd timer depend on a mount point such that it waits until the mount
> >> > point exists before running? Or will it fail after a timeout? I want to
> >> > research this myself, but haven't gotten around to it yet.
> >>
> >> So, timer units are units, and units can have dependencies, and mounts
> >> can be dependencies since mounts are units. However, if you set the
> >> dependency on the timer itself, then the timer won't start running
> >> until the mount exists. You probably want the depencency to be on the
> >> service started by the timer (so the timer is watching the clock, but
> >> the service won't start without the mount).
> >
> > Wait, so the timer won't start watching the clock until its dependencies are
> > met (i.e, the mount point appears)? Is that what you mean? Because that might
> > be more in line with what I want (though I'm not sure yet).
>
> If you set the dependency on the timer, then the timer doesn't start
> watching the clock until they're met. If you set the dependency on
> the service started by the timer then it will watch the clock but not
> launch the service if the dependency isn't met. You can set the
> dependency in either or both places. The timer and the service are
> both units.

OK, I think I got it.

> >
> >> If you set a
> >> Requires=foo.mount and After=foo.mount, then the service shouldn't run
> >> unless foo.mount is available. I suspect systemd will attempt to
> >> mount the filesystem when it runs the service, and you'll get units in
> >> the failed state if that doesn't work.
> >>
> >> However, I haven't tested any of this. I suspect it wouldn't take
> >> much to work this out. I have a mount dependency in one of my
> >> services. Just look at the mount units in /run/systemd/generator for
> >> the name of the mount unit systemd is creating from fstab.
> >
> > Right, so IIUC, I would have a oneshot service that does the backup, and the timer
> > runs that, and of course the timer can depend on the mount point. And if the
> > mount point doesn't exist, then the service started by the timer will fail.
> >
> > What I would prefer to have is a timer that only runs if *both* the time *and*
> > mount conditions are met. Skimming the man page, this does not seem possible.
> > I suppose it would be nice if timers learned "conditions" on which they should
> > wait in addition to the time condition, but maybe that's outside the scope of
> > systemd?
>
> I think if you just set the dependency on the service you'll get the
> behavior you desire. Systemd will try to mount the backup filesystem,
> and if that fails it won't run the backup.
>
> You can set conditions on units as well, like only running if they're
> on AC power or on amd64 or to run one unit the first time you start a
> service and a different unit every other time. Some of that was
> designed to implement some of the stateless system features they're
> adding to systemd.

Right, I'll have a look at them.

Thanks
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 10:13:14 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted:
>
> > On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@gmx.de> wrote:
> >>
> >> == Things I have *not* gotten rid of (yet) ==
> >>
> >> I plan on uninstalling syslog-ng, but haven't done so yet. I simply
> >> feel better waiting a bit, even though I don't run it anymore. Man, I
> >> feel silly after typing that...
> >
> > Yeah, took me a while to uninstall it, but I don't find much value in
> > running both.
>
> First, welcome to systemd! =:^) I may respond to other parts too but
> this one's reasonably easy, while my choice is different, so I'll explain
> what and why...
>
> Intro: I chose to keep syslog-ng running here, while I (of course) have
> journald running as well. I believe the way I have configured both plays
> to the advantages of each, while discounting the weak areas of each as
> the other one covers that area.
>
> What:
>
> 1) Journald is configured to use volatile storage (the /run/log/journal
> subdir, with /run being tmpfs) ONLY, effectively limiting it to single-
> session.

Right, I remember that from your thread a while back.

> 2) Syslog-ng is configured pretty much as before, except that I have its
> systemd USE flag turned on, which builds it with journald support so the
> two cooperate a bit better.
[...]
> Meanwhile, for logs beyond one session, I still prefer traditional text-
> based logs, tho I suppose that's partly because that's what I'm most used
> to. But there's several advantages to it that I see:

Personally, I don't mind the binary format, since the engineering advantages
convinced me (constant-time access, lots of meta-data which you can filter by,
git-inspired chain of hashes for integrity checking, etc.).

> a) If the system crashes, partially corrupted in-the-crash text logs can
> be of at least some use after a reboot. Binary journals, not so much.

As Rich mentioned, and as I remember reading myself (on the btrfs ML?), the
journal will do its best to read back from corrupted logs and is supposedly
pretty resilient.

> b) As far as I could find, journald has absolutely no mechanism to filter
> incoming (pre-storage) log entries, while syslog-ng has had (pre-storage)
> filter mechanisms for ages. Journald seems to have all sorts of outgoing
> post-storage filter-on-output options, some of which are pretty nifty,
> but there have been several times in the past where logs filled up with
> "noise", thousands of repeat entries for something or other that occurs
> regularly, say once a minute, that I simply do not care about and don't
> want in my logs at all, but which I can't directly stop at the source for
> whatever reason. One of them was a kernel bug that triggered for a few
> kernel cycles, then eventually went away, another was a ksysguard
> triggered message, with that ksysguard graph updating every second, etc.
> Unfortunately, while journald makes it easy to filter these out on
> output, I found no way of preventing their appearance in the journal at
> all.
>
> But syslog-ng lets me dump them without ever actually logging them, or
> route them to a different log file if I prefer, keeping my primary logs
> clean. =:^)

That is a very good point, I hadn't thought of that.

However, I think that I prefer the post-filtering approach, since it is in
principle the more flexible one. At work I sometimes mix the logs from
different units (e.g., units A and B, and then units B, C, and D).

But I realised today that journalctl has no negation operator! There is,
however, an RFE for it.

But regardless of what you use, I think that the worst offenders are services
that write logs themselves (I'm looking at you, samba).

> As a counter-point, however, at least journald does compression to some
> degree, so tens of thousands of identical or nearly identical messages
> don't take up as much room as you might expect as they compress very
> well. =:^) I guess after the first instance, dups basically consist of a
> timestamp and a pointer to the first message. So they don't take much
> space. Which is fortunate when some "noise" message starts appearing
> once a second... and you're storing it in tmpfs, thus in memory!
> Obviously persistent logs would be similarly space efficient, but I still
> don't like the idea of not being able to filter out the noise, BEFORE it
> hits permanent storage, so I just make sure it /doesn't/ hit permanent
> storage. =:^)

Yeah, I think that's a good use case for pre-filtering: an overly noisy service
whose logs you purposely want to ignore (for example, because you're still
testing it - but then again, for that particular case you could probably use
systemd-nspawn and log to an isolated journal).

> c) I use btrfs for my primary filesystems, and btrfs and journald's
> binary-format journals don't play so well together. FWIW, the latest
> systemd-219 has lots of improvements for btrfs users, including some new
> features that are btrfs-specific (which I never-the-less don't expect to
> use right away as they're mostly VM/container oriented and I don't do
> much of that sort of thing, but I may well eventually use them), as well
> as some changes that should make journaling on btrfs work somewhat
> better. So this point should be to some degree eliminated after that
> stuff stabilizes tho part of the fix is journald setting nocow on the
> journal files and that has other implications I'll skip further
> discussion of here. However, by limiting journald to tmpfs only, I still
> get the best benefits of systemd/journald integration (like status
> including the last few log messages as mentioned above), while
> eliminating the entire class of btrfs/journald issues. Best of both
> worlds. =:^)

Well, I'm on an SSD, but even on the laptop I haven't noticed any performance
issues (yet). Then again, I use autodefrag, so that probably helps.

What's funny though is that the systemd news file
(http://cgit.freedesktop.org/systemd/systemd/tree/NEWS) occasionally
refers to non-btrfs file systems as "legacy file sysetms". At least, as a
btrfs user I think it's funny :) .

> d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng
> can actually make use of some of the extra information journald passes to
> it, including trusted per-app filtering and routing. One of the problems
> that the old syslog approach had was that anything that could log
> messages could claim to be anything it wanted, and syslog really couldn't
> tell whether it was telling the truth or not, all it could do is sort and
> report the messages based on what was claimed. A big bullet-point
> feature of systemd/journald is that because systemd enforces service
> separation via control-classes and the like, there's a much stronger
> guarantee that what systemd/journald says is service X, is REALLY service
> X and not service Y! But that's not limited to journald. Syslog-ng has
> for several versions now had the ability to filter, route and report on
> out-of-log-channel information such as the extra properties that journald
> associates with each message, that make features such as systemctl status
> reporting the last few log messages from that service possible, because
> systemd/journald actually tracks it, AND passes it on to other loggers as
> well now, providing they're built to use the API provided by the systemd
> library I mentioned above. And again, for gentooers, syslog-ng is built
> with that API, against that library, based on USE=systemd. So once
> that's there, if desired, you can do all sorts of fancy filtering/log-
> routing/etc based on information such as the application/service name,
> that systemd passes along to syslog-ng.
>
> And again, with syslog-ng, this can be done BEFORE the actual logging if
> desired, instead of logging everything and filtering only on output, as
> journald does. =:^)

That's actually pretty cool :) . You're basically using the journal as a log
accumulator and syslog-ng for routing and post-processing its messages.

Whiles that's not how I want to do things, I still appreciate how cool that
is :) (and the fact that the journal can be used that way).

> e) I'm simply more comfortable dealing with text-based logs, since I'm
> used to them and can use whatever tools I want on them, as well as being
> able to split/sort/route to multiple log files in as simple or complex a
> setup as I want. =:^)
>
> Summary:
>
> For me, splitting up logging duties so journald does the collection, but
> only does session journaling to tmpfs, while syslog-ng still handles the
> "persistent" logging, gives me the best of both worlds. I love systemd/
> journald features such as log-messages-in-status and having seen how
> useful it is, would seriously find it hard to do without. But at the
> same time, I can't bring myself to accept persistent logging without the
> ability to filter out the noise before it hits persistent storage, and
> journal storage on btrfs is problematic anyway, with both problems nicely
> solved by handing off to syslog-ng for my persistent storage needs while
> keeping journald for same-session use, so that's exactly what I do. =:^)

Right :) .

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 10:33:37 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Marc Joliet posted on Tue, 24 Feb 2015 21:15:45 +0100 as excerpted:
>
> > I would like to convert some programs I start in .xprofile to units that
> > are started by my users's systemd instance. I started off with mpd,
>
> It looks like you got this sorted, but FWIW...
>
> I just use the standard system level mpd.service here. It drops privs
> and runs as its own user, and it listens localhost for connections from
> clients run as normal users, which pretty well limits any potential
> security issues, and that means I can control it via either my normal X
> user and GUI client, or mpc or whatever as either my normal user or my
> admin user (or root, for that matter).
>
> In fact, IMO that's mpd's biggest advantage over most other players. I
> don't have to be in X to play music, and in fact, I can not only quit X,
> but log out as the user I started playing the music with, and have it
> continue uninterrupted, controlling it when I need to using mpc from
> whatever other user (here, normally my admin user).
>
> IOW, I really do use it as the system service/daemon that the name, mpd,
> music player daemon, implies.
>
> And as such, I don't /want/ to start it from user login or service, I
> want it as a system service, which is how I run it. =:^)

I used to run it like that when I was still using pure ALSA, but then I
switched to pulseaudio, and didn't want to be in the unsupported "running
pulseaudio in system mode" class of users. I actually did run mpd as a system
service that connected to my users pulseaudio instance, but that was... such an
ugly hack.

But either way, if you start mpd in ~/.xprofile, for example, and let it
daemonise, it will still keep running after you log out. And mpd still doesn't
exit when I quit my desktop session, so that has yet to be a problem (more
precisely, the user instance of systemd doesn't exit).

But if you want to just turn your computer on and listen to music, then that's
obviously suboptimal, having to log in to start your music player ;) .

(Personally, I want to set up a NAS and a RaspberryPi or similar as a sort of
music center (i.e., I don't plan on archiving DVD rips or anything, at least
not currently). My desktop would be too obtrusive and power hungry for such a
setup.)

> But you're king of your own boxes. If you want to run it as a user-level
> service and have it quit when you logout that user, go right ahead.
> Don't let the fact that such usage bothers me and I don't see the point,
> if you're running it as your normal user anyway, there's all those other
> player apps, why use mpd in that case, bother you. (No, that isn't
> sarcasm. Yes, it does bother me, but it's your box and your choice, and
> presuming to take that away from you would bother me **FAR** more. So do
> it the way that works best for you! =:^)

Again, it *doesn't* terminate when I log out. Secondly, I like the various UIs
for it (I primarily use mpc with the media keys of my keyboard, and ncmpcpp and
sonata otherwise, though mostly the former), and that I can control it remotely
(I do so from the laptop often).

But really, what I like the most is the background nature of it. Or put
differently, mpd is unobtrusive (I mean, it has to be, it's a daemon for crying
out loud ;) ). I can start playing music and forget that MPD is there. The
music is simply "there", if you get what I mean.

On the laptop, however, I do in fact run something else when I'm not at home,
namely MOC, which is the next best thing (it can daemonise, too, and is
similarly unobtrusive, but only has one UI).

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Re: Systemd migration: opinion and questions [ In reply to ]
On Wed, Feb 25, 2015 at 2:17 PM, Marc Joliet <marcec@gmx.de> wrote:
> Am Wed, 25 Feb 2015 10:33:37 +0000 (UTC)
> schrieb Duncan <1i5t5.duncan@cox.net>:
>
>> But you're king of your own boxes. If you want to run it as a user-level
>> service and have it quit when you logout that user, go right ahead.
>
> Again, it *doesn't* terminate when I log out.

FYI - this behavior is completely configurable - you can enable or
disable linger for any particular user. I believe you can also use
this to do things like kill screen sessions left behind by a user. I
think the default is for systemd to not clean up when a user's session
ends, but systemd DOES cleanup with a vengeance when a service ends
(so if you're like me and spawn background processes from shell
scripts in something like cron.daily then make sure you wait somewhere
so that systemd doesn't kill everything when your timer script reaches
the end - or you can tell systemd to not do cleanup but I don't think
that is a great practice).


--
Rich
Re: Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 14:31:55 -0500
schrieb Rich Freeman <rich0@gentoo.org>:

> On Wed, Feb 25, 2015 at 2:17 PM, Marc Joliet <marcec@gmx.de> wrote:
> > Am Wed, 25 Feb 2015 10:33:37 +0000 (UTC)
> > schrieb Duncan <1i5t5.duncan@cox.net>:
> >
> >> But you're king of your own boxes. If you want to run it as a user-level
> >> service and have it quit when you logout that user, go right ahead.
> >
> > Again, it *doesn't* terminate when I log out.
>
> FYI - this behavior is completely configurable - you can enable or
> disable linger for any particular user.

Ah, then the *-linger commands to loginctl are related to this? However,
loginctl says:

% loginctl show-user 1000
UID=1000
GID=100
Name=marcec
Timestamp=Mi 2015-02-25 18:36:59 CET
TimestampMonotonic=11724536
RuntimePath=/run/user/1000
Service=user@1000.service
Slice=user-1000.slice
State=active
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0
Linger=no

Hmm, one more thing to look this up, I guess.

Ah, I think I found it: I think it's the KillUserProcesses option
in logind.conf(5), which defaults to "no";

KillUserProcesses=
Takes a boolean argument. Configures whether the processes of a user
should be killed when the user completely logs out (i.e. after the
user's last session ended). Defaults to "no".

Perhaps I'll explicitly configure that, just so an upgrade doesn't accidentally
break things.

Ah, and looking at loginctl(1) now I understand what linger means: it lets you
start systemd user sessions at boot, without having to log in (so I was wrong
in the MPD sub-thread). Nice!

[...]

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Am Tue, 24 Feb 2015 16:51:11 -0500
schrieb Frank Peters <frank.peters@comcast.net>:

> On Tue, 24 Feb 2015 21:15:45 +0100
> Marc Joliet <marcec@gmx.de> wrote:
>
> >
> > I liked systemd enough to want to switch my desktop to it before I
> > managed to finish writing this. And holy crap, Duncan was right: systemd is
> > *so* *fast* on an SSD, it's just not funny. It takes *3 seconds* after the
> > kernel boots for me to get a login screen
> >
>
> It requires about just as long (3 secs) on my desktop machine *without*
> systemd by just using my custom bash init script (no sysvinit or openrc either).
> Furthermore, I use no SSD.

FWIW, I was implicitly comparing systemd to OpenRC, which I was running
previously (hell, I started using it in April 2009 when it was at version
0.4.3-r1, a bit more than than two years before it was first marked stable).
What I was trying to say was that the speed improvement after installing an SSD
was noticeable, but not overly so, whereas systemd exploits it to a much higher
degree (as Duncan mentioned a while back, which is why I referred to him).

And just to make it clear: I'm not bashing OpenRC. It was a really nice init
system in the years that I used it (I especially grew fond of netifrc, and am
glad that netctl is similarly declarative), but systemd serves my purposes
better now.

> But there's also no login required so it's probably even faster -- and
> there's also no long list of permanently running daemons as well.
> (After a boot, ps ax shows a very sparse process list.)

A system that barely runs anything will boot quickly no matter whether it's on
an SSD or on a hard disk. Though your point still stands: you run a very
minimal system, so you see no need for anything to manage it. Put differently:
why try to manage complexity where there is none?

> For me, systemd is totally unnecessary, excessively burdensome, and
> highly obfuscating to a complete control and understanding of my
> system.
>
> To each his own.

Exactly. I fully respect your wish to control your system totally :) .
Something like systemd would just get in the way.

> If you want/need it, fine. But don't expect me,
> or everyone else, to slavishly follow.

Please don't start with this "us vs. them" crap, it's wholly unnecessary. We
are not antagonists, and nobody is trying to enslave anybody.

> As has been more or less officially stated, systemd is an attempt to
> provide a uniform and monolithic kernel-user space interface FOR THE
> BENEFIT OF POTENTIAL (COMMERCIAL) SOFTWARE DEVELOPERS. RedHat wants
> Linux to be just like MS Windows so that *they* (RedHat) can be just
> like MS Windows.

I've come to a different conclusion regarding the motives of the systemd
developers (unification, yes, but to the benefit of as many Linux users as
possible, and not just for commercial interests).

But again, we both want different things from our systems, and that's *fine*.
So what if I like systemd and you don't? Then I'll use it, and you won't.
There's no need to turn this into a conflict.

[...]
> Frank Peters

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Am Tue, 24 Feb 2015 21:15:45 +0100
schrieb Marc Joliet <marcec@gmx.de>:

> However, I did find something unexpected. I haven't looked deeply yet, but I
> noticed some dmesg entries not ending up in the journal (i.e., a bunch of "-"
> lines in-between the "+" lines in the diff output). Did I do something wrong, or
> is this known behaviour when running the journal and syslog-ng simultaneously?
> I'll look more closely tomorrow, but wondered if anybody here noticed anything
> similar.

OK, I looked at the logs. It turns out that it was simply that various log
entries (mostly dmesg logs) were positioned differently or given different time
stamps. For example, this pinentry segfault:

Feb 15 01:11:03 arthur lightdm[1445]: ** (lightdm:1445): WARNING **: Could not create user data directory /var/lib/lightdm-data/marcec: Error creating directory: No such file or directory
-Feb 15 01:12:13 arthur kernel: [ 159.526696] pinentry[2323]: segfault at 18 ip 00007fe8e025965c sp 00007fffa2ba4678 error 4 in libncursesw.so.5.9[7fe8e023e000+5d000]
+Feb 15 01:11:03 arthur lightdm[1891]: pam_unix(lightdm:session): session opened for user marcec by (uid=0)
[...]
+Feb 15 01:12:13 arthur kernel: pinentry[2323]: segfault at 18 ip 00007fe8e025965c sp 00007fffa2ba4678 error 4 in libncursesw.so.5.9[7fe8e023e000+5d000]

Notice that the journal entry is missing the dmesg time stamp. The reason for
that (whatever it is) is probably also the reason why some entries have slightly
different time stamps (e.g., 01:09:54 vs. 01:09:40).

I didn't look at every single difference, but it looks like no messages was
missed.

Ah, no, there is one class of messages that only went to syslog-ng:

-Feb 18 19:48:27 arthur kernel: 6.803784] systemd-journald[132]: Received request to flush runtime journal from PID 1
-Feb 18 19:52:14 arthur kernel: 245.293713] systemd-journald[132]: Failed to read ACL on /var/log/journal/b3a495d35e890b80816684a4521fc1cc/user-1000.journal, ignoring: Operation not supported

(Note that the ACL thing is fixed, I simply did't have btrfs POSIX ACL support
activated before the migration.)

That's... interesting. Why does the journal not contain these entries
relating to itself? All I can find in the journal related to flushing are lots
of "Time spent on flushing" entries:

% journalctl|grep flush
Feb 15 00:19:45 arthur systemd-journal[131]: Time spent on flushing to /var is 20.220ms for 798 entries.
Feb 15 00:25:20 arthur kernel: nr_pdflush_threads exported in /proc is scheduled for removal
Feb 15 00:32:36 arthur systemd-journal[131]: Time spent on flushing to /var is 570.566ms for 775 entries.
[...]

Grepping for ACL also turns up empty. What gives?

--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 00:11:18 +0100
schrieb Marc Joliet <marcec@gmx.de>:

[...]
> What was more difficult was getting it to depend on the right network target
> properly. The same "Requires" line that worked for samba and cups fails with
> "failed to add dependency" type errors:
>
> Feb 24 21:20:01 marcec systemd[540]: Cannot add dependency job for unit
> mpd.service, ignoring: Unit netctl@dhcp-wan0.service failed to load: No such file or directory.
>
> So apparently user units can't "Require" system units? However, adding a
> symlink from the service unit in /etc to .../mpd.service.requires/ solved that
> problem, but yielded a new one:
>
> Feb 24 23:26:49 marcec network[2200]: netctl needs root privileges
>
> Oh, well... Maybe it would work with systemd-networkd?
[...]

FWIW, I found this earlier today:
http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/, which even has
its own FAQ entry (guess I should have found it earlier, then). Systemd has a
network-online.target, which looks like what I want(ed). I'm too tired to try it
now, but I'll do so tomorrow.

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Rich Freeman posted on Wed, 25 Feb 2015 07:13:03 -0500 as excerpted:

> On Wed, Feb 25, 2015 at 5:13 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>>
>> a) If the system crashes, partially corrupted in-the-crash text logs
>> can be of at least some use after a reboot. Binary journals, not so
>> much.
>
> Have you tested this, or found some other data to support this? I think
> that journalctl does parse truncated files.

I believe I've seen posts to that effect on the btrfs list (my biggest
non-gentoo participation). Note that journald does rotation of I believe
8 journal files by default, and ideally, it'd only be the last one chewed
up. Also, cleanly truncated is one thing; file-system-corrupted so it
ends up with a piece of some other totally different file in it (as most
more modern filesystems take pains to prevent, at least on Linux, as the
other file might be from another user and that ends up being a security
risk, so modern filesystems generally zero-truncate it if they think it
might be corrupted in that manner) is something entirely different.

However, fair point. My support on this bit /is/ a bit fuzzier than I'd
like. Thanks for pointing that out, to me as well. =:^)

>> But syslog-ng lets me dump them without ever actually logging them, or
>> route them to a different log file if I prefer, keeping my primary logs
>> clean. =:^)
>
> I was thinking about this and another advantage of split log files is
> that you could have different rotation/retention policies for each. I
> believe journald's log rotation is one-size-fits-all.

I believe so... because everything journald logs (well, everything
system, user session logs are I'm pretty sure treated separately, at
least with persistent storage enabled, there's a note somewhere in the
documentation that if volatile-only is set, journald throws user journals
in with the system journals too, which implies it doesn't, with
persistent storage) is thrown in the same log, so there's little choice
on rotation strategy.

But of course syslog-ng allows you to split the logs however you want,
and route individual log messages to one or more logs. And those logs in
turn can be rotated on entirely different policies.

Thanks for that point too. I'm so used to thinking in terms of being
able to split logs and setup individual logrotate policies for them, I
hadn't even considered that implication of the common journald journals.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
Marc Joliet posted on Wed, 25 Feb 2015 19:56:32 +0100 as excerpted:

> But regardless of what you use, I think that the worst offenders are
> services that write logs themselves (I'm looking at you, samba).

>> c) I use btrfs for my primary filesystems, and btrfs and journald's
>> binary-format journals don't play so well together. [...]
>
> Well, I'm on an SSD, but even on the laptop I haven't noticed any
> performance issues (yet). Then again, I use autodefrag, so that
> probably helps.

Autodefrag does help.

There are two related issues at work, here.

The primary one is that pretty much any COW-based filesystem, including
btrfs, is going to have problems with internal-rewrite-pattern (as
opposed to append-only rewrites) files of any significant size. At the
small end this includes sqlite database files such as those firefox and
other mozilla products use. These, autodefrag manages well.

At the larger end are multi-gig VM images and similarly sized database
files. These, autodefrag doesn't manage so well, particularly if writes
are coming in at any significant rate, because at some point it's going
to take longer to rewrite the entire file (or even the affected normally
one-gig data chunk) than the time between incoming writes.

And the place where such fragmentation REALLY shows up is trying to run
btrfs filesystem maintenance commands like balance. On a sufficiently
fragmented filesystemsystem, particularly with quotas on too as their
tracking significantly complicates things, balance can take WEEKS on a
single-digits terabyte filesystem.

IOW, a lot of people don't notice it until something goes wrong and
they're trying to replace a failed device with one of the btrfs raid
modes, etc. That's a nasty time to find out how tangled things were, and
realize it'll take weeks to sort out, during which another device could
well fail, leaving you high and dry!

The immediate (partial) solution to the problem with these large files,
typically over a gig, is to set them nocow (which on btrfs must be done
at creation time, while the file is still zero-sized, in ordered to take
proper effect; this is normally accomplished by setting the directory
they'll be in to nocow, which doesn't affect the directory itself, but
does cause any newly created files or subdirs in it to inherit the nocow
attribute).

And this is actually what systemd-219 is doing with the journal files now.

But, setting nocow automatically disables both transparent compression
(if otherwise enabled) and checksumming. The latter isn't actually as
bad as one might expect, because most applications (including systemd/
journald) that deal with such files already have some sort of builtin
corruption detection and possible repair functionality -- they have to in
ordered to work acceptably on traditional filesystems that didn't do
filesystem level checksumming, and letting them have at it would indeed
seem to be the best policy in this case.

The second, related problem, is snapshotting. Because snapshotting
relies on COW, snapshotting a nocow file forces it to effectively cow-1
-- the first time a block is rewritten after a snapshot, it is cowed,
despite the ordinary nocow. Now setup say hourly auto-snapshotting using
snapper or the like, and continue to write to that "nocow" file, and
pretty soon it'll be as fragmented as if it weren't nocow at all!

With careful planning, separate subvolumes for the nocow files so they
aren't snapshotted with the rest of the system, snapshotting the nocow
subvolume with a period near the low frequency end of your target range
(say every other day or weekly instead of daily or twice a day), and if
they aren't rotated out regularly, periodic scripted btrfs defrags (say
weekly or monthly) of the affected files, good admins generally can keep
fragmentation from this source at least within reason.

And systemd-219 is actually creating a separate subvolume for its journal
files now, by default, thus keeping them out of the general system (or
/var) snapshot. But while both that and nocowing the journal files now
does help, it's still a reasonably fragile solution, as long as admins
don't realize what's going on, and can be tempted to set daily or more
frequent snapshotting on the journal subvolume too (or if the subvolume
doesn't take, say because it's an existing installation where there's
already a directory by that name and thus there can't be a subvolume at
the same place with the same name).


**BUT A BIG CAVEAT** lest anyone on stable with btrfs and systemd jump
onto 219 too fast. Yes, 219 DOES have some nice new features.
Unfortunately, it's broken in a few new ways as well.

* Apparently, systemd-219's networkd breaks with at least static IPv4-
only configurations, as my network failed to come up with it. From the
errors it was trying IPv6 and because that failed (it's not even in my
kernel), it gave up and didn't even try IPv4, instead trying to set the
IPv4 IP and gateway values into IPv6, which obviously isn't going to work
at all!

* There's also issues with the new tmpfiles.d configuration that has
replaced d lines (create a directory if it doesn't exist) with new v
lines (create a subvolume if on btrfs and possible, else fallback to d
behavior and create a directory), because subvolume creation fails
differently than directory creation, and the differences aren't all
sorted, yet.

Hopefully, systemd-220 will fix the IPv4 issue and bring a bit more
maturity to the tmpfiles.d subvolumes-creation feature by properly
falling back to d/directories if need be, instead of erroring out.
Meanwhile, hopefully a gentoo systemd-219-rX release will fix some of
these issues as well. But for right now, I'd suggest staying away from
it, as it's definitely not prime-time ready in its current form.

FWIW, I'm back on 218-r3 for now, done with a quick emerge --pkgonly
<systemd-219. I've not yet masked 219, however, so an update will try to
bring it back in, and I will thus have to see what changes have happened
and either mask it or try building it again, next time I update.

> What's funny though is that the systemd news file
> (http://cgit.freedesktop.org/systemd/systemd/tree/NEWS) occasionally
> refers to non-btrfs file systems as "legacy file sysetms". At least, as
> a btrfs user I think it's funny :) .

Indeed. They've definitely adopted btrfs and are running with it. If
you've read anything about their plans, the features of btrfs really do
provide a filesystem-side ready-made solution for them to adopt, altho
I'd still not call btrfs itself exactly mature -- even more than with
other filesystems, if an admin is putting data on btrfs and doesn't have
tested backups available, they really do NOT value that data, claims to
the contrary not withstanding.

And in a way, it's good, because systemd pushing it like that means
systemd based distros will be pushing it too, which will bring far wider
deployment of btrfs, ready or not, which will in turn help btrfs mature
faster with all those additional strange-corner-case bug reports and
hopefully fixes. I just feel for the poor admins trusting their distro
as they head into this without the backups they really should have... as
ultimately, a lot of them are unfortunately going to have to learn that
no backups really DOES mean you'd rather lose that data than bother with
backups, lesson, the HARD way! =:^(

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 23:42:20 +0100
schrieb Marc Joliet <marcec@gmx.de>:

> Am Wed, 25 Feb 2015 00:11:18 +0100
> schrieb Marc Joliet <marcec@gmx.de>:
>
> [...]
> > What was more difficult was getting it to depend on the right network target
> > properly. The same "Requires" line that worked for samba and cups fails with
> > "failed to add dependency" type errors:
> >
> > Feb 24 21:20:01 marcec systemd[540]: Cannot add dependency job for unit
> > mpd.service, ignoring: Unit netctl@dhcp-wan0.service failed to load: No such file or directory.
> >
> > So apparently user units can't "Require" system units? However, adding a
> > symlink from the service unit in /etc to .../mpd.service.requires/ solved that
> > problem, but yielded a new one:
> >
> > Feb 24 23:26:49 marcec network[2200]: netctl needs root privileges
> >
> > Oh, well... Maybe it would work with systemd-networkd?
> [...]
>
> FWIW, I found this earlier today:
> http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/, which even has
> its own FAQ entry (guess I should have found it earlier, then). Systemd has a
> network-online.target, which looks like what I want(ed). I'm too tired to try it
> now, but I'll do so tomorrow.

OK, I looked at this today and it turns out that the way I reconfigured mpd (to
use 'bind_to_address "any"') is not only a simpler way to express what I had
already configured it to do anyway, but is also in line with the recommendations
in the above article (in the last section "What does this mean for me, a
Developer?").

That article also pointed me to the correct solution for my cups and dovecot
socket units: what one *should* do when one wants/has to listen to a particular
address is set "FreeBind=true", which allows the bind to succeed even if the
address is not configured yet. This is obviously much better than trying to
meddle with dependencies as I was doing above.

(Current versions (i.e., ~arch) of MPD ship a socket unit, so one can use
FreeBind there, too, if necessary.)

--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
Re: Systemd migration: opinion and questions [ In reply to ]
Am Wed, 25 Feb 2015 19:25:08 +0100
schrieb Marc Joliet <marcec@gmx.de>:

> Am Wed, 25 Feb 2015 07:01:59 -0500
> schrieb Rich Freeman <rich0@gentoo.org>:
>
> > On Wed, Feb 25, 2015 at 2:50 AM, Marc Joliet <marcec@gmx.de> wrote:
> > > Am Tue, 24 Feb 2015 16:44:59 -0500
> > > schrieb Rich Freeman <rich0@gentoo.org>:
> > >
> > >> > === Timers ===
> > >> >
> > >> > Can a systemd timer depend on a mount point such that it waits until the mount
> > >> > point exists before running? Or will it fail after a timeout? I want to
> > >> > research this myself, but haven't gotten around to it yet.
> > >>
> > >> So, timer units are units, and units can have dependencies, and mounts
> > >> can be dependencies since mounts are units. However, if you set the
> > >> dependency on the timer itself, then the timer won't start running
> > >> until the mount exists. You probably want the depencency to be on the
> > >> service started by the timer (so the timer is watching the clock, but
> > >> the service won't start without the mount).
> > >
> > > Wait, so the timer won't start watching the clock until its dependencies are
> > > met (i.e, the mount point appears)? Is that what you mean? Because that might
> > > be more in line with what I want (though I'm not sure yet).
> >
> > If you set the dependency on the timer, then the timer doesn't start
> > watching the clock until they're met. If you set the dependency on
> > the service started by the timer then it will watch the clock but not
> > launch the service if the dependency isn't met. You can set the
> > dependency in either or both places. The timer and the service are
> > both units.
>
> OK, I think I got it.

I finally looked at this more closely yesterday. No, dependencies don't do
what I want. If a dependency is not met, a unit goes into a failed state.
Since the problem is that my external drive doesn't show up properly until I
unplug it and plug it back in (well, strictly speaking, its device name shows
up, but not its partitions), that won't work because that would require manual
intervention, which I would very much like to avoid.

> > >> If you set a
> > >> Requires=foo.mount and After=foo.mount, then the service shouldn't run
> > >> unless foo.mount is available. I suspect systemd will attempt to
> > >> mount the filesystem when it runs the service, and you'll get units in
> > >> the failed state if that doesn't work.

Exactly, setting a dependency on a mount point will make systemd attempt to
mount the file system before starting the unit. Its the fact that it goes into
a failed state if that attempt fails that's the problem. Again: manual
intervention.

> > >> However, I haven't tested any of this. I suspect it wouldn't take
> > >> much to work this out. I have a mount dependency in one of my
> > >> services. Just look at the mount units in /run/systemd/generator for
> > >> the name of the mount unit systemd is creating from fstab.
> > >
> > > Right, so IIUC, I would have a oneshot service that does the backup, and the timer
> > > runs that, and of course the timer can depend on the mount point. And if the
> > > mount point doesn't exist, then the service started by the timer will fail.
> > >
> > > What I would prefer to have is a timer that only runs if *both* the time *and*
> > > mount conditions are met. Skimming the man page, this does not seem possible.
> > > I suppose it would be nice if timers learned "conditions" on which they should
> > > wait in addition to the time condition, but maybe that's outside the scope of
> > > systemd?
> >
> > I think if you just set the dependency on the service you'll get the
> > behavior you desire. Systemd will try to mount the backup filesystem,
> > and if that fails it won't run the backup.
> >
> > You can set conditions on units as well, like only running if they're
> > on AC power or on amd64 or to run one unit the first time you start a
> > service and a different unit every other time. Some of that was
> > designed to implement some of the stateless system features they're
> > adding to systemd.
>
> Right, I'll have a look at them.

The problem with conditions (as they exist in systemd currently) is the same as
with dependencies: the unit does not wait until the condition is met, but
immediately stops (only that it doesn't enter a failed state). I mean, this is
what conditions in systemd are *supposed* to do, and they do they're
designated job, but I would like the timer to wait until the condition is met
and *then* run the job. I.e., I want a *delay*.

An example of where something like this exists is fcron's lavg* options, where
the jobs is delayed until the specified load average is reached. I want this,
only for mount points.

Another possibility would be something akin to PartOf that would additionally
link two units at *startup*, i.e., the depending unit starts when its
dependency appears. Then the timers would come and go along with the mount
point, and at bootup, when the drive doesn't show up properly, once I get it
going the timers show up and elapse appropriately. However, I'm not sure
whether that would be a good/robust system design (e.g., would that mask error
modes I care about?).

*sigh*

Maybe I'm over-thinking this.

Anyway, what I ended up doing is setting Restart=on-failure with appropriate
intervals so that I get a five minute window to unplug the drive and plug it
back in. My backup script already returns appropriate error codes, so this
just worked.

I'm not entirely happy, but it's still better (in various ways) than what I had
with fcron, despite the greater verbosity of systemd timers compared to crontab
entries.

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

1 2  View All