Mailing List Archive: rsyslog vs systemd

rsyslog vs systemd

Jun 28, 2023, 8:00 PM

Post #1 of 9 (397 views)

I have eight machines, three laptops, three desktops, and two virtual,
all running Fedora 38, fully updated. All of them include systemd,
of course, and all are also running rsyslog. Seven of them update
log files in /var/log as configured by rsyslog.conf, one does not. I
can't find *any* difference in the configuration between that one
and the others.

I've looked at rsyslog.conf and the systemd service files for rsyslog
and systemd-journald. I've checked the active systemd units and,
while there are differences, none that seem relevant (mostly different
devices, etc).

When I say there are no logs in /var/log, I really mean that they are
empty. /var/log/messages, for example, does contain about 900 lines
from the last time the machine was rebooted but nothing else. After
logrotate runs, /var/log/messages is completely empty. Other empty
files include boot.log, cron, maillog, sa-update.log, secure, spooler,
and all the logs in the anaconda, cups, httpd, and sssd subdirectories.

Most of the time, of course, I don't even notice this issue. But when
I want to dianose a problem, it becomes a real headache. I'm really not
good with journalctl syntax, though I'm learning. Most of the time, I'd
really prefer to read plain text logs.

What else can I check to see why this one machine doesn't get the logs?
--
Dave Close, Compata, Irvine CA "Nothing wilts faster than laurels
dave@compata.com, +1 714 434 7359 that have been rested upon."
dhclose@alumni.caltech.edu -- Percy Bysshe Shelley

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 28, 2023, 8:37 PM

Post #2 of 9 (397 views)

Permalink

if you post the configs we can make guesses.

if the rsyslog instances are not using imjournal, then you are depending on
systemd sending logs to the non-standard place that they define and rsyslog
listening on that same non-standard place instead of the standard /dev/log
(systemd insists on taking over /dev/log)

David Lang

On Wed, 28 Jun 2023, Dave Close via rsyslog wrote:

> Date: Wed, 28 Jun 2023 20:00:28 -0700
> From: Dave Close via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Dave Close <dave@compata.com>
> Subject: [rsyslog] rsyslog vs systemd
>
> I have eight machines, three laptops, three desktops, and two virtual,
> all running Fedora 38, fully updated. All of them include systemd,
> of course, and all are also running rsyslog. Seven of them update
> log files in /var/log as configured by rsyslog.conf, one does not. I
> can't find *any* difference in the configuration between that one
> and the others.
>
> I've looked at rsyslog.conf and the systemd service files for rsyslog
> and systemd-journald. I've checked the active systemd units and,
> while there are differences, none that seem relevant (mostly different
> devices, etc).
>
> When I say there are no logs in /var/log, I really mean that they are
> empty. /var/log/messages, for example, does contain about 900 lines
> from the last time the machine was rebooted but nothing else. After
> logrotate runs, /var/log/messages is completely empty. Other empty
> files include boot.log, cron, maillog, sa-update.log, secure, spooler,
> and all the logs in the anaconda, cups, httpd, and sssd subdirectories.
>
> Most of the time, of course, I don't even notice this issue. But when
> I want to dianose a problem, it becomes a real headache. I'm really not
> good with journalctl syntax, though I'm learning. Most of the time, I'd
> really prefer to read plain text logs.
>
> What else can I check to see why this one machine doesn't get the logs?
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 1:45 AM

Post #3 of 9 (397 views)

Permalink

I would check and make sure rsyslogd is running:

systemctl is-enabled rsyslog
systemctl status -l rsyslog

To see where your environment file is, run:

systemctl cat rsyslog

Look for the EnvironmentFile line. Check the contents of that file and
make sure they make sense.

On Wed, Jun 28, 2023 at 11:38?PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> if you post the configs we can make guesses.
>
> if the rsyslog instances are not using imjournal, then you are depending
> on
> systemd sending logs to the non-standard place that they define and
> rsyslog
> listening on that same non-standard place instead of the standard /dev/log
> (systemd insists on taking over /dev/log)
>
> David Lang
>
> On Wed, 28 Jun 2023, Dave Close via rsyslog wrote:
>
> > Date: Wed, 28 Jun 2023 20:00:28 -0700
> > From: Dave Close via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: Dave Close <dave@compata.com>
> > Subject: [rsyslog] rsyslog vs systemd
> >
> > I have eight machines, three laptops, three desktops, and two virtual,
> > all running Fedora 38, fully updated. All of them include systemd,
> > of course, and all are also running rsyslog. Seven of them update
> > log files in /var/log as configured by rsyslog.conf, one does not. I
> > can't find *any* difference in the configuration between that one
> > and the others.
> >
> > I've looked at rsyslog.conf and the systemd service files for rsyslog
> > and systemd-journald. I've checked the active systemd units and,
> > while there are differences, none that seem relevant (mostly different
> > devices, etc).
> >
> > When I say there are no logs in /var/log, I really mean that they are
> > empty. /var/log/messages, for example, does contain about 900 lines
> > from the last time the machine was rebooted but nothing else. After
> > logrotate runs, /var/log/messages is completely empty. Other empty
> > files include boot.log, cron, maillog, sa-update.log, secure, spooler,
> > and all the logs in the anaconda, cups, httpd, and sssd subdirectories.
> >
> > Most of the time, of course, I don't even notice this issue. But when
> > I want to dianose a problem, it becomes a real headache. I'm really not
> > good with journalctl syntax, though I'm learning. Most of the time, I'd
> > really prefer to read plain text logs.
> >
> > What else can I check to see why this one machine doesn't get the logs?
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 3:30 PM

Post #4 of 9 (394 views)

Permalink

I wrote:

>I have eight machines, three laptops, three desktops, and two virtual,
>all running Fedora 38, fully updated. All of them include systemd,
>of course, and all are also running rsyslog. Seven of them update
>log files in /var/log as configured by rsyslog.conf, one does not. I
>can't find *any* difference in the configuration between that one
>and the others.

>I've looked at rsyslog.conf and the systemd service files for rsyslog
>and systemd-journald. I've checked the active systemd units and,
>while there are differences, none that seem relevant (mostly different
>devices, etc).

>When I say there are no logs in /var/log, I really mean that they are
>empty. /var/log/messages, for example, does contain about 900 lines
>from the last time the machine was rebooted but nothing else. After
>logrotate runs, /var/log/messages is completely empty. Other empty
>files include boot.log, cron, maillog, sa-update.log, secure, spooler,
>and all the logs in the anaconda, cups, httpd, and sssd subdirectories.

>What else can I check to see why this one machine doesn't get the logs?

Brad Van Orden wrote:

>I would check and make sure rsyslogd is running:

Yes, it is running.

>systemctl cat rsyslog
>Look for the EnvironmentFile line. Check the contents of that file and
>make sure they make sense.

The EnvironmentFile is /etc/sysconfig/rsyslog, it is identical on all
eight machines, and it is the original file from Fedora. The only non-
comment line in it is,

SYSLOGD_OPTIONS=""

David Lang wrote:

>if you post the configs we can make guesses.
>
>if the rsyslog instances are not using imjournal, then you are
>depending on systemd sending logs to the non-standard place that they
>define and rsyslog listening on that same non-standard place instead
>of the standard /dev/log (systemd insists on taking over /dev/log)

I have made no customization changes to journald on any of the eight
machines. There are some changes in rsyslog.conf on other machines but
not on the misbehaving one. On the misbehaving machine, no files from
the package rsyslog-8.2306.0-1.fc38.x86_64.rpm have changed. This
includes /usr/lib/systemd/system/rsyslog.service, /etc/rsyslog.conf,
and /etc/sysconfig/rsyslog. Every file I know of which seems relevant
has been compared to the seven working machines and I can't find any
unexpected differences. For most of the working machines, the files
are identical.

To me, the fact that boot messages, but nothing after, make it into
/var/log/messages seems like it ought to be significant. The last two
lines recorded are,

systemd[1]: Switching root.
systemd-journald[205]: Journal stopped

But those lines also appear after boot on the other machines and the
log then continues without even a gap in time.
--
Dave Close, Compata, Irvine CA +1 714 434 7359
dave@compata.com dhclose@alumni.caltech.edu
"Reality leaves a lot to the imagination." -- John Lennon

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 3:35 PM

Post #5 of 9 (394 views)

Permalink

we don't know what the default config files are on Fedora (and are all machines
the same release of fedora??)

does systemd report that it has any logs from rsyslog? is journald (the systemd
logging mechanism) working?

David Lang

On Thu, 29 Jun 2023, Dave Close via rsyslog wrote:

> I wrote:
>
>> I have eight machines, three laptops, three desktops, and two virtual,
>> all running Fedora 38, fully updated. All of them include systemd,
>> of course, and all are also running rsyslog. Seven of them update
>> log files in /var/log as configured by rsyslog.conf, one does not. I
>> can't find *any* difference in the configuration between that one
>> and the others.
>
>> I've looked at rsyslog.conf and the systemd service files for rsyslog
>> and systemd-journald. I've checked the active systemd units and,
>> while there are differences, none that seem relevant (mostly different
>> devices, etc).
>
>> When I say there are no logs in /var/log, I really mean that they are
>> empty. /var/log/messages, for example, does contain about 900 lines
>> from the last time the machine was rebooted but nothing else. After
>> logrotate runs, /var/log/messages is completely empty. Other empty
>> files include boot.log, cron, maillog, sa-update.log, secure, spooler,
>> and all the logs in the anaconda, cups, httpd, and sssd subdirectories.
>
>> What else can I check to see why this one machine doesn't get the logs?
>
> Brad Van Orden wrote:
>
>> I would check and make sure rsyslogd is running:
>
> Yes, it is running.
>
>> systemctl cat rsyslog
>> Look for the EnvironmentFile line. Check the contents of that file and
>> make sure they make sense.
>
> The EnvironmentFile is /etc/sysconfig/rsyslog, it is identical on all
> eight machines, and it is the original file from Fedora. The only non-
> comment line in it is,
>
> SYSLOGD_OPTIONS=""
>
> David Lang wrote:
>
>> if you post the configs we can make guesses.
>>
>> if the rsyslog instances are not using imjournal, then you are
>> depending on systemd sending logs to the non-standard place that they
>> define and rsyslog listening on that same non-standard place instead
>> of the standard /dev/log (systemd insists on taking over /dev/log)
>
> I have made no customization changes to journald on any of the eight
> machines. There are some changes in rsyslog.conf on other machines but
> not on the misbehaving one. On the misbehaving machine, no files from
> the package rsyslog-8.2306.0-1.fc38.x86_64.rpm have changed. This
> includes /usr/lib/systemd/system/rsyslog.service, /etc/rsyslog.conf,
> and /etc/sysconfig/rsyslog. Every file I know of which seems relevant
> has been compared to the seven working machines and I can't find any
> unexpected differences. For most of the working machines, the files
> are identical.
>
> To me, the fact that boot messages, but nothing after, make it into
> /var/log/messages seems like it ought to be significant. The last two
> lines recorded are,
>
> systemd[1]: Switching root.
> systemd-journald[205]: Journal stopped
>
> But those lines also appear after boot on the other machines and the
> log then continues without even a gap in time.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 3:56 PM

Post #6 of 9 (394 views)

Permalink

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 4:28 PM

Post #7 of 9 (394 views)

Permalink

I set Storage=volatile and the ForwardToSyslog=true, so that no logs are
written to disk, saving I/O and disk space, but the journal has the logs
since boot. Rsyslog+RELP gets the logs over to my collector and phpLogCon
is what I review the logs with.

HTH,
Brendan

On Thu, Jun 29, 2023, 6:56 PM Dave Close via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> I wrote:
>
> >I have eight machines, three laptops, three desktops, and two virtual,
> >all running Fedora 38, fully updated. All of them include systemd,
> >of course, and all are also running rsyslog. Seven of them update
> >log files in /var/log as configured by rsyslog.conf, one does not. I
> >can't find *any* difference in the configuration between that one
> >and the others.
>
> >I've looked at rsyslog.conf and the systemd service files for rsyslog
> >and systemd-journald. I've checked the active systemd units and,
> >while there are differences, none that seem relevant (mostly different
> >devices, etc).
>
> >When I say there are no logs in /var/log, I really mean that they are
> >empty. /var/log/messages, for example, does contain about 900 lines
> >from the last time the machine was rebooted but nothing else. After
> >logrotate runs, /var/log/messages is completely empty. Other empty
> >files include boot.log, cron, maillog, sa-update.log, secure, spooler,
> >and all the logs in the anaconda, cups, httpd, and sssd subdirectories.
>
> Problem solved. Apologies for the bandwidth but good to know people
> are here to help when needed.
>
> I had overlooked that there was a one-character change to
> /etc/systemd/journald.conf. The comment character at the start of
> the line, "Storage=none", had been removed. Restoring that character
> and restarting journald seems to have fixed the issue.
>
> I must have made this change a long time ago, probably out of concern
> by the disk storage occupied by the journal. But as I recall, the
> documentation said only that changing this would reduce disk usage,
> not that it would stop the journal from running entirely. And if it
> were still running, it should still be able to pass data to rsyslog.
> --
> Dave Close, Compata, Irvine CA +1 714 434 7359
> dave@compata.com dhclose@alumni.caltech.edu
> "Controlling complexity is the essence of computer programming."
> -- Brian Kernighan
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 4:46 PM

Post #8 of 9 (394 views)

Permalink

brendan kearney wrote:

>I set Storage=volatile and the ForwardToSyslog=true, so that no logs are
>written to disk, saving I/O and disk space, but the journal has the logs
>since boot. Rsyslog+RELP gets the logs over to my collector and phpLogCon
>is what I review the logs with.

Thanks, Brendan. Good to know.

Interesting that this list doesn't send me a copy of your message when
I am included as a CC. That does cut duplicates but it means that my
filtering based on the List-id in the message header doesn't work. I am
already filtering for duplicates based on the Message-id.
--
Dave Close, Compata, Irvine CA +1 714 434 7359
dave@compata.com dhclose@alumni.caltech.edu
"A man who says, 'I have learned enough and will learn no further,'
should be considered as knowing nothing at all." --Haile Selassie

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslog vs systemd [ In reply to ]

rsyslog at lists

Jun 29, 2023, 5:27 PM

Post #9 of 9 (394 views)

Permalink

On Thu, 29 Jun 2023, Dave Close via rsyslog wrote:

> Interesting that this list doesn't send me a copy of your message when
> I am included as a CC. That does cut duplicates but it means that my
> filtering based on the List-id in the message header doesn't work. I am
> already filtering for duplicates based on the Message-id.

I would guess that the direct message arrived before the one through the list,
and you threw away the list one because of the duplicate message-id

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.