Mailing List Archive: Capturing shutdown logs

Capturing shutdown logs

Mar 13, 2024, 5:49 AM

Post #1 of 13 (178 views)

Recently I came across an observation where we are not able to capture
normal reboot/shutdown logs on Fedora/RHEL distributions. In these
environments, systemd is responsible for starting the rsyslog service. A
service can have multiple dependencies, which influence how early or how
late rsyslog is started or stopped. Many years ago, we added dependency for
the network.target and network-online.target into the service file [1]. If
rsyslog started before establishing network access, it would be unable to
transmit messages to remote destinations during that period, resulting in
the generation of misleading information about the unavailability of
certain remote targets (e.g. not able to resolve hostnames).
However, this approach results in a significant tradeoff. While it prevents
misleading unavailability messages during network setup and shutdown, it
also causes rsyslog to *exit* *early* during shutdown, leading to missed
logs regarding the graceful termination of other programs. This limitation
extends to system reboots as well. Thus, while addressing one issue, the
current service configuration introduces another.
By default, we retrieve shutdown events from the journal using the
imjournal module. Journal log data is stored in memory so after shutdown,
logs are not preserved.

Has someone faced this problem? Are there any known workarounds?

[1]
https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 13, 2024, 6:47 AM

Post #2 of 13 (177 views)

Permalink

You could split rsyslog into two separate service instances.

Service 1 would do only one thing, read imjournal and write to file(s). This service would not have the network.target dependency.
Service 2 would do everything else, including reading the file(s) output from above (which survive the reboot) and sending the events within to a network destination. This service would have the network.target dependency so as to be able to deliver reliably.

Regards,

> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Recently I came across an observation where we are not able to capture
> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> environments, systemd is responsible for starting the rsyslog service. A
> service can have multiple dependencies, which influence how early or how
> late rsyslog is started or stopped. Many years ago, we added dependency for
> the network.target and network-online.target into the service file [1]. If
> rsyslog started before establishing network access, it would be unable to
> transmit messages to remote destinations during that period, resulting in
> the generation of misleading information about the unavailability of
> certain remote targets (e.g. not able to resolve hostnames).
> However, this approach results in a significant tradeoff. While it prevents
> misleading unavailability messages during network setup and shutdown, it
> also causes rsyslog to *exit* *early* during shutdown, leading to missed
> logs regarding the graceful termination of other programs. This limitation
> extends to system reboots as well. Thus, while addressing one issue, the
> current service configuration introduces another.
> By default, we retrieve shutdown events from the journal using the
> imjournal module. Journal log data is stored in memory so after shutdown,
> logs are not preserved.
>
> Has someone faced this problem? Are there any known workarounds?
>
> [1]
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 13, 2024, 7:34 AM

Post #3 of 13 (177 views)

Permalink

Anyway, you still have the part where the system as such (kernel,
systemd) is still running and rsyslog is not running anymore. You can't
help it. You can minimize the "hole" a bit, but can't force a process to
run when it had already been shut down.

If you really have such high availability needs about your logging,
maybe it's time to think about redirecting your messages to a (virtual)
serial console and capturing them externally?

But that's a bit out of scope of this list I think.

MK

On 13.03.2024 14:47, John Chivian via rsyslog wrote:
> You could split rsyslog into two separate service instances.
>
> Service 1 would do only one thing, read imjournal and write to file(s). This service would not have the network.target dependency.
> Service 2 would do everything else, including reading the file(s) output from above (which survive the reboot) and sending the events within to a network destination. This service would have the network.target dependency so as to be able to deliver reliably.
>
> Regards,
>
>
>> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Recently I came across an observation where we are not able to capture
>> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
>> environments, systemd is responsible for starting the rsyslog service. A
>> service can have multiple dependencies, which influence how early or how
>> late rsyslog is started or stopped. Many years ago, we added dependency for
>> the network.target and network-online.target into the service file [1]. If
>> rsyslog started before establishing network access, it would be unable to
>> transmit messages to remote destinations during that period, resulting in
>> the generation of misleading information about the unavailability of
>> certain remote targets (e.g. not able to resolve hostnames).
>> However, this approach results in a significant tradeoff. While it prevents
>> misleading unavailability messages during network setup and shutdown, it
>> also causes rsyslog to *exit* *early* during shutdown, leading to missed
>> logs regarding the graceful termination of other programs. This limitation
>> extends to system reboots as well. Thus, while addressing one issue, the
>> current service configuration introduces another.
>> By default, we retrieve shutdown events from the journal using the
>> imjournal module. Journal log data is stored in memory so after shutdown,
>> logs are not preserved.
>>
>> Has someone faced this problem? Are there any known workarounds?
>>
>> [1]
>> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 13, 2024, 11:25 AM

Post #4 of 13 (174 views)

Permalink

you could put the remote sender things in a seprate ruleset with a queue on that
ruleset, that would let the rest of the config run without the network
(accumulating early logs and gathering shutdown logs up to the point that
rsyslog gets shut down)

you can configure rsyslog to save the queue to disk at shutdown (but this can
take time, so you may need to increase the systemd timeout for letting rsyslog
do a clean shutdown)

David Lang

On Wed, 13 Mar 2024, Attila Lakatos via rsyslog wrote:

> Date: Wed, 13 Mar 2024 13:49:19 +0100
> From: Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Attila Lakatos <alakatos@redhat.com>
> Subject: [rsyslog] Capturing shutdown logs
>
> Recently I came across an observation where we are not able to capture
> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> environments, systemd is responsible for starting the rsyslog service. A
> service can have multiple dependencies, which influence how early or how
> late rsyslog is started or stopped. Many years ago, we added dependency for
> the network.target and network-online.target into the service file [1]. If
> rsyslog started before establishing network access, it would be unable to
> transmit messages to remote destinations during that period, resulting in
> the generation of misleading information about the unavailability of
> certain remote targets (e.g. not able to resolve hostnames).
> However, this approach results in a significant tradeoff. While it prevents
> misleading unavailability messages during network setup and shutdown, it
> also causes rsyslog to *exit* *early* during shutdown, leading to missed
> logs regarding the graceful termination of other programs. This limitation
> extends to system reboots as well. Thus, while addressing one issue, the
> current service configuration introduces another.
> By default, we retrieve shutdown events from the journal using the
> imjournal module. Journal log data is stored in memory so after shutdown,
> logs are not preserved.
>
> Has someone faced this problem? Are there any known workarounds?
>
> [1]
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 15, 2024, 1:36 AM

Post #5 of 13 (172 views)

Permalink

The solution is clean to me, however I think this could be a bottleneck for
busy systems.
Also, this would mean that I need to maintain a copy of journal logs in one
or more files.

On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com> wrote:

> You could split rsyslog into two separate service instances.
>
> Service 1 would do *only one thing*, read imjournal and write to
> file(s). This service would *not* have the network.target dependency.
> Service 2 would do *everything else*, including reading the file(s)
> output from above (which survive the reboot) and sending the events within
> to a network destination. This service would have the network.target
> dependency so as to be able to deliver reliably.
>
> Regards,
>
>
> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> Recently I came across an observation where we are not able to capture
> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> environments, systemd is responsible for starting the rsyslog service. A
> service can have multiple dependencies, which influence how early or how
> late rsyslog is started or stopped. Many years ago, we added dependency for
> the network.target and network-online.target into the service file [1]. If
> rsyslog started before establishing network access, it would be unable to
> transmit messages to remote destinations during that period, resulting in
> the generation of misleading information about the unavailability of
> certain remote targets (e.g. not able to resolve hostnames).
> However, this approach results in a significant tradeoff. While it prevents
> misleading unavailability messages during network setup and shutdown, it
> also causes rsyslog to *exit* *early* during shutdown, leading to missed
> logs regarding the graceful termination of other programs. This limitation
> extends to system reboots as well. Thus, while addressing one issue, the
> current service configuration introduces another.
> By default, we retrieve shutdown events from the journal using the
> imjournal module. Journal log data is stored in memory so after shutdown,
> logs are not preserved.
>
> Has someone faced this problem? Are there any known workarounds?
>
> [1]
>
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 15, 2024, 2:52 AM

Post #6 of 13 (171 views)

Permalink

Actually, I realized that dependency on the network.target is not the sole
problem why shutdown logs are not persisted.
Upon shutdown, there's a delay in rsyslog fetching new logs from the
journal via the journal API. By the time it can access
the journal, a SIGTERM signal has already been sent to the main rsyslog
process. While this signal doesn't immediately
terminate rsyslog, it does halt the processing of incoming data instantly.

On Wed, Mar 13, 2024 at 7:32?PM David Lang <david@lang.hm> wrote:

> you could put the remote sender things in a seprate ruleset with a queue
> on that
> ruleset, that would let the rest of the config run without the network
> (accumulating early logs and gathering shutdown logs up to the point that
> rsyslog gets shut down)
>
>
In my use case, messages enter rsyslog via the imjournal module. By
default, I have only one ruleset (the default).
I want to store logs locally and also forward them to a remote rsyslog
server. So do I understand correctly,
that I need to create a ruleset that will have a queue and will contain an
omfwd action and I need to call it in the
default ruleset? The default ruleset would also handle writing logs to
local files.

> you can configure rsyslog to save the queue to disk at shutdown (but this
> can
> take time, so you may need to increase the systemd timeout for letting
> rsyslog
> do a clean shutdown)
>
> David Lang
>
> On Wed, 13 Mar 2024, Attila Lakatos via rsyslog wrote:
>
> > Date: Wed, 13 Mar 2024 13:49:19 +0100
> > From: Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Attila Lakatos <alakatos@redhat.com>
> > Subject: [rsyslog] Capturing shutdown logs
> >
> > Recently I came across an observation where we are not able to capture
> > normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> > environments, systemd is responsible for starting the rsyslog service. A
> > service can have multiple dependencies, which influence how early or how
> > late rsyslog is started or stopped. Many years ago, we added dependency
> for
> > the network.target and network-online.target into the service file [1].
> If
> > rsyslog started before establishing network access, it would be unable to
> > transmit messages to remote destinations during that period, resulting in
> > the generation of misleading information about the unavailability of
> > certain remote targets (e.g. not able to resolve hostnames).
> > However, this approach results in a significant tradeoff. While it
> prevents
> > misleading unavailability messages during network setup and shutdown, it
> > also causes rsyslog to *exit* *early* during shutdown, leading to missed
> > logs regarding the graceful termination of other programs. This
> limitation
> > extends to system reboots as well. Thus, while addressing one issue, the
> > current service configuration introduces another.
> > By default, we retrieve shutdown events from the journal using the
> > imjournal module. Journal log data is stored in memory so after shutdown,
> > logs are not preserved.
> >
> > Has someone faced this problem? Are there any known workarounds?
> >
> > [1]
> >
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 15, 2024, 9:30 AM

Post #7 of 13 (167 views)

Permalink

imjournal uses the journal api to fetch the logs (fetching them in
near-real-time), journald keeps files internally to support it.

David Lang

On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:

> The solution is clean to me, however I think this could be a bottleneck for
> busy systems. Also, this would mean that I need to maintain a copy of journal
> logs in one or more files.
>
> On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com> wrote:
>
>> You could split rsyslog into two separate service instances.
>>
>> Service 1 would do *only one thing*, read imjournal and write to
>> file(s). This service would *not* have the network.target dependency.
>> Service 2 would do *everything else*, including reading the file(s)
>> output from above (which survive the reboot) and sending the events within
>> to a network destination. This service would have the network.target
>> dependency so as to be able to deliver reliably.
>>
>> Regards,
>>
>>
>> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>>
>> Recently I came across an observation where we are not able to capture
>> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
>> environments, systemd is responsible for starting the rsyslog service. A
>> service can have multiple dependencies, which influence how early or how
>> late rsyslog is started or stopped. Many years ago, we added dependency for
>> the network.target and network-online.target into the service file [1]. If
>> rsyslog started before establishing network access, it would be unable to
>> transmit messages to remote destinations during that period, resulting in
>> the generation of misleading information about the unavailability of
>> certain remote targets (e.g. not able to resolve hostnames).
>> However, this approach results in a significant tradeoff. While it prevents
>> misleading unavailability messages during network setup and shutdown, it
>> also causes rsyslog to *exit* *early* during shutdown, leading to missed
>> logs regarding the graceful termination of other programs. This limitation
>> extends to system reboots as well. Thus, while addressing one issue, the
>> current service configuration introduces another.
>> By default, we retrieve shutdown events from the journal using the
>> imjournal module. Journal log data is stored in memory so after shutdown,
>> logs are not preserved.
>>
>> Has someone faced this problem? Are there any known workarounds?
>>
>> [1]
>>
>> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 19, 2024, 8:02 AM

Post #8 of 13 (134 views)

Permalink

Attila, any reason you can't just use persistent journald? That is
what we did to solve the lost shutdown and crash logs. -Peter

On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
<rsyslog@lists.adiscon.com> wrote:
>
> imjournal uses the journal api to fetch the logs (fetching them in
> near-real-time), journald keeps files internally to support it.
>
> David Lang
>
> On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
>
> > The solution is clean to me, however I think this could be a bottleneck for
> > busy systems. Also, this would mean that I need to maintain a copy of journal
> > logs in one or more files.
> >
> > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com> wrote:
> >
> >> You could split rsyslog into two separate service instances.
> >>
> >> Service 1 would do *only one thing*, read imjournal and write to
> >> file(s). This service would *not* have the network.target dependency.
> >> Service 2 would do *everything else*, including reading the file(s)
> >> output from above (which survive the reboot) and sending the events within
> >> to a network destination. This service would have the network.target
> >> dependency so as to be able to deliver reliably.
> >>
> >> Regards,
> >>
> >>
> >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
> >> rsyslog@lists.adiscon.com> wrote:
> >>
> >> Recently I came across an observation where we are not able to capture
> >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> >> environments, systemd is responsible for starting the rsyslog service. A
> >> service can have multiple dependencies, which influence how early or how
> >> late rsyslog is started or stopped. Many years ago, we added dependency for
> >> the network.target and network-online.target into the service file [1]. If
> >> rsyslog started before establishing network access, it would be unable to
> >> transmit messages to remote destinations during that period, resulting in
> >> the generation of misleading information about the unavailability of
> >> certain remote targets (e.g. not able to resolve hostnames).
> >> However, this approach results in a significant tradeoff. While it prevents
> >> misleading unavailability messages during network setup and shutdown, it
> >> also causes rsyslog to *exit* *early* during shutdown, leading to missed
> >> logs regarding the graceful termination of other programs. This limitation
> >> extends to system reboots as well. Thus, while addressing one issue, the
> >> current service configuration introduces another.
> >> By default, we retrieve shutdown events from the journal using the
> >> imjournal module. Journal log data is stored in memory so after shutdown,
> >> logs are not preserved.
> >>
> >> Has someone faced this problem? Are there any known workarounds?
> >>
> >> [1]
> >>
> >> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 20, 2024, 1:16 AM

Post #9 of 13 (132 views)

Permalink

Hello Peter,

I think that would be the best solution from rsyslog point of view.
However, this would mean that logs would be stored in both
/var/log/{messages|secure|...} and the journal.
Ideally, it would be better to have them only in one place.

Thanks,
Attila

On Tue, Mar 19, 2024 at 4:03?PM Peter Portante via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Attila, any reason you can't just use persistent journald? That is
> what we did to solve the lost shutdown and crash logs. -Peter
>
> On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
> <rsyslog@lists.adiscon.com> wrote:
> >
> > imjournal uses the journal api to fetch the logs (fetching them in
> > near-real-time), journald keeps files internally to support it.
> >
> > David Lang
> >
> > On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
> >
> > > The solution is clean to me, however I think this could be a
> bottleneck for
> > > busy systems. Also, this would mean that I need to maintain a copy of
> journal
> > > logs in one or more files.
> > >
> > > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com>
> wrote:
> > >
> > >> You could split rsyslog into two separate service instances.
> > >>
> > >> Service 1 would do *only one thing*, read imjournal and write to
> > >> file(s). This service would *not* have the network.target dependency.
> > >> Service 2 would do *everything else*, including reading the file(s)
> > >> output from above (which survive the reboot) and sending the events
> within
> > >> to a network destination. This service would have the network.target
> > >> dependency so as to be able to deliver reliably.
> > >>
> > >> Regards,
> > >>
> > >>
> > >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
> > >> rsyslog@lists.adiscon.com> wrote:
> > >>
> > >> Recently I came across an observation where we are not able to capture
> > >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> > >> environments, systemd is responsible for starting the rsyslog
> service. A
> > >> service can have multiple dependencies, which influence how early or
> how
> > >> late rsyslog is started or stopped. Many years ago, we added
> dependency for
> > >> the network.target and network-online.target into the service file
> [1]. If
> > >> rsyslog started before establishing network access, it would be
> unable to
> > >> transmit messages to remote destinations during that period,
> resulting in
> > >> the generation of misleading information about the unavailability of
> > >> certain remote targets (e.g. not able to resolve hostnames).
> > >> However, this approach results in a significant tradeoff. While it
> prevents
> > >> misleading unavailability messages during network setup and shutdown,
> it
> > >> also causes rsyslog to *exit* *early* during shutdown, leading to
> missed
> > >> logs regarding the graceful termination of other programs. This
> limitation
> > >> extends to system reboots as well. Thus, while addressing one issue,
> the
> > >> current service configuration introduces another.
> > >> By default, we retrieve shutdown events from the journal using the
> > >> imjournal module. Journal log data is stored in memory so after
> shutdown,
> > >> logs are not preserved.
> > >>
> > >> Has someone faced this problem? Are there any known workarounds?
> > >>
> > >> [1]
> > >>
> > >>
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >> DON'T LIKE THAT.
> > >>
> > >>
> > >>
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 20, 2024, 1:56 AM

Post #10 of 13 (132 views)

Permalink

Hello Atilla!

You can limit the persistent journald storage in size with SystemMaxUse
option. Just find a value which is good enough for you to save all the
messages while rsyslog is down. I guess 1Gb should be more than enough.

On Wed, 20 Mar 2024 at 12:17, Attila Lakatos via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hello Peter,
>
> I think that would be the best solution from rsyslog point of view.
> However, this would mean that logs would be stored in both
> /var/log/{messages|secure|...} and the journal.
> Ideally, it would be better to have them only in one place.
>
> Thanks,
> Attila
>
> On Tue, Mar 19, 2024 at 4:03?PM Peter Portante via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> > Attila, any reason you can't just use persistent journald? That is
> > what we did to solve the lost shutdown and crash logs. -Peter
> >
> > On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
> > <rsyslog@lists.adiscon.com> wrote:
> > >
> > > imjournal uses the journal api to fetch the logs (fetching them in
> > > near-real-time), journald keeps files internally to support it.
> > >
> > > David Lang
> > >
> > > On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
> > >
> > > > The solution is clean to me, however I think this could be a
> > bottleneck for
> > > > busy systems. Also, this would mean that I need to maintain a copy of
> > journal
> > > > logs in one or more files.
> > > >
> > > > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com>
> > wrote:
> > > >
> > > >> You could split rsyslog into two separate service instances.
> > > >>
> > > >> Service 1 would do *only one thing*, read imjournal and write to
> > > >> file(s). This service would *not* have the network.target
> dependency.
> > > >> Service 2 would do *everything else*, including reading the file(s)
> > > >> output from above (which survive the reboot) and sending the events
> > within
> > > >> to a network destination. This service would have the
> network.target
> > > >> dependency so as to be able to deliver reliably.
> > > >>
> > > >> Regards,
> > > >>
> > > >>
> > > >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
> > > >> rsyslog@lists.adiscon.com> wrote:
> > > >>
> > > >> Recently I came across an observation where we are not able to
> capture
> > > >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
> > > >> environments, systemd is responsible for starting the rsyslog
> > service. A
> > > >> service can have multiple dependencies, which influence how early or
> > how
> > > >> late rsyslog is started or stopped. Many years ago, we added
> > dependency for
> > > >> the network.target and network-online.target into the service file
> > [1]. If
> > > >> rsyslog started before establishing network access, it would be
> > unable to
> > > >> transmit messages to remote destinations during that period,
> > resulting in
> > > >> the generation of misleading information about the unavailability of
> > > >> certain remote targets (e.g. not able to resolve hostnames).
> > > >> However, this approach results in a significant tradeoff. While it
> > prevents
> > > >> misleading unavailability messages during network setup and
> shutdown,
> > it
> > > >> also causes rsyslog to *exit* *early* during shutdown, leading to
> > missed
> > > >> logs regarding the graceful termination of other programs. This
> > limitation
> > > >> extends to system reboots as well. Thus, while addressing one issue,
> > the
> > > >> current service configuration introduces another.
> > > >> By default, we retrieve shutdown events from the journal using the
> > > >> imjournal module. Journal log data is stored in memory so after
> > shutdown,
> > > >> logs are not preserved.
> > > >>
> > > >> Has someone faced this problem? Are there any known workarounds?
> > > >>
> > > >> [1]
> > > >>
> > > >>
> >
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
> > > >> _______________________________________________
> > > >> rsyslog mailing list
> > > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > >> http://www.rsyslog.com/professional-services/
> > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > > >> DON'T LIKE THAT.
> > > >>
> > > >>
> > > >>
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you DON'T LIKE THAT.
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

--
Yury Bushmelev
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 20, 2024, 10:03 AM

Post #11 of 13 (132 views)

Permalink

Implied is not to store locally via rsyslog but use rsyslog to send off-host.

On Wed, Mar 20, 2024 at 4:16?AM Attila Lakatos <alakatos@redhat.com> wrote:
>
> Hello Peter,
>
> I think that would be the best solution from rsyslog point of view.
> However, this would mean that logs would be stored in both /var/log/{messages|secure|...} and the journal.
> Ideally, it would be better to have them only in one place.
>
> Thanks,
> Attila
>
> On Tue, Mar 19, 2024 at 4:03?PM Peter Portante via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Attila, any reason you can't just use persistent journald? That is
>> what we did to solve the lost shutdown and crash logs. -Peter
>>
>> On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
>> <rsyslog@lists.adiscon.com> wrote:
>> >
>> > imjournal uses the journal api to fetch the logs (fetching them in
>> > near-real-time), journald keeps files internally to support it.
>> >
>> > David Lang
>> >
>> > On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
>> >
>> > > The solution is clean to me, however I think this could be a bottleneck for
>> > > busy systems. Also, this would mean that I need to maintain a copy of journal
>> > > logs in one or more files.
>> > >
>> > > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com> wrote:
>> > >
>> > >> You could split rsyslog into two separate service instances.
>> > >>
>> > >> Service 1 would do *only one thing*, read imjournal and write to
>> > >> file(s). This service would *not* have the network.target dependency.
>> > >> Service 2 would do *everything else*, including reading the file(s)
>> > >> output from above (which survive the reboot) and sending the events within
>> > >> to a network destination. This service would have the network.target
>> > >> dependency so as to be able to deliver reliably.
>> > >>
>> > >> Regards,
>> > >>
>> > >>
>> > >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
>> > >> rsyslog@lists.adiscon.com> wrote:
>> > >>
>> > >> Recently I came across an observation where we are not able to capture
>> > >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
>> > >> environments, systemd is responsible for starting the rsyslog service. A
>> > >> service can have multiple dependencies, which influence how early or how
>> > >> late rsyslog is started or stopped. Many years ago, we added dependency for
>> > >> the network.target and network-online.target into the service file [1]. If
>> > >> rsyslog started before establishing network access, it would be unable to
>> > >> transmit messages to remote destinations during that period, resulting in
>> > >> the generation of misleading information about the unavailability of
>> > >> certain remote targets (e.g. not able to resolve hostnames).
>> > >> However, this approach results in a significant tradeoff. While it prevents
>> > >> misleading unavailability messages during network setup and shutdown, it
>> > >> also causes rsyslog to *exit* *early* during shutdown, leading to missed
>> > >> logs regarding the graceful termination of other programs. This limitation
>> > >> extends to system reboots as well. Thus, while addressing one issue, the
>> > >> current service configuration introduces another.
>> > >> By default, we retrieve shutdown events from the journal using the
>> > >> imjournal module. Journal log data is stored in memory so after shutdown,
>> > >> logs are not preserved.
>> > >>
>> > >> Has someone faced this problem? Are there any known workarounds?
>> > >>
>> > >> [1]
>> > >>
>> > >> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
>> > >> _______________________________________________
>> > >> rsyslog mailing list
>> > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > >> http://www.rsyslog.com/professional-services/
>> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > >> DON'T LIKE THAT.
>> > >>
>> > >>
>> > >>
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 20, 2024, 12:58 PM

Post #12 of 13 (129 views)

Permalink

The jousnal is storing them somewhere anyway (in ram if nothing else), that's a
'feature' of journald.

you can set how much space you allocate to journald for it's fixed storage and
so can set it small enough to not be an issue.

David Lang

On Wed, 20 Mar 2024, Attila Lakatos via rsyslog wrote:

> Hello Peter,
>
> I think that would be the best solution from rsyslog point of view.
> However, this would mean that logs would be stored in both
> /var/log/{messages|secure|...} and the journal.
> Ideally, it would be better to have them only in one place.
>
> Thanks,
> Attila
>
> On Tue, Mar 19, 2024 at 4:03?PM Peter Portante via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
>> Attila, any reason you can't just use persistent journald? That is
>> what we did to solve the lost shutdown and crash logs. -Peter
>>
>> On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
>> <rsyslog@lists.adiscon.com> wrote:
>> >
>> > imjournal uses the journal api to fetch the logs (fetching them in
>> > near-real-time), journald keeps files internally to support it.
>> >
>> > David Lang
>> >
>> > On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
>> >
>> > > The solution is clean to me, however I think this could be a
>> bottleneck for
>> > > busy systems. Also, this would mean that I need to maintain a copy of
>> journal
>> > > logs in one or more files.
>> > >
>> > > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com>
>> wrote:
>> > >
>> > >> You could split rsyslog into two separate service instances.
>> > >>
>> > >> Service 1 would do *only one thing*, read imjournal and write to
>> > >> file(s). This service would *not* have the network.target dependency.
>> > >> Service 2 would do *everything else*, including reading the file(s)
>> > >> output from above (which survive the reboot) and sending the events
>> within
>> > >> to a network destination. This service would have the network.target
>> > >> dependency so as to be able to deliver reliably.
>> > >>
>> > >> Regards,
>> > >>
>> > >>
>> > >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
>> > >> rsyslog@lists.adiscon.com> wrote:
>> > >>
>> > >> Recently I came across an observation where we are not able to capture
>> > >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
>> > >> environments, systemd is responsible for starting the rsyslog
>> service. A
>> > >> service can have multiple dependencies, which influence how early or
>> how
>> > >> late rsyslog is started or stopped. Many years ago, we added
>> dependency for
>> > >> the network.target and network-online.target into the service file
>> [1]. If
>> > >> rsyslog started before establishing network access, it would be
>> unable to
>> > >> transmit messages to remote destinations during that period,
>> resulting in
>> > >> the generation of misleading information about the unavailability of
>> > >> certain remote targets (e.g. not able to resolve hostnames).
>> > >> However, this approach results in a significant tradeoff. While it
>> prevents
>> > >> misleading unavailability messages during network setup and shutdown,
>> it
>> > >> also causes rsyslog to *exit* *early* during shutdown, leading to
>> missed
>> > >> logs regarding the graceful termination of other programs. This
>> limitation
>> > >> extends to system reboots as well. Thus, while addressing one issue,
>> the
>> > >> current service configuration introduces another.
>> > >> By default, we retrieve shutdown events from the journal using the
>> > >> imjournal module. Journal log data is stored in memory so after
>> shutdown,
>> > >> logs are not preserved.
>> > >>
>> > >> Has someone faced this problem? Are there any known workarounds?
>> > >>
>> > >> [1]
>> > >>
>> > >>
>> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
>> > >> _______________________________________________
>> > >> rsyslog mailing list
>> > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > >> http://www.rsyslog.com/professional-services/
>> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > >> DON'T LIKE THAT.
>> > >>
>> > >>
>> > >>
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Capturing shutdown logs [ In reply to ]

rsyslog at lists

Mar 20, 2024, 1:00 PM

Post #13 of 13 (129 views)

Permalink

when you use imjournal with rsyslog, journald is storing the logs in it's
database, then rsyslog is periodically querying the database for new logs. that
database can be all in ram, or partially on disk.

David Lang

On Wed, 20 Mar 2024, David Lang via rsyslog wrote:

> Date: Wed, 20 Mar 2024 12:58:52 -0700 (PDT)
> From: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> To: Attila Lakatos via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] Capturing shutdown logs
>
> The jousnal is storing them somewhere anyway (in ram if nothing else), that's
> a
> 'feature' of journald.
>
> you can set how much space you allocate to journald for it's fixed storage
> and
> so can set it small enough to not be an issue.
>
> David Lang
>
> On Wed, 20 Mar 2024, Attila Lakatos via rsyslog wrote:
>
>> Hello Peter,
>>
>> I think that would be the best solution from rsyslog point of view.
>> However, this would mean that logs would be stored in both
>> /var/log/{messages|secure|...} and the journal.
>> Ideally, it would be better to have them only in one place.
>>
>> Thanks,
>> Attila
>>
>> On Tue, Mar 19, 2024 at 4:03?PM Peter Portante via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>>
>>> Attila, any reason you can't just use persistent journald? That is
>>> what we did to solve the lost shutdown and crash logs. -Peter
>>>
>>> On Fri, Mar 15, 2024 at 12:31?PM David Lang via rsyslog
>>> <rsyslog@lists.adiscon.com> wrote:
>>> >
>>> > imjournal uses the journal api to fetch the logs (fetching them in
>>> > near-real-time), journald keeps files internally to support it.
>>> >
>>> > David Lang
>>> >
>>> > On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote:
>>> >
>>> > > The solution is clean to me, however I think this could be a
>>> bottleneck for
>>> > > busy systems. Also, this would mean that I need to maintain a copy of
>>> journal
>>> > > logs in one or more files.
>>> > >
>>> > > On Wed, Mar 13, 2024 at 2:53?PM John Chivian <jchivian@chivian.com>
>>> wrote:
>>> > >
>>> > >> You could split rsyslog into two separate service instances.
>>> > >>
>>> > >> Service 1 would do *only one thing*, read imjournal and write to
>>> > >> file(s). This service would *not* have the network.target
> dependency.
>>> > >> Service 2 would do *everything else*, including reading the file(s)
>>> > >> output from above (which survive the reboot) and sending the events
>>> within
>>> > >> to a network destination. This service would have the network.target
>>> > >> dependency so as to be able to deliver reliably.
>>> > >>
>>> > >> Regards,
>>> > >>
>>> > >>
>>> > >> On Mar 13, 2024, at 07:49, Attila Lakatos via rsyslog <
>>> > >> rsyslog@lists.adiscon.com> wrote:
>>> > >>
>>> > >> Recently I came across an observation where we are not able to
> capture
>>> > >> normal reboot/shutdown logs on Fedora/RHEL distributions. In these
>>> > >> environments, systemd is responsible for starting the rsyslog
>>> service. A
>>> > >> service can have multiple dependencies, which influence how early or
>>> how
>>> > >> late rsyslog is started or stopped. Many years ago, we added
>>> dependency for
>>> > >> the network.target and network-online.target into the service file
>>> [1]. If
>>> > >> rsyslog started before establishing network access, it would be
>>> unable to
>>> > >> transmit messages to remote destinations during that period,
>>> resulting in
>>> > >> the generation of misleading information about the unavailability of
>>> > >> certain remote targets (e.g. not able to resolve hostnames).
>>> > >> However, this approach results in a significant tradeoff. While it
>>> prevents
>>> > >> misleading unavailability messages during network setup and shutdown,
>>> it
>>> > >> also causes rsyslog to *exit* *early* during shutdown, leading to
>>> missed
>>> > >> logs regarding the graceful termination of other programs. This
>>> limitation
>>> > >> extends to system reboots as well. Thus, while addressing one issue,
>>> the
>>> > >> current service configuration introduces another.
>>> > >> By default, we retrieve shutdown events from the journal using the
>>> > >> imjournal module. Journal log data is stored in memory so after
>>> shutdown,
>>> > >> logs are not preserved.
>>> > >>
>>> > >> Has someone faced this problem? Are there any known workarounds?
>>> > >>
>>> > >> [1]
>>> > >>
>>> > >>
>>>
> https://github.com/deoren/rsyslog-examples/blob/master/etc/systemd/system/rsyslog.service.d/10-wait-on-network.conf
>>> > >> _______________________________________________
>>> > >> rsyslog mailing list
>>> > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> > >> http://www.rsyslog.com/professional-services/
>>> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad
>>> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
>>> > >> DON'T LIKE THAT.
>>> > >>
>>> > >>
>>> > >>
>>> > > _______________________________________________
>>> > > rsyslog mailing list
>>> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> > > http://www.rsyslog.com/professional-services/
>>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>> you DON'T LIKE THAT.
>>> > _______________________________________________
>>> > rsyslog mailing list
>>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> > http://www.rsyslog.com/professional-services/
>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.