Mailing List Archive

rsyslog and config reload
Hi All,

This is Shirisha Dasari from Broadcom. We were trying to understand what
may be needed to support rsyslog config reload without a restart.

In this regard, we came across this related write-up that outlines the
possible steps to achieve the same (
https://rainer.gerhards.net/2011/04/rsyslog-config-reload-random-thoughts.html).
From the rsyslog source code, it does seem like the write-up holds true
even today since support for dynamic loading of config without a restart
still seems missing in rsyslog.

From what we understand, apart from bringing in a notion of running-config
and intended-config (a delta config change), multiple changes may be needed
in each supported module to allow it to accept this delta configuration as
well. Also, does such a change offer any real benefits over the current
implementation is a topic we are trying to understand.

I did a quick search in the forum and found
http://rsyslog-users.1305293.n2.nabble.com/making-config-changes-to-a-running-rsyslog-td7591667.html
that discusses something similar. Please share thoughts on this topic and
how this can be achieved.

--
Thanks & Regards,
Shirisha.

--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
Re: rsyslog and config reload [ In reply to ]
Currently rsyslog does not support reloading the config without a full stop and
start.

A large part of this is that the config loads code that changes the internals of
rsyslog, and rsyslog has not gone to the effort to make it so that this code can
be removed cleanly.

There has been some discussion about having some limited portions able to be
changed (add/remove inputs to support containers for example, but there has not
been enough interest to move this up to the top of the todo list.

The advantage of reloading the config rather than restarting rsyslog is that if
you are using UDP to deliver logs to rsyslog, during the restart some logs will
be lost that _may_ not be lost in a reload (depending on system queue sizes and
how long the reload takes, etc)

In prctice, this has not been a very significant problem (outsie of issue
supporting containers as I allude to above)

David Lang

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog and config reload [ In reply to ]
Thanks David for the response. Our use-case involves imudp (UDP reception of
syslog messages), imuxsock and imklog. The major config changes expected are
w.r.t remote server logging details.

We are evaluating if there is any way to alleviate the possible loss of
messages when rsyslog is restarted due to a change in the configuration
file. We were evaluating if there is a chance of keeping the input modules
running to avoid this loss while parts of rsyslog may restart due to a new
configuration. I understand this in itself may be a herculean task
considering the interdependencies and config being tied together.

Could you please provide inputs on the above approach? Also, it looks like
there was a text API based interface you had proposed to allow some basic
runtime-config changes to rsyslog
(http://rsyslog-users.1305293.n2.nabble.com/making-config-changes-to-a-running-rsyslog-td7591667.html).
Could you please elaborate on the same as well?



--
Sent from: http://rsyslog-users.1305293.n2.nabble.com/
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog and config reload [ In reply to ]
It's doable, but it is a (very) lot of work. I myself am not
interested in this work, and no company ever considered it important
enough to provide the (considerable) amount of sponsorship needed to
get this going (or implement it itself and contribute).

Sorry for being blunt, but that's simply the truth and will probably
save you a lot of research.

Rainer

El mar, 5 ene 2021 a las 8:40, Shirisha Dasari via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Thanks David for the response. Our use-case involves imudp (UDP reception of
> syslog messages), imuxsock and imklog. The major config changes expected are
> w.r.t remote server logging details.
>
> We are evaluating if there is any way to alleviate the possible loss of
> messages when rsyslog is restarted due to a change in the configuration
> file. We were evaluating if there is a chance of keeping the input modules
> running to avoid this loss while parts of rsyslog may restart due to a new
> configuration. I understand this in itself may be a herculean task
> considering the interdependencies and config being tied together.
>
> Could you please provide inputs on the above approach? Also, it looks like
> there was a text API based interface you had proposed to allow some basic
> runtime-config changes to rsyslog
> (http://rsyslog-users.1305293.n2.nabble.com/making-config-changes-to-a-running-rsyslog-td7591667.html).
> Could you please elaborate on the same as well?
>
>
>
> --
> Sent from: http://rsyslog-users.1305293.n2.nabble.com/
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog and config reload [ In reply to ]
I can suggest a few things to work this around.

At first you should clearly understand log messages can (and will) be lost.
It's ok. rsyslog (and any log collector) is not a database.

Then I guess you're using UDP on localhost to deliver messages from your
application to the rsyslog daemon.

I see 2 solutions there.

1. Switch to datagram unix socket. It'll block on write though so you
should adopt your application. But if the DGRAM socket is created by
systemd e.g. then OS will be able to hold a few DGRAM packets in the OS
queue during rsyslog daemon restart.

2. Use 2 rsyslog daemons running on localhost. First one is just to accept
messages on a UDP port and deliver them to another rsyslog instance via TCP
or RELP with a queue attached. Second rsyslog instance will do complex
things. So in most cases you will change 2nd instance configuration
changes. As you have a queue on 1st instance output you can restart 2nd
instance w/o losing messages here.

I'd even suggest you use the distro-bundled rsyslog version as 1st instance
and the latest rsyslog version in a container as 2nd instance.


On Tue, 5 Jan 2021 at 15:41, Shirisha Dasari via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Thanks David for the response. Our use-case involves imudp (UDP reception
> of
> syslog messages), imuxsock and imklog. The major config changes expected
> are
> w.r.t remote server logging details.
>
> We are evaluating if there is any way to alleviate the possible loss of
> messages when rsyslog is restarted due to a change in the configuration
> file. We were evaluating if there is a chance of keeping the input modules
> running to avoid this loss while parts of rsyslog may restart due to a new
> configuration. I understand this in itself may be a herculean task
> considering the interdependencies and config being tied together.
>
> Could you please provide inputs on the above approach? Also, it looks like
> there was a text API based interface you had proposed to allow some basic
> runtime-config changes to rsyslog
> (
> http://rsyslog-users.1305293.n2.nabble.com/making-config-changes-to-a-running-rsyslog-td7591667.html).
>
> Could you please elaborate on the same as well?
>
>
>
> --
> Sent from: http://rsyslog-users.1305293.n2.nabble.com/
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>


--
Yury Bushmelev
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog and config reload [ In reply to ]
Thanks Rainer and Yury for the responses.

I was looking at another option for this message loss issue. I see that
rsyslog supports systemd socket activation for the imuxsock module where-in
systemd spawns the socket and hands over the fd to imuxsock. In the case of
an rsyslog restart, the socket is still available and hence no messages are
lost.

rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd
3) from systemd. [v8.1901.0]

Can this be extended to the imudp module? We could have systemd create the
UDP socket too and pass the fd to imudp like imuxsock. I do understand that
the socket parameters in this case are user-defined.







--
Sent from: http://rsyslog-users.1305293.n2.nabble.com/
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog and config reload [ In reply to ]
note that even if the UDP socket does not get closed, if you go too long without
reading from it the OS buffer will fill up and messages will be lost.

David Lang

On Tue, 5 Jan 2021, Shirisha Dasari via rsyslog wrote:

> Date: Tue, 5 Jan 2021 04:22:22 -0700 (MST)
> From: Shirisha Dasari via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Shirisha Dasari <shirisha.dasari@broadcom.com>
> Subject: Re: [rsyslog] rsyslog and config reload
>
> Thanks Rainer and Yury for the responses.
>
> I was looking at another option for this message loss issue. I see that
> rsyslog supports systemd socket activation for the imuxsock module where-in
> systemd spawns the socket and hands over the fd to imuxsock. In the case of
> an rsyslog restart, the socket is still available and hence no messages are
> lost.
>
> rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd
> 3) from systemd. [v8.1901.0]
>
> Can this be extended to the imudp module? We could have systemd create the
> UDP socket too and pass the fd to imudp like imuxsock. I do understand that
> the socket parameters in this case are user-defined.
>
>
>
>
>
>
>
> --
> Sent from: http://rsyslog-users.1305293.n2.nabble.com/
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.