Mailing List Archive

What happens when the main message queue is full ?
Hello,

My system is a Debian 11 with rsyslog 8.2102.0-2+deb11u1.

It is said in https://www.rsyslog.com/doc/v8-stable/concepts/queues.html#filled-up-queues :

"For example, throttling the local log socket too long would cause the system at whole come to a standstill. To prevent this, rsyslogd times out after a configured period (”$<object>QueueTimeoutEnqueue“, specified in milliseconds) if no space becomes available. As a last resort, it then discards the newly arrived message."

And I read in the warning section of https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-working_with_queues_in_rsyslog :

"If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database."

I find these two statements contradictory because according the first one, when the main message queue is full, rsyslog will enventually drop newly arrived messages when according to the second one "the inputs block until it is no longer full".

I would like to be sure that the processes loging to rsyslog via unix socket won't be blocked in the cas my main message queue is full.

thanks,
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: What happens when the main message queue is full ? [ In reply to ]
the inputs block, but eventually many inputs will drop messages (think network
connections)

you can configure how to drop messages (look at the watermark configs)

In general, when the queue is full, anything attempting to put messages into the
queue blocks, if that thing is reading messages from a source that can wait, the
messages wait, if the source can't wait, the message gets dropped.

David Lang

On Mon, 31 Jul 2023, Raphaël Laguerre via rsyslog wrote:

> Date: Mon, 31 Jul 2023 14:15:59 +0200
> From: Raphaël Laguerre via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Raphaël Laguerre <rlaguerre@easter-eggs.com>
> Subject: [rsyslog] What happens when the main message queue is full ?
>
> Hello,
>
> My system is a Debian 11 with rsyslog 8.2102.0-2+deb11u1.
>
> It is said in https://www.rsyslog.com/doc/v8-stable/concepts/queues.html#filled-up-queues :
>
> "For example, throttling the local log socket too long would cause the system at whole come to a standstill. To prevent this, rsyslogd times out after a configured period (”$<object>QueueTimeoutEnqueue“, specified in milliseconds) if no space becomes available. As a last resort, it then discards the newly arrived message."
>
> And I read in the warning section of https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-working_with_queues_in_rsyslog :
>
> "If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database."
>
> I find these two statements contradictory because according the first one, when the main message queue is full, rsyslog will enventually drop newly arrived messages when according to the second one "the inputs block until it is no longer full".
>
> I would like to be sure that the processes loging to rsyslog via unix socket won't be blocked in the cas my main message queue is full.
>
> thanks,
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: What happens when the main message queue is full ? [ In reply to ]
Well, it depends.

For a busy system, the default timeout in the main queue is way to
long (I think I have shortened it recently, but still). This means
while the message ultimately gets deleted, the system can become
sluggish to a point where it looks totally unresponsive.

HOWEVER, you can change the default via config, and you can also set
the timeout to zero, which means "immediately". I suggest this for
many scenarios. Obviously, it causes loss of logs, but usually this is
preferred over loss of function.

Just be aware that attackers may use this approach to hide their traces.

HTH
Rainer

El lun, 31 jul 2023 a las 15:13, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> the inputs block, but eventually many inputs will drop messages (think network
> connections)
>
> you can configure how to drop messages (look at the watermark configs)
>
> In general, when the queue is full, anything attempting to put messages into the
> queue blocks, if that thing is reading messages from a source that can wait, the
> messages wait, if the source can't wait, the message gets dropped.
>
> David Lang
>
> On Mon, 31 Jul 2023, Raphaël Laguerre via rsyslog wrote:
>
> > Date: Mon, 31 Jul 2023 14:15:59 +0200
> > From: Raphaël Laguerre via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: Raphaël Laguerre <rlaguerre@easter-eggs.com>
> > Subject: [rsyslog] What happens when the main message queue is full ?
> >
> > Hello,
> >
> > My system is a Debian 11 with rsyslog 8.2102.0-2+deb11u1.
> >
> > It is said in https://www.rsyslog.com/doc/v8-stable/concepts/queues.html#filled-up-queues :
> >
> > "For example, throttling the local log socket too long would cause the system at whole come to a standstill. To prevent this, rsyslogd times out after a configured period (”$<object>QueueTimeoutEnqueue“, specified in milliseconds) if no space becomes available. As a last resort, it then discards the newly arrived message."
> >
> > And I read in the warning section of https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-working_with_queues_in_rsyslog :
> >
> > "If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database."
> >
> > I find these two statements contradictory because according the first one, when the main message queue is full, rsyslog will enventually drop newly arrived messages when according to the second one "the inputs block until it is no longer full".
> >
> > I would like to be sure that the processes loging to rsyslog via unix socket won't be blocked in the cas my main message queue is full.
> >
> > thanks,
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: What happens when the main message queue is full ? [ In reply to ]
Thank you for your answers and all your work !

Le Mon, Jul 31, 2023 at 05:05:42PM +0200, Rainer Gerhards via rsyslog a écrit :
> Well, it depends.
>
> For a busy system, the default timeout in the main queue is way to
> long (I think I have shortened it recently, but still). This means
> while the message ultimately gets deleted, the system can become
> sluggish to a point where it looks totally unresponsive.
>
> HOWEVER, you can change the default via config, and you can also set
> the timeout to zero, which means "immediately". I suggest this for
> many scenarios. Obviously, it causes loss of logs, but usually this is
> preferred over loss of function.
>
> Just be aware that attackers may use this approach to hide their traces.
>
> HTH
> Rainer
>
> El lun, 31 jul 2023 a las 15:13, David Lang via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > the inputs block, but eventually many inputs will drop messages (think network
> > connections)
> >
> > you can configure how to drop messages (look at the watermark configs)
> >
> > In general, when the queue is full, anything attempting to put messages into the
> > queue blocks, if that thing is reading messages from a source that can wait, the
> > messages wait, if the source can't wait, the message gets dropped.
> >
> > David Lang
> >
> > On Mon, 31 Jul 2023, Raphaël Laguerre via rsyslog wrote:
> >
> > > Date: Mon, 31 Jul 2023 14:15:59 +0200
> > > From: Raphaël Laguerre via rsyslog <rsyslog@lists.adiscon.com>
> > > To: rsyslog@lists.adiscon.com
> > > Cc: Raphaël Laguerre <rlaguerre@easter-eggs.com>
> > > Subject: [rsyslog] What happens when the main message queue is full ?
> > >
> > > Hello,
> > >
> > > My system is a Debian 11 with rsyslog 8.2102.0-2+deb11u1.
> > >
> > > It is said in https://www.rsyslog.com/doc/v8-stable/concepts/queues.html#filled-up-queues :
> > >
> > > "For example, throttling the local log socket too long would cause the system at whole come to a standstill. To prevent this, rsyslogd times out after a configured period (”$<object>QueueTimeoutEnqueue“, specified in milliseconds) if no space becomes available. As a last resort, it then discards the newly arrived message."
> > >
> > > And I read in the warning section of https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-working_with_queues_in_rsyslog :
> > >
> > > "If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database."
> > >
> > > I find these two statements contradictory because according the first one, when the main message queue is full, rsyslog will enventually drop newly arrived messages when according to the second one "the inputs block until it is no longer full".
> > >
> > > I would like to be sure that the processes loging to rsyslog via unix socket won't be blocked in the cas my main message queue is full.
> > >
> > > thanks,
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.