Mailing List Archive

action queues
Hello,

i have several questions related to legacy configuration parameters:
* If i use ActionExecOnlyWhenPreviousIsSuspended, will the action queue parameters kept for the next action, if the previous is stuck (e.g. due to an outage of a TCP syslog server):
$ActionQueueFileName agg_backlog
$ActionQueueMaxDiskSpace 128m
$ActionQueueSize 100000

*.* @@192.168.0.100:513;SyslogForwarderTemplate
* Is there an parameter to copy the action queue settings from the previous action and do not do the reset of the queue settings? i do not want to define Queue's multiple times
* If i define action queue parameters, use Rainerscript and define an action in between and then define a legacy action afterwards, will the legacy action parameters defined on the top get reset after the Rainerscript action?
$ActionQueueFileName agg_backlog
$ActionQueueMaxDiskSpace 128m
$ActionQueueSize 100000

*.* @@192.168.0.100:513;SyslogForwarderTemplate

action(type="omrelp" target="..." port="..." timeout="90" rebindInterval="1000" template="SyslogForwarderTemplate")
* If a direct action queue (by default) is used as a queue for an action and the action will stall due to an unavailability of the destination syslog server, does this mean that the chain of actions will stall and some kind of backpressure will get triggered or does it mean that the message will get discarded? This is not really clear according to the documentation.
https://www.rsyslog.com/doc/v8-stable/concepts/queues.html (https://www.rsyslog.com/doc/v8-stable/concepts/queues.html)
Direct queues are non-queuing queues. A queue in direct mode does neither queue nor buffer any of the queue elements but rather passes the element directly (and immediately) from the producer to the consumer
Any help would be appreciated.

Best Regards
Hajo
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: action queues [ In reply to ]
On Mon, 2 Sep 2019, info--- via rsyslog wrote:

> Hello,
>
> i have several questions related to legacy configuration parameters:
> * If i use ActionExecOnlyWhenPreviousIsSuspended, will the action queue parameters kept for the next action, if the previous is stuck (e.g. due to an outage of a TCP syslog server):
> $ActionQueueFileName agg_backlog
> $ActionQueueMaxDiskSpace 128m
> $ActionQueueSize 100000

the answer is (mostly) no, action queues only affect the next item. But there
are parameters that affect all actions after that.

This is one of the big reasons why the new syntax was created, so that it's far
less confusing what applies to what.

> *.* @@192.168.0.100:513;SyslogForwarderTemplate
> * Is there an parameter to copy the action queue settings from the previous action and do not do the reset of the queue settings? i do not want to define Queue's multiple times
> * If i define action queue parameters, use Rainerscript and define an action in between and then define a legacy action afterwards, will the legacy action parameters defined on the top get reset after the Rainerscript action?
> $ActionQueueFileName agg_backlog
> $ActionQueueMaxDiskSpace 128m
> $ActionQueueSize 100000

no, just don't do action queues with the obsolete legacy syntax


> *.* @@192.168.0.100:513;SyslogForwarderTemplate
>
> action(type="omrelp" target="..." port="..." timeout="90" rebindInterval="1000" template="SyslogForwarderTemplate")

> * If a direct action queue (by default) is used as a queue for an action
> and the action will stall due to an unavailability of the destination syslog
> server, does this mean that the chain of actions will stall and some kind of
> backpressure will get triggered or does it mean that the message will get
> discarded? This is not really clear according to the documentation.

it will either stall or discard the message depending on the retries and queue
settings

> https://www.rsyslog.com/doc/v8-stable/concepts/queues.html (https://www.rsyslog.com/doc/v8-stable/concepts/queues.html)
> Direct queues are non-queuing queues. A queue in direct mode does neither
> queue nor buffer any of the queue elements but rather passes the element
> directly (and immediately) from the producer to the consumer

remember that the main queue is there, so your message is almost always going to
be from a queue to a worker thread that's going through your config

it may help you to know that you can put a queue on a ruleset as well as an
action, so you can group multiple actions together and have one queue for them.

note that when you use ActionExecOnlyWhenPreviousIsSuspended, putting the
message in an action queue counts as success

you may want to look at
https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html

> Any help would be appreciated.

it's not clear what you are trying to do, but when you start working with
queues, stop using the obsolete legacy format for them and specify them in your
action() or ruleset() statements and remember that putting a messagee in a queue
counts as success

watch your batch sizes, contention on queues can slow your system down a lot.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: action queues [ In reply to ]
Hello David,

thanks for your answer. I'm forced to use legacy syntax, since the module is written with legacy syntax: https://github.com/cloudfoundry/syslog-release/tree/master/jobs/syslog_forwarder/templates

Since i need to ensure log loss and i have boundary conditions, may i ask to be precise regarding
your answer "don't do action queues with the obsolete legacy syntax"

>> If i define action queue parameters, use Rainerscript and define an action in between and then
>> define a legacy action afterwards, will the legacy action parameters defined on the top get reset
>> after the Rainerscript action?

Does this mean the legacy action queue parameters will be kept across a Rainerscript action?

Best Regards
Hajo


2. September 2019 11:12, "David Lang" <david@lang.hm> schrieb:

> On Mon, 2 Sep 2019, info--- via rsyslog wrote:
>
>> Hello,
>>
>> i have several questions related to legacy configuration parameters:
>> * If i use ActionExecOnlyWhenPreviousIsSuspended, will the action queue parameters kept for the
>> next action, if the previous is stuck (e.g. due to an outage of a TCP syslog server):
>> $ActionQueueFileName agg_backlog
>> $ActionQueueMaxDiskSpace 128m
>> $ActionQueueSize 100000
>
> the answer is (mostly) no, action queues only affect the next item. But there are parameters that
> affect all actions after that.
>
> This is one of the big reasons why the new syntax was created, so that it's far less confusing what
> applies to what.
>
>> *.* @@192.168.0.100:513;SyslogForwarderTemplate
>> * Is there an parameter to copy the action queue settings from the previous action and do not do
>> the reset of the queue settings? i do not want to define Queue's multiple times
>> * If i define action queue parameters, use Rainerscript and define an action in between and then
>> define a legacy action afterwards, will the legacy action parameters defined on the top get reset
>> after the Rainerscript action?
>> $ActionQueueFileName agg_backlog
>> $ActionQueueMaxDiskSpace 128m
>> $ActionQueueSize 100000
>
> no, just don't do action queues with the obsolete legacy syntax
>
>> *.* @@192.168.0.100:513;SyslogForwarderTemplate
>>
>> action(type="omrelp" target="..." port="..." timeout="90" rebindInterval="1000"
>> template="SyslogForwarderTemplate")
>>
>> * If a direct action queue (by default) is used as a queue for an action > and the action will
>> stall due to an unavailability of the destination syslog > server, does this mean that the chain of
>> actions will stall and some kind of > backpressure will get triggered or does it mean that the
>> message will get > discarded? This is not really clear according to the documentation.
>
> it will either stall or discard the message depending on the retries and queue settings
>
>> https://www.rsyslog.com/doc/v8-stable/concepts/queues.html
>> (https://www.rsyslog.com/doc/v8-stable/concepts/queues.html)
>> Direct queues are non-queuing queues. A queue in direct mode does neither > queue nor buffer any of
>> the queue elements but rather passes the element > directly (and immediately) from the producer to
>> the consumer
>
> remember that the main queue is there, so your message is almost always going to be from a queue to
> a worker thread that's going through your config
>
> it may help you to know that you can put a queue on a ruleset as well as an action, so you can
> group multiple actions together and have one queue for them.
>
> note that when you use ActionExecOnlyWhenPreviousIsSuspended, putting the message in an action
> queue counts as success
>
> you may want to look at https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html
>
>> Any help would be appreciated.
>
> it's not clear what you are trying to do, but when you start working with queues, stop using the
> obsolete legacy format for them and specify them in your action() or ruleset() statements and
> remember that putting a messagee in a queue counts as success
>
> watch your batch sizes, contention on queues can slow your system down a lot.
>
> David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: action queues [ In reply to ]
On Mon, 2 Sep 2019, info@kliemeck.de wrote:

> Hello David,
>
> thanks for your answer. I'm forced to use legacy syntax, since the module is written with legacy syntax: https://github.com/cloudfoundry/syslog-release/tree/master/jobs/syslog_forwarder/templates
>
> Since i need to ensure log loss and i have boundary conditions, may i ask to be precise regarding
> your answer "don't do action queues with the obsolete legacy syntax"
>
>>> If i define action queue parameters, use Rainerscript and define an action in between and then
>>> define a legacy action afterwards, will the legacy action parameters defined on the top get reset
>>> after the Rainerscript action?
>
> Does this mean the legacy action queue parameters will be kept across a Rainerscript action?

I suspect that they would, but since they won't across a regular action, I don't
see what the value would be defining them and then putting in an action()
statement. It seems like it adds potential confusion with no actual benefit.

on a current version you can use the -o option to have rsyslog spit out what it
understands the config to be. It will answer a lot of your questions about how
it interprets the config

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: action queues [ In reply to ]
El lun., 2 sept. 2019 a las 12:03, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> On Mon, 2 Sep 2019, info@kliemeck.de wrote:
>
> > Hello David,
> >
> > thanks for your answer. I'm forced to use legacy syntax, since the module is written with legacy syntax: https://github.com/cloudfoundry/syslog-release/tree/master/jobs/syslog_forwarder/templates
> >
> > Since i need to ensure log loss and i have boundary conditions, may i ask to be precise regarding
> > your answer "don't do action queues with the obsolete legacy syntax"
> >
> >>> If i define action queue parameters, use Rainerscript and define an action in between and then
> >>> define a legacy action afterwards, will the legacy action parameters defined on the top get reset
> >>> after the Rainerscript action?
> >
> > Does this mean the legacy action queue parameters will be kept across a Rainerscript action?
>
> I suspect that they would, but since they won't across a regular action, I don't
> see what the value would be defining them and then putting in an action()
> statement. It seems like it adds potential confusion with no actual benefit.

They'll probably apply to an innocent action who has the misfortune of
being in the wrong place, possibly inside an include ;-) It's an
extremely bad idea to split the action parameters and the actual
action in obsolete legacy config. sooner or later leads to disaster.

>
> on a current version you can use the -o option to have rsyslog spit out what it
> understands the config to be. It will answer a lot of your questions about how
> it interprets the config

Be careful: the current version just spits out the consolidated config
text. It's not yet internal processing. For the latter I need to craft
new internal interfaces. Unfortunately I am too busy again for that.
Was on short notice asked to hold an 8hr lecture on syslog and I am
right now preparing material....

Rainer
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: action queues [ In reply to ]
On Mon, 2 Sep 2019, Rainer Gerhards wrote:

> Unfortunately I am too busy again for that. Was on short notice asked to hold
> an 8hr lecture on syslog and I am right now preparing material....

sounds very interesting, any chance of a recording?

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.