Mailing List Archive

On Thu, 16 Feb 2023, Dimi Onobodies via rsyslog wrote:

> 1. According to documentation, "By default, rulesets do not have their own queue." . Does that mean that indeed there is a queue, but in Direct mode/type?

No, this means that when running the ruleset, it's pulling from the main queue.

I try to explain more below, please ask followup questions if I haven't
clarified things enough.

> 2. Is the main_Q (default ruleset) a linkedList/FixedArray type of queue?

No, it's a memory queue

> 3. When there is no queue, does the input module delivers logs directly to an action queue?

by default, the main queue is a memory queue. Input modules deliver logs to this
main queue.

you can define an input module to deliver to a ruleset, and if that ruleset has
it's own queue defined, it will deliver to that queue. If that queue type is
direct, it would deliver directly to the actions.

action queues and ruleset queues mean that instead of rsyslog waiting for that
ruleset/action to complete before moving on to the next action, they make a copy
of the log entry to the new queue and continue processing actions in the ruleset
they are working in (there is a default ruleset that every action is in by
default)


There is a writeup of how queues work that uses the example of a road with
turnoffs. Each turnoff is either a driveway (an action) or a side road with it's
own turnoffs (a ruleset). As each car (log entry) comes down the road, it makes
a turn into each turnoff does it's stuff there, then re-enters the main road. If
the action is busy (there is a car there), then other cars stop on the main road
with their blinkers on, and all log processing stalls.

a queue on an action/ruleset is like adding a magic turning lane on the main
road, when a car reaches the turning lane, it gets cloned, with one copy going
in the turning lane and one continuing on past it to the next turnoff

when the turning lane (the action/ruleset queue) is full, cars behind it aren't
able to hit the magic start of the turing lane and processing again stalls



to make things a little more complicated in practice, we have the ability to
process things in batches, so instead of individual cars turning into a
driveway, you have a traffic light that can let multiple cars turn at once. It
doesn't wait for an optimal number of cars to arrive, it just lets those that
are waiting in the turn lane (up to batch size) all turn at once.

This is for performance:
1. some actions can be much faster if they can process multiple logs at once
(database inserts where there's a large per-transaction overhead, so sending
multiple logs at once is almost the same time as sending a single log. In some
cases you can process 1000 logs as a batch in the time it would take to process
2 logs separately

2. since the queue is shared between threads (input thread and multiple worker
threads), we have to do locking to prevent collisions as logs enter and leave
the queue. This locking creates a significant overhead when you have lots of
threads involved, and processing multiple logs at once is a big performance win.


> For example. I have imtcp bound to a specific ruleset named "incoming". There
> are no parameters specified apart from ruleset name. Inside the ruleset, there
> is only one omfwd action with DA queue. Is the imtcp module passes the logs to
> omfwd queue directly?

no, the input module delivers to the main queue and then the 'incoming' ruleset
is executed on those logs instead of the default ruleset (it is a bit ugly under
the covers to have some things on the main queue acted on by the default ruleset
and some by the 'incoming' ruleset, but for ease of use when the highest
preformance isn't needed, it makes it easy.

> 4. In Scenario No3 above, any incoming logs from imtcp would go directly to omfwd DA queue without having to go through the main_Q first, correct?

no.

> 5. In my impstats data, i can see two records for a disk assisted queue named forward_cntr
>
> forward_cntr queue core_queue
> forward_cntr queue[DA] core_queue
>
> Does the first metric refer to the LinkedList/memory queue and the second one to the disk queue?

yes.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.