Mailing List Archive

Processing one syslog message in multiple rulesets in parallel
Hi there,

I am looking into the following configuration but a bit unsure about
some implementation details...

Several originators send syslog messages to a rsyslog relay which shall
do the following tasks:

1) Write every log message to separate files based on hostnames.

2) Apply a few filters and send remaining syslog messages to central
SIEM solution.

3) In parallel to 2) apply a different set of filters (not necessarily a
superset) and write remaining syslog messages to a separate file for
realtime monitoring. (I'll just tail this one.)

Concatenating steps 2 and 3 within a single ruleset is straightforward
but only works under the assumption that the filters applied in 2 are a
subset of the filters applied in 3.

This brings me to my questions: Is there a way to process a single
syslog messages in multiple rulesets (action chains) in parallel without
affecting each other? Is "call()" the right way to go, like use "call
rs_siem; call rs_rtmon" inside a ruleset rs_main?

If that isn't possible what would be the best alternative to achieve
this? From the top of my head I could think about sending each message
twice for the two rulesets rs_siem and rs_rtmon...

Best,
Matthias

PS: Apologies if this question has been asked and answered on the
mailinglist but I didn't manage to find it. Please simply point to the
email thread so I can digest the info from there.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
ruleset (name=“from_input”) {
call write_to_file
call to_central_siem
call to_separate_file
}

If I understand the internals, each call is essentially a “duplicated fork” that will operate independently of each other. It’s in the documentation somewhere, but using the call mechanism is correct.

Regards,



> On Jan 19, 2021, at 09:48, Matthias Sitte via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi there,
>
> I am looking into the following configuration but a bit unsure about some implementation details...
>
> Several originators send syslog messages to a rsyslog relay which shall do the following tasks:
>
> 1) Write every log message to separate files based on hostnames.
>
> 2) Apply a few filters and send remaining syslog messages to central SIEM solution.
>
> 3) In parallel to 2) apply a different set of filters (not necessarily a superset) and write remaining syslog messages to a separate file for realtime monitoring. (I'll just tail this one.)
>
> Concatenating steps 2 and 3 within a single ruleset is straightforward but only works under the assumption that the filters applied in 2 are a subset of the filters applied in 3.
>
> This brings me to my questions: Is there a way to process a single syslog messages in multiple rulesets (action chains) in parallel without affecting each other? Is "call()" the right way to go, like use "call rs_siem; call rs_rtmon" inside a ruleset rs_main?
>
> If that isn't possible what would be the best alternative to achieve this? From the top of my head I could think about sending each message twice for the two rulesets rs_siem and rs_rtmon...
>
> Best,
> Matthias
>
> PS: Apologies if this question has been asked and answered on the mailinglist but I didn't manage to find it. Please simply point to the email thread so I can digest the info from there.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
El mar, 19 ene 2021 a las 17:24, John Chivian via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> ruleset (name=“from_input”) {
> call write_to_file
> call to_central_siem
> call to_separate_file
> }
>
> If I understand the internals, each call is essentially a “duplicated fork” that will operate independently of each other. It’s in the documentation somewhere, but using the call mechanism is correct.

YES, BUT ... you need to assign a queue to the called ruleset. Else
it's synchronous.

Rainer

>
> Regards,
>
>
>
> > On Jan 19, 2021, at 09:48, Matthias Sitte via rsyslog <rsyslog@lists.adiscon.com> wrote:
> >
> > Hi there,
> >
> > I am looking into the following configuration but a bit unsure about some implementation details...
> >
> > Several originators send syslog messages to a rsyslog relay which shall do the following tasks:
> >
> > 1) Write every log message to separate files based on hostnames.
> >
> > 2) Apply a few filters and send remaining syslog messages to central SIEM solution.
> >
> > 3) In parallel to 2) apply a different set of filters (not necessarily a superset) and write remaining syslog messages to a separate file for realtime monitoring. (I'll just tail this one.)
> >
> > Concatenating steps 2 and 3 within a single ruleset is straightforward but only works under the assumption that the filters applied in 2 are a subset of the filters applied in 3.
> >
> > This brings me to my questions: Is there a way to process a single syslog messages in multiple rulesets (action chains) in parallel without affecting each other? Is "call()" the right way to go, like use "call rs_siem; call rs_rtmon" inside a ruleset rs_main?
> >
> > If that isn't possible what would be the best alternative to achieve this? From the top of my head I could think about sending each message twice for the two rulesets rs_siem and rs_rtmon...
> >
> > Best,
> > Matthias
> >
> > PS: Apologies if this question has been asked and answered on the mailinglist but I didn't manage to find it. Please simply point to the email thread so I can digest the info from there.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
On Tue, 19 Jan 2021, Matthias Sitte via rsyslog wrote:

> This brings me to my questions: Is there a way to process a single syslog
> messages in multiple rulesets (action chains) in parallel without affecting
> each other? Is "call()" the right way to go, like use "call rs_siem; call
> rs_rtmon" inside a ruleset rs_main?
>
> If that isn't possible what would be the best alternative to achieve this?
> From the top of my head I could think about sending each message twice for
> the two rulesets rs_siem and rs_rtmon...

why do you need to process a single message in parallel rather than having two
worker threads working on different batches of messages at the same time?

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
I have tended to put the queues on the enclosed actions rather than on the rulesets, especially in situations where multiple outbound actions within a ruleset might block independently of each other, but yes… the queue (somewhere) is required for it to be asynchronous. That part is in the documentation too.

Question for Rainer (and David)… If the queue goes on the ruleset, not the enclosed actions, does each action within the ruleset use the queue independently, regardless of whether or not it’s also in use for other actions within the ruleset? What happens if one action is blocked but another is not? And then in corollary, what happens if rsyslog shuts down and saves the queue while this is happening? On resumption, is the queued event fed back through all actions in the ruleset?

I think for what I want to do (multiple network output actions within a ruleset) having the queue on those actions (one per destination) is correct, but please correct me if in fact it doesn’t really make any functional difference.

I didn’t see such detail in the docs. Thanks,


> On Jan 19, 2021, at 10:28, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>
> YES, BUT ... you need to assign a queue to the called ruleset. Else
> it's synchronous.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
On Tue, 19 Jan 2021, John Chivian via rsyslog wrote:

> I have tended to put the queues on the enclosed actions rather than on the rulesets, especially in situations where multiple outbound actions within a ruleset might block independently of each other, but yes… the queue (somewhere) is required for it to be asynchronous. That part is in the documentation too.
>
> Question for Rainer (and David)… If the queue goes on the ruleset, not the enclosed actions, does each action within the ruleset use the queue independently, regardless of whether or not it’s also in use for other actions within the ruleset? What happens if one action is blocked but another is not? And then in corollary, what happens if rsyslog shuts down and saves the queue while this is happening? On resumption, is the queued event fed back through all actions in the ruleset?

I believe that if a batch fails and is retried (with or without shutdown in the
middle), it will go through all actions in the queue.

the worker thread gets the message batch from the queue and then for each
message in the batch, works it's way down through all the actions (remember,
there can be dependencies between actions, setting variables, stop, etc)

If you have multiple outputs that can block independently, then you want
separate queues, but you may want to think about how independed they really are
(is there likely to be a common problem, netork, congested server, etc that
really makes them less independent than they look at first glance)

David Lang

> I think for what I want to do (multiple network output actions within a
> ruleset) having the queue on those actions (one per destination) is correct,
> but please correct me if in fact it doesn’t really make any functional
> difference.
>
> I didn’t see such detail in the docs. Thanks,
>
>
>> On Jan 19, 2021, at 10:28, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>>
>> YES, BUT ... you need to assign a queue to the called ruleset. Else
>> it's synchronous.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
On 19/01/2021 17:28, Rainer Gerhards via rsyslog wrote:
> El mar, 19 ene 2021 a las 17:24, John Chivian via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>> ruleset (name=“from_input”) {
>> call write_to_file
>> call to_central_siem
>> call to_separate_file
>> }
>>
>> If I understand the internals, each call is essentially a “duplicated fork” that will operate independently of each other. It’s in the documentation somewhere, but using the call mechanism is correct.
> YES, BUT ... you need to assign a queue to the called ruleset. Else
> it's synchronous.
>
> Rainer

Let me dig a bit deeper ;-)

Let's assume that all those called rulesets manipulate a variable (let's
call it  $.var).

So far I relied on one ruleset modifying said $.var and the resulting
value being retained and accessible in subsequent ruleset. So, for
example, first subruleset could assign a value to a variable and in the
second subruleset you would choose action or assign another variable
based on the value of $.var. Like this:

ruleset (name="get_source") {

    if ( $hostname == "localhost") then

    {

        set $.src = $fromhost-ip;

    } else

    {

        set $.src = $hostname;

    }

}


ruleset (name="output_action) {

    if (( $.src starts with "172.16.") or ($.src ends with ".local")) then

    {

        action(type="omfile" file="/var/log/local");

    } else

    {

        action(type="omfile" file="/var/log/remote");

    }

}


ruleset(name="main) {

    call get_source

    call output_action

}

Let's not dig too deeply into the general idea (and possibly syntactic
correctness; I might have missed some semicolon or something like that
;-)) but as you can see, we assign a variable $.src in get_source
ruleset and use it again in output_action ruleset.

I've used this form of passing variables from one ruleset to another
many times and it does work.

Question is - will the behaviour change if those get_source and
output_action rulesets get their own queues?


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
Not much to add to David's response.

A queue on a ruleset is just like the default main queue on the
default ruleset. It means that

1. the to-be-processed messages are grabbed from the queue,
2. run through all actions synchronously
3. after that purged from queue

If one message blocks during phase 2, the following actions also block
(rough approximation, may happen in the commit phase and as such a bit
non-immediately - but that's even hard to observe).

If your actions can actually block individually, you need a queue on
each of them.

Nevertheless: to the original question: if you have a rule set which
has no queue, and all are on the actions, the ruleset is called
synchronously, but the actions are than carried out async. It's always
the same, not matter where the queue is put.

Rainer

El mar, 19 ene 2021 a las 21:25, David Lang (<david@lang.hm>) escribió:
>
> On Tue, 19 Jan 2021, John Chivian via rsyslog wrote:
>
> > I have tended to put the queues on the enclosed actions rather than on the rulesets, especially in situations where multiple outbound actions within a ruleset might block independently of each other, but yes… the queue (somewhere) is required for it to be asynchronous. That part is in the documentation too.
> >
> > Question for Rainer (and David)… If the queue goes on the ruleset, not the enclosed actions, does each action within the ruleset use the queue independently, regardless of whether or not it’s also in use for other actions within the ruleset? What happens if one action is blocked but another is not? And then in corollary, what happens if rsyslog shuts down and saves the queue while this is happening? On resumption, is the queued event fed back through all actions in the ruleset?
>
> I believe that if a batch fails and is retried (with or without shutdown in the
> middle), it will go through all actions in the queue.
>
> the worker thread gets the message batch from the queue and then for each
> message in the batch, works it's way down through all the actions (remember,
> there can be dependencies between actions, setting variables, stop, etc)
>
> If you have multiple outputs that can block independently, then you want
> separate queues, but you may want to think about how independed they really are
> (is there likely to be a common problem, netork, congested server, etc that
> really makes them less independent than they look at first glance)
>
> David Lang
>
> > I think for what I want to do (multiple network output actions within a
> > ruleset) having the queue on those actions (one per destination) is correct,
> > but please correct me if in fact it doesn’t really make any functional
> > difference.
> >
> > I didn’t see such detail in the docs. Thanks,
> >
> >
> >> On Jan 19, 2021, at 10:28, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
> >>
> >> YES, BUT ... you need to assign a queue to the called ruleset. Else
> >> it's synchronous.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
On Wed, 20 Jan 2021, Mariusz Kruk via rsyslog wrote:

> Question is - will the behaviour change if those get_source and
> output_action rulesets get their own queues?

Yes, when you introduce a queue, the message gets duplicated into that queue and
any changes to the original message won't apply to the copy.

Rainer has a post up explaining queues as traffic turn lanes. with no queues,
the cars make a left turn, do the action/ruleset, do a u-turn and go back to the
main road.

a queue is like a magic turn lane, when there is a queue, the car gets
duplicated, one copy goes into the turn lane and makes the left (waiting as
needed), the other copy continues without delay.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Processing one syslog message in multiple rulesets in parallel [ In reply to ]
On 20/01/2021 14:28, David Lang wrote:
> On Wed, 20 Jan 2021, Mariusz Kruk via rsyslog wrote:
>
>> Question is - will the behaviour change if those get_source and
>> output_action rulesets get their own queues?
>
> Yes, when you introduce a queue, the message gets duplicated into that
> queue and any changes to the original message won't apply to the copy.
>
> Rainer has a post up explaining queues as traffic turn lanes. with no
> queues, the cars make a left turn, do the action/ruleset, do a u-turn
> and go back to the main road.
>
> a queue is like a magic turn lane, when there is a queue, the car gets
> duplicated, one copy goes into the turn lane and makes the left
> (waiting as needed), the other copy continues without delay.
>
That's pretty much what I expected. Thanks for confirmation.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.