Mailing List Archive

Message modification in DIRECT Queues have different semantics when using with rulesets vs actions
Hi Rsyslog Users,

I am trying to understand how queues work in Rsyslog. In case of non-direct
queues, message is copied and placed in the queue and message modifications
done by queue workers won't have any impact on the original message. Makes
sense here.

However, I am a little confused when it comes to direct Queue. What
happens in Direct Queue ? Will the message be copied or is it the same
message ? I tested out two scenarios (wrt rulesets and actions) and both
of them gave different results.

When ruleset is backed by a direct Queue, message modifications done in
ruleset don't reflect back in original flow where as in case of actions
(such mmnormalize, mmkubernetes) which are by default backed by direct
Queue, message modifications done with action reflects in original flow

Scenario 1:

template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")

ruleset(name="relay.htp1" queue.type="Direct") {
call rs1
* // $!var1, $.var2 aren't available here*
action(type="omfile" file="/tmp/output.log" template="abc")
call relay.htp
}

ruleset(name="rs1" queue.type="Direct"){
set $!var1 = "hello";
set $.var2 = "bye";
}

input(type="imfile"
File="/tmp/input.log"
Ruleset="relay.htp1"
Tag="tag")


Scenario 2:

module(load = "mmnormalize")
ruleset(name = "relay.htp1" queue.type="Direct") {
action(type = "mmnormalize" ruleBase="/etc/rsyslog.d/service.rulebase"
path="$!msg")
* // $!msg will be available here even though action is backed by a
default Queue. *
}

input(type="imfile"
File="/tmp/input.log"
Ruleset="relay.htp1"
Tag="tag")


How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
available when both are using Direct Queue. Am I missing something here ?

--
Regards,
Rajesh KSV
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
The definitive reference is https://www.rsyslog.com/doc/v8-stable/concepts/queues.html <https://www.rsyslog.com/doc/v8-stable/concepts/queues.html>

In scenario one you do not need a queue on ruleset rs1 as there are no outputs which would require one.
Also, although the maintainers will need to say for sure, user variable defined in a call may not be visible to the parent. Same as in most programming languages.

Regards,


> On Sep 16, 2021, at 08:21, rajeshksv via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi Rsyslog Users,
>
> I am trying to understand how queues work in Rsyslog. In case of non-direct
> queues, message is copied and placed in the queue and message modifications
> done by queue workers won't have any impact on the original message. Makes
> sense here.
>
> However, I am a little confused when it comes to direct Queue. What
> happens in Direct Queue ? Will the message be copied or is it the same
> message ? I tested out two scenarios (wrt rulesets and actions) and both
> of them gave different results.
>
> When ruleset is backed by a direct Queue, message modifications done in
> ruleset don't reflect back in original flow where as in case of actions
> (such mmnormalize, mmkubernetes) which are by default backed by direct
> Queue, message modifications done with action reflects in original flow
>
> Scenario 1:
>
> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>
> ruleset(name="relay.htp1" queue.type="Direct") {
> call rs1
> * // $!var1, $.var2 aren't available here*
> action(type="omfile" file="/tmp/output.log" template="abc")
> call relay.htp
> }
>
> ruleset(name="rs1" queue.type="Direct"){
> set $!var1 = "hello";
> set $.var2 = "bye";
> }
>
> input(type="imfile"
> File="/tmp/input.log"
> Ruleset="relay.htp1"
> Tag="tag")
>
>
> Scenario 2:
>
> module(load = "mmnormalize")
> ruleset(name = "relay.htp1" queue.type="Direct") {
> action(type = "mmnormalize" ruleBase="/etc/rsyslog.d/service.rulebase"
> path="$!msg")
> * // $!msg will be available here even though action is backed by a
> default Queue. *
> }
>
> input(type="imfile"
> File="/tmp/input.log"
> Ruleset="relay.htp1"
> Tag="tag")
>
>
> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> available when both are using Direct Queue. Am I missing something here ?
>
> --
> Regards,
> Rajesh KSV
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
I believe it's not about queue, but about mmnormalize. I haven't used
mmnormalize much myself but from the docs I understand that mmnormalize
should only be called once on a message and is supposed to parse the
message into a set of "global" properties.


On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
> Hi Rsyslog Users,
>
> I am trying to understand how queues work in Rsyslog. In case of non-direct
> queues, message is copied and placed in the queue and message modifications
> done by queue workers won't have any impact on the original message. Makes
> sense here.
>
> However, I am a little confused when it comes to direct Queue. What
> happens in Direct Queue ? Will the message be copied or is it the same
> message ? I tested out two scenarios (wrt rulesets and actions) and both
> of them gave different results.
>
> When ruleset is backed by a direct Queue, message modifications done in
> ruleset don't reflect back in original flow where as in case of actions
> (such mmnormalize, mmkubernetes) which are by default backed by direct
> Queue, message modifications done with action reflects in original flow
>
> Scenario 1:
>
> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>
> ruleset(name="relay.htp1" queue.type="Direct") {
> call rs1
> * // $!var1, $.var2 aren't available here*
> action(type="omfile" file="/tmp/output.log" template="abc")
> call relay.htp
> }
>
> ruleset(name="rs1" queue.type="Direct"){
> set $!var1 = "hello";
> set $.var2 = "bye";
> }
>
> input(type="imfile"
> File="/tmp/input.log"
> Ruleset="relay.htp1"
> Tag="tag")
>
>
> Scenario 2:
>
> module(load = "mmnormalize")
> ruleset(name = "relay.htp1" queue.type="Direct") {
> action(type = "mmnormalize" ruleBase="/etc/rsyslog.d/service.rulebase"
> path="$!msg")
> * // $!msg will be available here even though action is backed by a
> default Queue. *
> }
>
> input(type="imfile"
> File="/tmp/input.log"
> Ruleset="relay.htp1"
> Tag="tag")
>
>
> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> available when both are using Direct Queue. Am I missing something here ?
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
There is no reason you can't call mmnormalize multiple times, you can specify
where to put the results (by default, they go under $!)

I'm not really sure what a direct queue on a ruleset means, you should just call
the ruleset without it having any queue on it.

David Lang

On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:

> I believe it's not about queue, but about mmnormalize. I haven't used
> mmnormalize much myself but from the docs I understand that mmnormalize
> should only be called once on a message and is supposed to parse the message
> into a set of "global" properties.
>
>
> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
>> Hi Rsyslog Users,
>>
>> I am trying to understand how queues work in Rsyslog. In case of non-direct
>> queues, message is copied and placed in the queue and message modifications
>> done by queue workers won't have any impact on the original message. Makes
>> sense here.
>>
>> However, I am a little confused when it comes to direct Queue. What
>> happens in Direct Queue ? Will the message be copied or is it the same
>> message ? I tested out two scenarios (wrt rulesets and actions) and both
>> of them gave different results.
>>
>> When ruleset is backed by a direct Queue, message modifications done in
>> ruleset don't reflect back in original flow where as in case of actions
>> (such mmnormalize, mmkubernetes) which are by default backed by direct
>> Queue, message modifications done with action reflects in original flow
>>
>> Scenario 1:
>>
>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>>
>> ruleset(name="relay.htp1" queue.type="Direct") {
>> call rs1
>> * // $!var1, $.var2 aren't available here*
>> action(type="omfile" file="/tmp/output.log" template="abc")
>> call relay.htp
>> }
>>
>> ruleset(name="rs1" queue.type="Direct"){
>> set $!var1 = "hello";
>> set $.var2 = "bye";
>> }
>>
>> input(type="imfile"
>> File="/tmp/input.log"
>> Ruleset="relay.htp1"
>> Tag="tag")
>>
>>
>> Scenario 2:
>>
>> module(load = "mmnormalize")
>> ruleset(name = "relay.htp1" queue.type="Direct") {
>> action(type = "mmnormalize" ruleBase="/etc/rsyslog.d/service.rulebase"
>> path="$!msg")
>> * // $!msg will be available here even though action is backed by a
>> default Queue. *
>> }
>>
>> input(type="imfile"
>> File="/tmp/input.log"
>> Ruleset="relay.htp1"
>> Tag="tag")
>>
>>
>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
>> available when both are using Direct Queue. Am I missing something here ?
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html

"Note that mmnormalize should only be called once on each message.
Behaviour is undefined if multiple calls to mmnormalize happen for the
same message."

From my tests it turns out that even with direct queue the message gets
copied when entering a separate queue and thus the results are not
inherited on the ruleset (and queue) exit.

So it seems to be the mmnormalize that's causing the OP's variable to be
retained after the ruleset exit.

On 16.09.2021 20:54, David Lang wrote:
> There is no reason you can't call mmnormalize multiple times, you can
> specify where to put the results (by default, they go under $!)
>
> I'm not really sure what a direct queue on a ruleset means, you should
> just call the ruleset without it having any queue on it.
>
> David Lang
>
> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>
>> I believe it's not about queue, but about mmnormalize. I haven't used
>> mmnormalize much myself but from the docs I understand that
>> mmnormalize should only be called once on a message and is supposed
>> to parse the message into a set of "global" properties.
>>
>>
>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
>>> Hi Rsyslog Users,
>>>
>>> I am trying to understand how queues work in Rsyslog. In case of
>>> non-direct
>>> queues, message is copied and placed in the queue and message
>>> modifications
>>> done by queue workers won't have any impact on the original message.
>>> Makes
>>> sense here.
>>>
>>> However, I am a little confused when it comes to direct Queue.  What
>>> happens in Direct Queue ? Will the message be copied or is it the same
>>> message ? I tested out two scenarios (wrt rulesets and actions)  and
>>> both
>>> of them gave different results.
>>>
>>> When ruleset is backed by a direct Queue, message modifications done in
>>> ruleset don't reflect back in original flow where as in case of actions
>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
>>> Queue, message modifications done with action reflects in original flow
>>>
>>> Scenario 1:
>>>
>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>>>
>>> ruleset(name="relay.htp1" queue.type="Direct") {
>>>       call rs1
>>>     * // $!var1, $.var2 aren't available here*
>>>       action(type="omfile" file="/tmp/output.log" template="abc")
>>>       call relay.htp
>>> }
>>>
>>> ruleset(name="rs1" queue.type="Direct"){
>>>      set $!var1 = "hello";
>>>      set $.var2 = "bye";
>>> }
>>>
>>> input(type="imfile"
>>>         File="/tmp/input.log"
>>>         Ruleset="relay.htp1"
>>>         Tag="tag")
>>>
>>>
>>> Scenario 2:
>>>
>>> module(load = "mmnormalize")
>>> ruleset(name = "relay.htp1"  queue.type="Direct") {
>>>     action(type = "mmnormalize"
>>> ruleBase="/etc/rsyslog.d/service.rulebase"
>>> path="$!msg")
>>>    * // $!msg will be available here even though action is backed by a
>>> default Queue. *
>>> }
>>>
>>> input(type="imfile"
>>>         File="/tmp/input.log"
>>>         Ruleset="relay.htp1"
>>>         Tag="tag")
>>>
>>>
>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
>>> available when both are using Direct Queue. Am I missing something
>>> here ?
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:

> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
>
> "Note that mmnormalize should only be called once on each message.
> Behaviour is undefined if multiple calls to mmnormalize happen for the
> same message."

I called that out and it should have been removed from current documentation.

> From my tests it turns out that even with direct queue the message gets
> copied when entering a separate queue and thus the results are not
> inherited on the ruleset (and queue) exit.
>
> So it seems to be the mmnormalize that's causing the OP's variable to be
> retained after the ruleset exit.

but if you don't specify any queue at all, then things inside a ruleset will
affect things outside the ruleset.

David Lang

> On 16.09.2021 20:54, David Lang wrote:
>> There is no reason you can't call mmnormalize multiple times, you can
>> specify where to put the results (by default, they go under $!)
>>
>> I'm not really sure what a direct queue on a ruleset means, you should
>> just call the ruleset without it having any queue on it.
>>
>> David Lang
>>
>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>>
>>> I believe it's not about queue, but about mmnormalize. I haven't used
>>> mmnormalize much myself but from the docs I understand that
>>> mmnormalize should only be called once on a message and is supposed
>>> to parse the message into a set of "global" properties.
>>>
>>>
>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
>>>> Hi Rsyslog Users,
>>>>
>>>> I am trying to understand how queues work in Rsyslog. In case of
>>>> non-direct
>>>> queues, message is copied and placed in the queue and message
>>>> modifications
>>>> done by queue workers won't have any impact on the original message.
>>>> Makes
>>>> sense here.
>>>>
>>>> However, I am a little confused when it comes to direct Queue.  What
>>>> happens in Direct Queue ? Will the message be copied or is it the same
>>>> message ? I tested out two scenarios (wrt rulesets and actions)  and
>>>> both
>>>> of them gave different results.
>>>>
>>>> When ruleset is backed by a direct Queue, message modifications done in
>>>> ruleset don't reflect back in original flow where as in case of actions
>>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
>>>> Queue, message modifications done with action reflects in original flow
>>>>
>>>> Scenario 1:
>>>>
>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>>>>
>>>> ruleset(name="relay.htp1" queue.type="Direct") {
>>>>       call rs1
>>>>     * // $!var1, $.var2 aren't available here*
>>>>       action(type="omfile" file="/tmp/output.log" template="abc")
>>>>       call relay.htp
>>>> }
>>>>
>>>> ruleset(name="rs1" queue.type="Direct"){
>>>>      set $!var1 = "hello";
>>>>      set $.var2 = "bye";
>>>> }
>>>>
>>>> input(type="imfile"
>>>>         File="/tmp/input.log"
>>>>         Ruleset="relay.htp1"
>>>>         Tag="tag")
>>>>
>>>>
>>>> Scenario 2:
>>>>
>>>> module(load = "mmnormalize")
>>>> ruleset(name = "relay.htp1"  queue.type="Direct") {
>>>>     action(type = "mmnormalize"
>>>> ruleBase="/etc/rsyslog.d/service.rulebase"
>>>> path="$!msg")
>>>>    * // $!msg will be available here even though action is backed by a
>>>> default Queue. *
>>>> }
>>>>
>>>> input(type="imfile"
>>>>         File="/tmp/input.log"
>>>>         Ruleset="relay.htp1"
>>>>         Tag="tag")
>>>>
>>>>
>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
>>>> available when both are using Direct Queue. Am I missing something
>>>> here ?
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
Agreed. We don't need a direct queue before ruleset. I added it only to
understand what's going with DIRECT Queues (Since its a very novel concept
- Haven't heard something like this before)

If you remove the DIRECT queue from ruleset(in Scenario I), then
modifications done in ruleset are visible to the main lane. It's working as
expected. But since I am interested in understanding DIRECT Queues, I added
it.

>>> Also, although the maintainers will need to say for sure, user variable
defined in a call may not be visible to the parent. Same as in most
programming languages.
Rsyslog works in a different way, in few scenarios, user variables defined
in a call will be visible to the parent (If ruleset is not backed by
Queue). So, its not similar to other languages ;)

>>> "Note that mmnormalize should only be called once on each message.
Behaviour is undefined if multiple calls to mmnormalize happen for the
same message."
Its a documentation bug, Rainer has raised a PR and will be resolved soon -
https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
multiple times.


On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>
> >
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
> >
> > "Note that mmnormalize should only be called once on each message.
> > Behaviour is undefined if multiple calls to mmnormalize happen for the
> > same message."
>
> I called that out and it should have been removed from current
> documentation.
>
> > From my tests it turns out that even with direct queue the message gets
> > copied when entering a separate queue and thus the results are not
> > inherited on the ruleset (and queue) exit.
> >
> > So it seems to be the mmnormalize that's causing the OP's variable to be
> > retained after the ruleset exit.
>
> but if you don't specify any queue at all, then things inside a ruleset
> will
> affect things outside the ruleset.
>
> David Lang
>
> > On 16.09.2021 20:54, David Lang wrote:
> >> There is no reason you can't call mmnormalize multiple times, you can
> >> specify where to put the results (by default, they go under $!)
> >>
> >> I'm not really sure what a direct queue on a ruleset means, you should
> >> just call the ruleset without it having any queue on it.
> >>
> >> David Lang
> >>
> >> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> >>
> >>> I believe it's not about queue, but about mmnormalize. I haven't used
> >>> mmnormalize much myself but from the docs I understand that
> >>> mmnormalize should only be called once on a message and is supposed
> >>> to parse the message into a set of "global" properties.
> >>>
> >>>
> >>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
> >>>> Hi Rsyslog Users,
> >>>>
> >>>> I am trying to understand how queues work in Rsyslog. In case of
> >>>> non-direct
> >>>> queues, message is copied and placed in the queue and message
> >>>> modifications
> >>>> done by queue workers won't have any impact on the original message.
> >>>> Makes
> >>>> sense here.
> >>>>
> >>>> However, I am a little confused when it comes to direct Queue. What
> >>>> happens in Direct Queue ? Will the message be copied or is it the same
> >>>> message ? I tested out two scenarios (wrt rulesets and actions) and
> >>>> both
> >>>> of them gave different results.
> >>>>
> >>>> When ruleset is backed by a direct Queue, message modifications done
> in
> >>>> ruleset don't reflect back in original flow where as in case of
> actions
> >>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
> >>>> Queue, message modifications done with action reflects in original
> flow
> >>>>
> >>>> Scenario 1:
> >>>>
> >>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
> >>>>
> >>>> ruleset(name="relay.htp1" queue.type="Direct") {
> >>>> call rs1
> >>>> * // $!var1, $.var2 aren't available here*
> >>>> action(type="omfile" file="/tmp/output.log" template="abc")
> >>>> call relay.htp
> >>>> }
> >>>>
> >>>> ruleset(name="rs1" queue.type="Direct"){
> >>>> set $!var1 = "hello";
> >>>> set $.var2 = "bye";
> >>>> }
> >>>>
> >>>> input(type="imfile"
> >>>> File="/tmp/input.log"
> >>>> Ruleset="relay.htp1"
> >>>> Tag="tag")
> >>>>
> >>>>
> >>>> Scenario 2:
> >>>>
> >>>> module(load = "mmnormalize")
> >>>> ruleset(name = "relay.htp1" queue.type="Direct") {
> >>>> action(type = "mmnormalize"
> >>>> ruleBase="/etc/rsyslog.d/service.rulebase"
> >>>> path="$!msg")
> >>>> * // $!msg will be available here even though action is backed by a
> >>>> default Queue. *
> >>>> }
> >>>>
> >>>> input(type="imfile"
> >>>> File="/tmp/input.log"
> >>>> Ruleset="relay.htp1"
> >>>> Tag="tag")
> >>>>
> >>>>
> >>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> >>>> available when both are using Direct Queue. Am I missing something
> >>>> here ?
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>> POST if you DON'T LIKE THAT.
> >>>
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.



--
Regards,
Rajesh KSV
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
The default queue mode is "direct" - no matter if it is explicitly
specified or not.

There are indeed semantic differences between the "direct" queue
(a.k.a. "no real queue exists") and any other (real) queue type. As
you say, in direct mode, the message object is NOT copied. So anything
done to it, will potentially affect other uses of the very same
message object. I real queues, the message is copied and thus no
processing outside the scope of the queue is affected.

NOTE WELL: if you have parts with direct AND real queues AND modify
messages in "direct queue constructs" AND have actions/rulesets with
real queues running in parallel, you have a race. If the modification
made by the direct queue part depends on whether processing of the
real queues has already started. For "regular" rule sets this is no
problem, because main processing is sequential. However, if you do
very complex setups with multipele rule sets calling into each other
and using different queue modes, this can affect you.

It is best, also for maintainability, to have a simple sequential
structure in top level processing, spawning off topic rule sets (if at
all required). then do all "global" modifications at the beginning.
And do local (intentionally non outside-visible) modifications in the
topic rule sets.

I hope this helps,
Rainer

El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Agreed. We don't need a direct queue before ruleset. I added it only to
> understand what's going with DIRECT Queues (Since its a very novel concept
> - Haven't heard something like this before)
>
> If you remove the DIRECT queue from ruleset(in Scenario I), then
> modifications done in ruleset are visible to the main lane. It's working as
> expected. But since I am interested in understanding DIRECT Queues, I added
> it.
>
> >>> Also, although the maintainers will need to say for sure, user variable
> defined in a call may not be visible to the parent. Same as in most
> programming languages.
> Rsyslog works in a different way, in few scenarios, user variables defined
> in a call will be visible to the parent (If ruleset is not backed by
> Queue). So, its not similar to other languages ;)
>
> >>> "Note that mmnormalize should only be called once on each message.
> Behaviour is undefined if multiple calls to mmnormalize happen for the
> same message."
> Its a documentation bug, Rainer has raised a PR and will be resolved soon -
> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
> multiple times.
>
>
> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> > On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> >
> > >
> > https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
> > >
> > > "Note that mmnormalize should only be called once on each message.
> > > Behaviour is undefined if multiple calls to mmnormalize happen for the
> > > same message."
> >
> > I called that out and it should have been removed from current
> > documentation.
> >
> > > From my tests it turns out that even with direct queue the message gets
> > > copied when entering a separate queue and thus the results are not
> > > inherited on the ruleset (and queue) exit.
> > >
> > > So it seems to be the mmnormalize that's causing the OP's variable to be
> > > retained after the ruleset exit.
> >
> > but if you don't specify any queue at all, then things inside a ruleset
> > will
> > affect things outside the ruleset.
> >
> > David Lang
> >
> > > On 16.09.2021 20:54, David Lang wrote:
> > >> There is no reason you can't call mmnormalize multiple times, you can
> > >> specify where to put the results (by default, they go under $!)
> > >>
> > >> I'm not really sure what a direct queue on a ruleset means, you should
> > >> just call the ruleset without it having any queue on it.
> > >>
> > >> David Lang
> > >>
> > >> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> > >>
> > >>> I believe it's not about queue, but about mmnormalize. I haven't used
> > >>> mmnormalize much myself but from the docs I understand that
> > >>> mmnormalize should only be called once on a message and is supposed
> > >>> to parse the message into a set of "global" properties.
> > >>>
> > >>>
> > >>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
> > >>>> Hi Rsyslog Users,
> > >>>>
> > >>>> I am trying to understand how queues work in Rsyslog. In case of
> > >>>> non-direct
> > >>>> queues, message is copied and placed in the queue and message
> > >>>> modifications
> > >>>> done by queue workers won't have any impact on the original message.
> > >>>> Makes
> > >>>> sense here.
> > >>>>
> > >>>> However, I am a little confused when it comes to direct Queue. What
> > >>>> happens in Direct Queue ? Will the message be copied or is it the same
> > >>>> message ? I tested out two scenarios (wrt rulesets and actions) and
> > >>>> both
> > >>>> of them gave different results.
> > >>>>
> > >>>> When ruleset is backed by a direct Queue, message modifications done
> > in
> > >>>> ruleset don't reflect back in original flow where as in case of
> > actions
> > >>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
> > >>>> Queue, message modifications done with action reflects in original
> > flow
> > >>>>
> > >>>> Scenario 1:
> > >>>>
> > >>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
> > >>>>
> > >>>> ruleset(name="relay.htp1" queue.type="Direct") {
> > >>>> call rs1
> > >>>> * // $!var1, $.var2 aren't available here*
> > >>>> action(type="omfile" file="/tmp/output.log" template="abc")
> > >>>> call relay.htp
> > >>>> }
> > >>>>
> > >>>> ruleset(name="rs1" queue.type="Direct"){
> > >>>> set $!var1 = "hello";
> > >>>> set $.var2 = "bye";
> > >>>> }
> > >>>>
> > >>>> input(type="imfile"
> > >>>> File="/tmp/input.log"
> > >>>> Ruleset="relay.htp1"
> > >>>> Tag="tag")
> > >>>>
> > >>>>
> > >>>> Scenario 2:
> > >>>>
> > >>>> module(load = "mmnormalize")
> > >>>> ruleset(name = "relay.htp1" queue.type="Direct") {
> > >>>> action(type = "mmnormalize"
> > >>>> ruleBase="/etc/rsyslog.d/service.rulebase"
> > >>>> path="$!msg")
> > >>>> * // $!msg will be available here even though action is backed by a
> > >>>> default Queue. *
> > >>>> }
> > >>>>
> > >>>> input(type="imfile"
> > >>>> File="/tmp/input.log"
> > >>>> Ruleset="relay.htp1"
> > >>>> Tag="tag")
> > >>>>
> > >>>>
> > >>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> > >>>> available when both are using Direct Queue. Am I missing something
> > >>>> here ?
> > >>>>
> > >>> _______________________________________________
> > >>> rsyslog mailing list
> > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> http://www.rsyslog.com/professional-services/
> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > >>> POST if you DON'T LIKE THAT.
> > >>>
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of
> > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T
> > > LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
>
>
>
> --
> Regards,
> Rajesh KSV
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On 16.09.2021 21:26, David Lang wrote:
> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>
>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
>>
>>
>> "Note that mmnormalize should only be called once on each message.
>> Behaviour is undefined if multiple calls to mmnormalize happen for
>> the same message."
>
> I called that out and it should have been removed from current
> documentation.
OK. Good to know.
>
>> From my tests it turns out that even with direct queue the message
>> gets copied when entering a separate queue and thus the results are
>> not inherited on the ruleset (and queue) exit.
>>
>> So it seems to be the mmnormalize that's causing the OP's variable to
>> be retained after the ruleset exit.
>
> but if you don't specify any queue at all, then things inside a
> ruleset will affect things outside the ruleset.
>
Sure. Then the flow inside a ruleset - which happens within the same
queue as the "outside", will affect the variables seen in the queue. I
use it heavily ;-)

But the OP asked about the direct queue and it seems that this queue
type is not "special" in anyway - it also creates a new scope.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
Hmm...

OK, so a simple setup of:

[cut]

ruleset(name="whatever" queue.type="direct") {

    set $.msg="whatever";

}

set $.msg=$msg;

call whatever

template using $.msg

output with the template

[cut]

Which from my tests shows that $.msg retains the $msg value even though
it was supposed to be overwritten by the "whatever" ruleset is caused
not by the queue mechanics but by non-linearity of execution and race?

MK

On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote:
> The default queue mode is "direct" - no matter if it is explicitly
> specified or not.
>
> There are indeed semantic differences between the "direct" queue
> (a.k.a. "no real queue exists") and any other (real) queue type. As
> you say, in direct mode, the message object is NOT copied. So anything
> done to it, will potentially affect other uses of the very same
> message object. I real queues, the message is copied and thus no
> processing outside the scope of the queue is affected.
>
> NOTE WELL: if you have parts with direct AND real queues AND modify
> messages in "direct queue constructs" AND have actions/rulesets with
> real queues running in parallel, you have a race. If the modification
> made by the direct queue part depends on whether processing of the
> real queues has already started. For "regular" rule sets this is no
> problem, because main processing is sequential. However, if you do
> very complex setups with multipele rule sets calling into each other
> and using different queue modes, this can affect you.
>
> It is best, also for maintainability, to have a simple sequential
> structure in top level processing, spawning off topic rule sets (if at
> all required). then do all "global" modifications at the beginning.
> And do local (intentionally non outside-visible) modifications in the
> topic rule sets.
>
> I hope this helps,
> Rainer
>
> El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>> Agreed. We don't need a direct queue before ruleset. I added it only to
>> understand what's going with DIRECT Queues (Since its a very novel concept
>> - Haven't heard something like this before)
>>
>> If you remove the DIRECT queue from ruleset(in Scenario I), then
>> modifications done in ruleset are visible to the main lane. It's working as
>> expected. But since I am interested in understanding DIRECT Queues, I added
>> it.
>>
>>>>> Also, although the maintainers will need to say for sure, user variable
>> defined in a call may not be visible to the parent. Same as in most
>> programming languages.
>> Rsyslog works in a different way, in few scenarios, user variables defined
>> in a call will be visible to the parent (If ruleset is not backed by
>> Queue). So, its not similar to other languages ;)
>>
>>>>> "Note that mmnormalize should only be called once on each message.
>> Behaviour is undefined if multiple calls to mmnormalize happen for the
>> same message."
>> Its a documentation bug, Rainer has raised a PR and will be resolved soon -
>> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
>> multiple times.
>>
>>
>> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>>
>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>>>
>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
>>>> "Note that mmnormalize should only be called once on each message.
>>>> Behaviour is undefined if multiple calls to mmnormalize happen for the
>>>> same message."
>>> I called that out and it should have been removed from current
>>> documentation.
>>>
>>>> From my tests it turns out that even with direct queue the message gets
>>>> copied when entering a separate queue and thus the results are not
>>>> inherited on the ruleset (and queue) exit.
>>>>
>>>> So it seems to be the mmnormalize that's causing the OP's variable to be
>>>> retained after the ruleset exit.
>>> but if you don't specify any queue at all, then things inside a ruleset
>>> will
>>> affect things outside the ruleset.
>>>
>>> David Lang
>>>
>>>> On 16.09.2021 20:54, David Lang wrote:
>>>>> There is no reason you can't call mmnormalize multiple times, you can
>>>>> specify where to put the results (by default, they go under $!)
>>>>>
>>>>> I'm not really sure what a direct queue on a ruleset means, you should
>>>>> just call the ruleset without it having any queue on it.
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
>>>>>
>>>>>> I believe it's not about queue, but about mmnormalize. I haven't used
>>>>>> mmnormalize much myself but from the docs I understand that
>>>>>> mmnormalize should only be called once on a message and is supposed
>>>>>> to parse the message into a set of "global" properties.
>>>>>>
>>>>>>
>>>>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
>>>>>>> Hi Rsyslog Users,
>>>>>>>
>>>>>>> I am trying to understand how queues work in Rsyslog. In case of
>>>>>>> non-direct
>>>>>>> queues, message is copied and placed in the queue and message
>>>>>>> modifications
>>>>>>> done by queue workers won't have any impact on the original message.
>>>>>>> Makes
>>>>>>> sense here.
>>>>>>>
>>>>>>> However, I am a little confused when it comes to direct Queue. What
>>>>>>> happens in Direct Queue ? Will the message be copied or is it the same
>>>>>>> message ? I tested out two scenarios (wrt rulesets and actions) and
>>>>>>> both
>>>>>>> of them gave different results.
>>>>>>>
>>>>>>> When ruleset is backed by a direct Queue, message modifications done
>>> in
>>>>>>> ruleset don't reflect back in original flow where as in case of
>>> actions
>>>>>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
>>>>>>> Queue, message modifications done with action reflects in original
>>> flow
>>>>>>> Scenario 1:
>>>>>>>
>>>>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
>>>>>>>
>>>>>>> ruleset(name="relay.htp1" queue.type="Direct") {
>>>>>>> call rs1
>>>>>>> * // $!var1, $.var2 aren't available here*
>>>>>>> action(type="omfile" file="/tmp/output.log" template="abc")
>>>>>>> call relay.htp
>>>>>>> }
>>>>>>>
>>>>>>> ruleset(name="rs1" queue.type="Direct"){
>>>>>>> set $!var1 = "hello";
>>>>>>> set $.var2 = "bye";
>>>>>>> }
>>>>>>>
>>>>>>> input(type="imfile"
>>>>>>> File="/tmp/input.log"
>>>>>>> Ruleset="relay.htp1"
>>>>>>> Tag="tag")
>>>>>>>
>>>>>>>
>>>>>>> Scenario 2:
>>>>>>>
>>>>>>> module(load = "mmnormalize")
>>>>>>> ruleset(name = "relay.htp1" queue.type="Direct") {
>>>>>>> action(type = "mmnormalize"
>>>>>>> ruleBase="/etc/rsyslog.d/service.rulebase"
>>>>>>> path="$!msg")
>>>>>>> * // $!msg will be available here even though action is backed by a
>>>>>>> default Queue. *
>>>>>>> }
>>>>>>>
>>>>>>> input(type="imfile"
>>>>>>> File="/tmp/input.log"
>>>>>>> Ruleset="relay.htp1"
>>>>>>> Tag="tag")
>>>>>>>
>>>>>>>
>>>>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
>>>>>>> available when both are using Direct Queue. Am I missing something
>>>>>>> here ?
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST if you DON'T LIKE THAT.
>>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T
>>>> LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>
>>
>> --
>> Regards,
>> Rajesh KSV
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
El vie, 17 sept 2021 a las 9:15, Mariusz Kruk via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hmm...
>
> OK, so a simple setup of:
>
> [cut]
>
> ruleset(name="whatever" queue.type="direct") {
>
> set $.msg="whatever";
>
> }
>
> set $.msg=$msg;
>
> call whatever
>
> template using $.msg
>
> output with the template
>
> [cut]
>
> Which from my tests shows that $.msg retains the $msg value even though
> it was supposed to be overwritten by the "whatever" ruleset is caused
> not by the queue mechanics but by non-linearity of execution and race?

I'll check. Maybe my memory served me incorrectly - but I don't think so.

Rainer

>
> MK


>
> On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote:
> > The default queue mode is "direct" - no matter if it is explicitly
> > specified or not.
> >
> > There are indeed semantic differences between the "direct" queue
> > (a.k.a. "no real queue exists") and any other (real) queue type. As
> > you say, in direct mode, the message object is NOT copied. So anything
> > done to it, will potentially affect other uses of the very same
> > message object. I real queues, the message is copied and thus no
> > processing outside the scope of the queue is affected.
> >
> > NOTE WELL: if you have parts with direct AND real queues AND modify
> > messages in "direct queue constructs" AND have actions/rulesets with
> > real queues running in parallel, you have a race. If the modification
> > made by the direct queue part depends on whether processing of the
> > real queues has already started. For "regular" rule sets this is no
> > problem, because main processing is sequential. However, if you do
> > very complex setups with multipele rule sets calling into each other
> > and using different queue modes, this can affect you.
> >
> > It is best, also for maintainability, to have a simple sequential
> > structure in top level processing, spawning off topic rule sets (if at
> > all required). then do all "global" modifications at the beginning.
> > And do local (intentionally non outside-visible) modifications in the
> > topic rule sets.
> >
> > I hope this helps,
> > Rainer
> >
> > El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> >> Agreed. We don't need a direct queue before ruleset. I added it only to
> >> understand what's going with DIRECT Queues (Since its a very novel concept
> >> - Haven't heard something like this before)
> >>
> >> If you remove the DIRECT queue from ruleset(in Scenario I), then
> >> modifications done in ruleset are visible to the main lane. It's working as
> >> expected. But since I am interested in understanding DIRECT Queues, I added
> >> it.
> >>
> >>>>> Also, although the maintainers will need to say for sure, user variable
> >> defined in a call may not be visible to the parent. Same as in most
> >> programming languages.
> >> Rsyslog works in a different way, in few scenarios, user variables defined
> >> in a call will be visible to the parent (If ruleset is not backed by
> >> Queue). So, its not similar to other languages ;)
> >>
> >>>>> "Note that mmnormalize should only be called once on each message.
> >> Behaviour is undefined if multiple calls to mmnormalize happen for the
> >> same message."
> >> Its a documentation bug, Rainer has raised a PR and will be resolved soon -
> >> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
> >> multiple times.
> >>
> >>
> >> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
> >> rsyslog@lists.adiscon.com> wrote:
> >>
> >>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> >>>
> >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
> >>>> "Note that mmnormalize should only be called once on each message.
> >>>> Behaviour is undefined if multiple calls to mmnormalize happen for the
> >>>> same message."
> >>> I called that out and it should have been removed from current
> >>> documentation.
> >>>
> >>>> From my tests it turns out that even with direct queue the message gets
> >>>> copied when entering a separate queue and thus the results are not
> >>>> inherited on the ruleset (and queue) exit.
> >>>>
> >>>> So it seems to be the mmnormalize that's causing the OP's variable to be
> >>>> retained after the ruleset exit.
> >>> but if you don't specify any queue at all, then things inside a ruleset
> >>> will
> >>> affect things outside the ruleset.
> >>>
> >>> David Lang
> >>>
> >>>> On 16.09.2021 20:54, David Lang wrote:
> >>>>> There is no reason you can't call mmnormalize multiple times, you can
> >>>>> specify where to put the results (by default, they go under $!)
> >>>>>
> >>>>> I'm not really sure what a direct queue on a ruleset means, you should
> >>>>> just call the ruleset without it having any queue on it.
> >>>>>
> >>>>> David Lang
> >>>>>
> >>>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> >>>>>
> >>>>>> I believe it's not about queue, but about mmnormalize. I haven't used
> >>>>>> mmnormalize much myself but from the docs I understand that
> >>>>>> mmnormalize should only be called once on a message and is supposed
> >>>>>> to parse the message into a set of "global" properties.
> >>>>>>
> >>>>>>
> >>>>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
> >>>>>>> Hi Rsyslog Users,
> >>>>>>>
> >>>>>>> I am trying to understand how queues work in Rsyslog. In case of
> >>>>>>> non-direct
> >>>>>>> queues, message is copied and placed in the queue and message
> >>>>>>> modifications
> >>>>>>> done by queue workers won't have any impact on the original message.
> >>>>>>> Makes
> >>>>>>> sense here.
> >>>>>>>
> >>>>>>> However, I am a little confused when it comes to direct Queue. What
> >>>>>>> happens in Direct Queue ? Will the message be copied or is it the same
> >>>>>>> message ? I tested out two scenarios (wrt rulesets and actions) and
> >>>>>>> both
> >>>>>>> of them gave different results.
> >>>>>>>
> >>>>>>> When ruleset is backed by a direct Queue, message modifications done
> >>> in
> >>>>>>> ruleset don't reflect back in original flow where as in case of
> >>> actions
> >>>>>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
> >>>>>>> Queue, message modifications done with action reflects in original
> >>> flow
> >>>>>>> Scenario 1:
> >>>>>>>
> >>>>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
> >>>>>>>
> >>>>>>> ruleset(name="relay.htp1" queue.type="Direct") {
> >>>>>>> call rs1
> >>>>>>> * // $!var1, $.var2 aren't available here*
> >>>>>>> action(type="omfile" file="/tmp/output.log" template="abc")
> >>>>>>> call relay.htp
> >>>>>>> }
> >>>>>>>
> >>>>>>> ruleset(name="rs1" queue.type="Direct"){
> >>>>>>> set $!var1 = "hello";
> >>>>>>> set $.var2 = "bye";
> >>>>>>> }
> >>>>>>>
> >>>>>>> input(type="imfile"
> >>>>>>> File="/tmp/input.log"
> >>>>>>> Ruleset="relay.htp1"
> >>>>>>> Tag="tag")
> >>>>>>>
> >>>>>>>
> >>>>>>> Scenario 2:
> >>>>>>>
> >>>>>>> module(load = "mmnormalize")
> >>>>>>> ruleset(name = "relay.htp1" queue.type="Direct") {
> >>>>>>> action(type = "mmnormalize"
> >>>>>>> ruleBase="/etc/rsyslog.d/service.rulebase"
> >>>>>>> path="$!msg")
> >>>>>>> * // $!msg will be available here even though action is backed by a
> >>>>>>> default Queue. *
> >>>>>>> }
> >>>>>>>
> >>>>>>> input(type="imfile"
> >>>>>>> File="/tmp/input.log"
> >>>>>>> Ruleset="relay.htp1"
> >>>>>>> Tag="tag")
> >>>>>>>
> >>>>>>>
> >>>>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> >>>>>>> available when both are using Direct Queue. Am I missing something
> >>>>>>> here ?
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com/professional-services/
> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>>>>> POST if you DON'T LIKE THAT.
> >>>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >>> of
> >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T
> >>>> LIKE THAT.
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T LIKE THAT.
> >>
> >>
> >> --
> >> Regards,
> >> Rajesh KSV
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On 17.09.2021 09:32, Rainer Gerhards wrote:
> El vie, 17 sept 2021 a las 9:15, Mariusz Kruk via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>> Hmm...
>>
>> OK, so a simple setup of:
>>
>> [cut]
>>
>> ruleset(name="whatever" queue.type="direct") {
>>
>> set $.msg="whatever";
>>
>> }
>>
>> set $.msg=$msg;
>>
>> call whatever
>>
>> template using $.msg
>>
>> output with the template
>>
>> [cut]
>>
>> Which from my tests shows that $.msg retains the $msg value even though
>> it was supposed to be overwritten by the "whatever" ruleset is caused
>> not by the queue mechanics but by non-linearity of execution and race?
> I'll check. Maybe my memory served me incorrectly - but I don't think so.
>
> Rainer
>
Sure. I'm not the one to argue with you about the internals of the code,
especially when I haven't looked into it ;-)

But - just to check - I did a config of:

[typical standard fedora rsyslog at the beginning and then:]

ruleset(name="whatever" queue.type="Direct") {
    set $.msg="static value";
}

template(name="dumper" type="list") {
    property(name=".msg")
    constant(value="\n")
}
set $.msg=$msg;
call whatever
action(type="omfile" file="/tmp/log.log" template="dumper")
action(type="omfile" file="/tmp/rsyslog.debug"
template="RSYSLOG_DebugFormat")

In the log.log file I got a normal log of system events (obviously
without the headers since the template includes only $.msg which is
assigned a value of $msg).

In the debug file I have entries like

Debug line with all properties:
FROMHOST: 'scmkrlx', fromhost-ip: '127.0.0.1', HOSTNAME: 'scmkrlx', PRI:
86, syslogtag 'sudo:', programname: 'sudo', APP-NAME: 'sudo', PROCID:
'-', MSGID: '-', TIMESTAMP: 'Sep 17 10:28:28', STRUCTURED-DATA: '-',
msg: ' pam_unix(sudo:session): session opened for user root(uid=0) by
(uid=1000)'
escaped msg: ' pam_unix(sudo:session): session opened for user
root(uid=0) by (uid=1000)'
inputname: imuxsock rawmsg: '<86>Sep 17 10:28:28 sudo:
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)'
$!:
$.:{ "msg": " pam_unix(sudo:session): session opened for user
root(uid=0) by (uid=1000)" }

As you can clearly see, the $.msg is getting assigned the $msg value and
_not_ overwritten by the static value.

But if I omit the queue type completely from the ruleset definition

ruleset(name="whatever") {
    set $.msg="static value";
}

The rest of the config being the same, I get:

Debug line with all properties:
FROMHOST: 'scmkrlx', fromhost-ip: '127.0.0.1', HOSTNAME: 'scmkrlx', PRI:
86, syslogtag 'sudo:', programname: 'sudo', APP-NAME: 'sudo', PROCID:
'-', MSGID: '-', TIMESTAMP: 'Sep 17 10:36:13', STRUCTURED-DATA: '-',
msg: ' pam_unix(sudo:session): session opened for user root(uid=0) by
(uid=1000)'
escaped msg: ' pam_unix(sudo:session): session opened for user
root(uid=0) by (uid=1000)'
inputname: imuxsock rawmsg: '<86>Sep 17 10:36:13 sudo:
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)'
$!:
$.:{ "msg": "static value" }
$/:

And of course in the "main" log I get only the lines with "static value"
instead of the original $msg.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
> > Which from my tests shows that $.msg retains the $msg value even though
> > it was supposed to be overwritten by the "whatever" ruleset is caused
> > not by the queue mechanics but by non-linearity of execution and race?
>
> I'll check. Maybe my memory served me incorrectly - but I don't think so.

OK, it's actually a kind of bug, but one that we better not fix.

Here is what happens:
Call, by design, is able to call a rule set either synchronously or
asynchronously. We did this, because practice showed that both modes
are needed. For various reasons (would need to dig out, but IIRC
"backwards compatibility" as among them), we decided to make async
calls if the ruleset has a queue assigned and sync if not.

Now comes the bug: to know if a "queue is assigned" we just check if
queue parameters are given. I overlooked the case of someone
explicitly specifying a "direct queue", aka "no queue". As such,
queue="direct" triggers async calls. That in turn means while the
actually the same message object IS modified, the mentioned race
occurs. In many cases (OS scheduling reasons), the next action will be
executed before the async rule set (active thread vs. waiting thread).
So the output uses the old value before the ruleset sets the new one.

Can we fix this?
Yes, it's easy, just check queue type and if it is "direct", do a sync call.

BUT: this would potentially break existing configurations, something
we do only if there is really a very good reason to do so. I do not
see the case strong enough for a breaking fix.

I think I will add a warning message when a direct queue type is
detected but explicitly set. So users can become aware of the issue.

Any objections?

Rainer


>
> Rainer
>
> >
> > MK
>
>
> >
> > On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote:
> > > The default queue mode is "direct" - no matter if it is explicitly
> > > specified or not.
> > >
> > > There are indeed semantic differences between the "direct" queue
> > > (a.k.a. "no real queue exists") and any other (real) queue type. As
> > > you say, in direct mode, the message object is NOT copied. So anything
> > > done to it, will potentially affect other uses of the very same
> > > message object. I real queues, the message is copied and thus no
> > > processing outside the scope of the queue is affected.
> > >
> > > NOTE WELL: if you have parts with direct AND real queues AND modify
> > > messages in "direct queue constructs" AND have actions/rulesets with
> > > real queues running in parallel, you have a race. If the modification
> > > made by the direct queue part depends on whether processing of the
> > > real queues has already started. For "regular" rule sets this is no
> > > problem, because main processing is sequential. However, if you do
> > > very complex setups with multipele rule sets calling into each other
> > > and using different queue modes, this can affect you.
> > >
> > > It is best, also for maintainability, to have a simple sequential
> > > structure in top level processing, spawning off topic rule sets (if at
> > > all required). then do all "global" modifications at the beginning.
> > > And do local (intentionally non outside-visible) modifications in the
> > > topic rule sets.
> > >
> > > I hope this helps,
> > > Rainer
> > >
> > > El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog
> > > (<rsyslog@lists.adiscon.com>) escribió:
> > >> Agreed. We don't need a direct queue before ruleset. I added it only to
> > >> understand what's going with DIRECT Queues (Since its a very novel concept
> > >> - Haven't heard something like this before)
> > >>
> > >> If you remove the DIRECT queue from ruleset(in Scenario I), then
> > >> modifications done in ruleset are visible to the main lane. It's working as
> > >> expected. But since I am interested in understanding DIRECT Queues, I added
> > >> it.
> > >>
> > >>>>> Also, although the maintainers will need to say for sure, user variable
> > >> defined in a call may not be visible to the parent. Same as in most
> > >> programming languages.
> > >> Rsyslog works in a different way, in few scenarios, user variables defined
> > >> in a call will be visible to the parent (If ruleset is not backed by
> > >> Queue). So, its not similar to other languages ;)
> > >>
> > >>>>> "Note that mmnormalize should only be called once on each message.
> > >> Behaviour is undefined if multiple calls to mmnormalize happen for the
> > >> same message."
> > >> Its a documentation bug, Rainer has raised a PR and will be resolved soon -
> > >> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
> > >> multiple times.
> > >>
> > >>
> > >> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
> > >> rsyslog@lists.adiscon.com> wrote:
> > >>
> > >>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> > >>>
> > >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
> > >>>> "Note that mmnormalize should only be called once on each message.
> > >>>> Behaviour is undefined if multiple calls to mmnormalize happen for the
> > >>>> same message."
> > >>> I called that out and it should have been removed from current
> > >>> documentation.
> > >>>
> > >>>> From my tests it turns out that even with direct queue the message gets
> > >>>> copied when entering a separate queue and thus the results are not
> > >>>> inherited on the ruleset (and queue) exit.
> > >>>>
> > >>>> So it seems to be the mmnormalize that's causing the OP's variable to be
> > >>>> retained after the ruleset exit.
> > >>> but if you don't specify any queue at all, then things inside a ruleset
> > >>> will
> > >>> affect things outside the ruleset.
> > >>>
> > >>> David Lang
> > >>>
> > >>>> On 16.09.2021 20:54, David Lang wrote:
> > >>>>> There is no reason you can't call mmnormalize multiple times, you can
> > >>>>> specify where to put the results (by default, they go under $!)
> > >>>>>
> > >>>>> I'm not really sure what a direct queue on a ruleset means, you should
> > >>>>> just call the ruleset without it having any queue on it.
> > >>>>>
> > >>>>> David Lang
> > >>>>>
> > >>>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
> > >>>>>
> > >>>>>> I believe it's not about queue, but about mmnormalize. I haven't used
> > >>>>>> mmnormalize much myself but from the docs I understand that
> > >>>>>> mmnormalize should only be called once on a message and is supposed
> > >>>>>> to parse the message into a set of "global" properties.
> > >>>>>>
> > >>>>>>
> > >>>>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
> > >>>>>>> Hi Rsyslog Users,
> > >>>>>>>
> > >>>>>>> I am trying to understand how queues work in Rsyslog. In case of
> > >>>>>>> non-direct
> > >>>>>>> queues, message is copied and placed in the queue and message
> > >>>>>>> modifications
> > >>>>>>> done by queue workers won't have any impact on the original message.
> > >>>>>>> Makes
> > >>>>>>> sense here.
> > >>>>>>>
> > >>>>>>> However, I am a little confused when it comes to direct Queue. What
> > >>>>>>> happens in Direct Queue ? Will the message be copied or is it the same
> > >>>>>>> message ? I tested out two scenarios (wrt rulesets and actions) and
> > >>>>>>> both
> > >>>>>>> of them gave different results.
> > >>>>>>>
> > >>>>>>> When ruleset is backed by a direct Queue, message modifications done
> > >>> in
> > >>>>>>> ruleset don't reflect back in original flow where as in case of
> > >>> actions
> > >>>>>>> (such mmnormalize, mmkubernetes) which are by default backed by direct
> > >>>>>>> Queue, message modifications done with action reflects in original
> > >>> flow
> > >>>>>>> Scenario 1:
> > >>>>>>>
> > >>>>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
> > >>>>>>>
> > >>>>>>> ruleset(name="relay.htp1" queue.type="Direct") {
> > >>>>>>> call rs1
> > >>>>>>> * // $!var1, $.var2 aren't available here*
> > >>>>>>> action(type="omfile" file="/tmp/output.log" template="abc")
> > >>>>>>> call relay.htp
> > >>>>>>> }
> > >>>>>>>
> > >>>>>>> ruleset(name="rs1" queue.type="Direct"){
> > >>>>>>> set $!var1 = "hello";
> > >>>>>>> set $.var2 = "bye";
> > >>>>>>> }
> > >>>>>>>
> > >>>>>>> input(type="imfile"
> > >>>>>>> File="/tmp/input.log"
> > >>>>>>> Ruleset="relay.htp1"
> > >>>>>>> Tag="tag")
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Scenario 2:
> > >>>>>>>
> > >>>>>>> module(load = "mmnormalize")
> > >>>>>>> ruleset(name = "relay.htp1" queue.type="Direct") {
> > >>>>>>> action(type = "mmnormalize"
> > >>>>>>> ruleBase="/etc/rsyslog.d/service.rulebase"
> > >>>>>>> path="$!msg")
> > >>>>>>> * // $!msg will be available here even though action is backed by a
> > >>>>>>> default Queue. *
> > >>>>>>> }
> > >>>>>>>
> > >>>>>>> input(type="imfile"
> > >>>>>>> File="/tmp/input.log"
> > >>>>>>> Ruleset="relay.htp1"
> > >>>>>>> Tag="tag")
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
> > >>>>>>> available when both are using Direct Queue. Am I missing something
> > >>>>>>> here ?
> > >>>>>>>
> > >>>>>> _______________________________________________
> > >>>>>> rsyslog mailing list
> > >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>>>> http://www.rsyslog.com/professional-services/
> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > >>>>>> POST if you DON'T LIKE THAT.
> > >>>>>>
> > >>>> _______________________________________________
> > >>>> rsyslog mailing list
> > >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>> http://www.rsyslog.com/professional-services/
> > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > >>> of
> > >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T
> > >>>> LIKE THAT.
> > >>> _______________________________________________
> > >>> rsyslog mailing list
> > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> http://www.rsyslog.com/professional-services/
> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T LIKE THAT.
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Rajesh KSV
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On 17.09.2021 10:53, Rainer Gerhards wrote:
>
> BUT: this would potentially break existing configurations, something
> we do only if there is really a very good reason to do so. I do not
> see the case strong enough for a breaking fix.
>
> I think I will add a warning message when a direct queue type is
> detected but explicitly set. So users can become aware of the issue.
>
> Any objections?
>
> Rainer
>
As long as the behaviour is documented and predictable/understandable I
think it's a good solution.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On Fri, 17 Sep 2021, Rainer Gerhards via rsyslog wrote:

> Can we fix this?
> Yes, it's easy, just check queue type and if it is "direct", do a sync call.
>
> BUT: this would potentially break existing configurations, something
> we do only if there is really a very good reason to do so. I do not
> see the case strong enough for a breaking fix.
>
> I think I will add a warning message when a direct queue type is
> detected but explicitly set. So users can become aware of the issue.
>
> Any objections?

given that the result of specifying queue=direct is an undefined race condition,
I think we need to at least add a big warning, and don't think it would be
unreasonable to change this behavior to make it defined (turn it into a sync
call)

in the docs, we tell people to use a queue type of linkedlist when they want
things to be async (at some point we should have examples that say fixedarray as
people fall into the trap of thinking only linkedlist does this), but nowhere do
we say that queue type direct would make anything async

yes, it is a behavior change, which normally I oppose, but in this case it seems
to be a behavior change away from something that is arguably against what the
docs say, is unlikly to be used in the real world, and currently produces
unpredictable results

in the race condition, could the current behavior trigger a different message
being changed? or writes to freed memory if the called ruleset is modifying the
message object after the parent has finished processing the message entirely? or
is it copied so that it's safe, just won't popogate changes back? (unexpectedly
per the docs)

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
On Fri, 17 Sep 2021, Mariusz Kruk via rsyslog wrote:

>>> From my tests it turns out that even with direct queue the message gets
>>> copied when entering a separate queue and thus the results are not
>>> inherited on the ruleset (and queue) exit.
>>>
>>> So it seems to be the mmnormalize that's causing the OP's variable to be
>>> retained after the ruleset exit.
>>
>> but if you don't specify any queue at all, then things inside a ruleset
>> will affect things outside the ruleset.
>>
> Sure. Then the flow inside a ruleset - which happens within the same queue as
> the "outside", will affect the variables seen in the queue. I use it heavily
> ;-)
>
> But the OP asked about the direct queue and it seems that this queue type is
> not "special" in anyway - it also creates a new scope.

the issue is that throughout the documentation, we say that not specifying a
queue is the same as specifying a queue type of direct. In this case, it's not.

so we can document this case where real-world behavior doesn't match the docs,
and add a large warning about it.

or

we can fix it to match the docs, and add a warning about the behavior change.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
This is what I now emit:

on or before line 26: rule set 'rs1' has queue type 'direct'
explicitly set, ruleset will still be called asynchronously. This is
often not what you want. If so, remove 'queue.type="direct"' from the
ruleset definition.

Rainer

El vie, 17 sept 2021 a las 10:57, Mariusz Kruk via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> On 17.09.2021 10:53, Rainer Gerhards wrote:
> >
> > BUT: this would potentially break existing configurations, something
> > we do only if there is really a very good reason to do so. I do not
> > see the case strong enough for a breaking fix.
> >
> > I think I will add a warning message when a direct queue type is
> > detected but explicitly set. So users can become aware of the issue.
> >
> > Any objections?
> >
> > Rainer
> >
> As long as the behaviour is documented and predictable/understandable I
> think it's a good solution.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
> the issue is that throughout the documentation, we say that not specifying a
> queue is the same as specifying a queue type of direct. In this case, it's not.

no, no!

All is right, except that the "call" statement calling a ruleset async
even when only a queue type direct is set.

Behaviour of queues is as it always was, and direct does not de-couple.

It is "call" which is executed on a different thread when we detect
queue parameters on a ruleset. Anything else is correct. This can be
clarified here:

https://www.rsyslog.com/doc/master/rainerscript/rainerscript_call.html

but nowhere else!

Tech details: if call detects a queue on the ruleset, it posts the
message to that queue. Otherwise, it runs the ruleset as a subroutine
(links directly to the rulesets AST). This ensures synchronicity even
when running on multiple threads.

Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
David,

I reconsidered your point and am currently having a more in-depth
look. Maybe something is indeed fishy... Just for everyone's info.
Will post more when I know more ;-)

Rainer

El vie, 17 sept 2021 a las 11:21, Rainer Gerhards
(<rgerhards@hq.adiscon.com>) escribió:
>
> > the issue is that throughout the documentation, we say that not specifying a
> > queue is the same as specifying a queue type of direct. In this case, it's not.
>
> no, no!
>
> All is right, except that the "call" statement calling a ruleset async
> even when only a queue type direct is set.
>
> Behaviour of queues is as it always was, and direct does not de-couple.
>
> It is "call" which is executed on a different thread when we detect
> queue parameters on a ruleset. Anything else is correct. This can be
> clarified here:
>
> https://www.rsyslog.com/doc/master/rainerscript/rainerscript_call.html
>
> but nowhere else!
>
> Tech details: if call detects a queue on the ruleset, it posts the
> message to that queue. Otherwise, it runs the ruleset as a subroutine
> (links directly to the rulesets AST). This ensures synchronicity even
> when running on multiple threads.
>
> Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Message modification in DIRECT Queues have different semantics when using with rulesets vs actions [ In reply to ]
OK, actually a tougher issue than I thought. After a couple of hours
of testing, it looks like fixing the bug is good. There was some
potential for a very slight optimization, but I left it out as it
would have caused quite some work to ensure that the rest of the
engine works well (hint: testing showed it did not work without
further changes). So I am not 100% happy, but I think this issue is
solved. More details from commit text:

https://github.com/rsyslog/rsyslog/pull/4688

The call rscript statement is able to call a rule set either synchronously or
asynchronously. We did this, because practice showed that both modes
are needed. For various reasons we decided to make async
calls if the ruleset has a queue assigned and sync if not.

To know if a "queue is assigned" we just checked if queue parameters were
given. It was overlookeded the case of someone explicitly specifying a
"direct queue", aka "no queue". As such, queue="direct" triggered async
calls. That in turn meant that when a write operation to a variable was
made inside that rule set, other rulesets could or could not see the
write. While if was often not seen, this was a data race where the
change could also be seen by the outside.

This is now fixed. No matter if queue.type="direct" is specified or
left out, the call will always by synchronous. Any values written to
variables will also be seen by the "outside world" in later processing
stages.

Note that this has some potential to BREAK EXISTING CONFIGURATIONS.
We deem this acceptable because:

1. this was racy at all, so unexpected behaviour could alwas occur
2. it is actually unlikely that someone used the triggering conditions
in practice. But we can not outrule this, especially when the
configuration was auto-generated.

Potential compatibility issues can be solved by defining a small
array-memory queue on the ruleset in question instead of specifying
direct type.

Again, we expect that almost all users will never experience any
problems. If you do, however, please let us know: we may add an
option to re-enable the bug.


Thanks everyone for being persistent.

Rainer

El vie, 17 sept 2021 a las 14:41, Rainer Gerhards
(<rgerhards@hq.adiscon.com>) escribió:
>
> David,
>
> I reconsidered your point and am currently having a more in-depth
> look. Maybe something is indeed fishy... Just for everyone's info.
> Will post more when I know more ;-)
>
> Rainer
>
> El vie, 17 sept 2021 a las 11:21, Rainer Gerhards
> (<rgerhards@hq.adiscon.com>) escribió:
> >
> > > the issue is that throughout the documentation, we say that not specifying a
> > > queue is the same as specifying a queue type of direct. In this case, it's not.
> >
> > no, no!
> >
> > All is right, except that the "call" statement calling a ruleset async
> > even when only a queue type direct is set.
> >
> > Behaviour of queues is as it always was, and direct does not de-couple.
> >
> > It is "call" which is executed on a different thread when we detect
> > queue parameters on a ruleset. Anything else is correct. This can be
> > clarified here:
> >
> > https://www.rsyslog.com/doc/master/rainerscript/rainerscript_call.html
> >
> > but nowhere else!
> >
> > Tech details: if call detects a queue on the ruleset, it posts the
> > message to that queue. Otherwise, it runs the ruleset as a subroutine
> > (links directly to the rulesets AST). This ensures synchronicity even
> > when running on multiple threads.
> >
> > Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.