Mailing List Archive

yes and yes - I suggest a simple script to run during the deployment process.

Rainer

El vie, 17 feb 2023 a las 16:13, Mariusz Kruk via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Just double-checking.
>
> Please confirm that I see it right and there is no way of "looping"
> outside of the message processing mechanism. I mean - if I have several
> dozens of lookup files I have to explicitly create those lookups one by
> one, I can't do a "loop" over some set of names to automatically create
> those lookups, right?
>
> And on a similar note - I can't dynamically create values for some
> configuration parts like - for example - input name. Use case - same
> config deployed over several hosts and I want to have inputs named
> differently so that when I capture stats I can easiliy distinguish
> between the nodes. I suppose provisioning tool like ansible/chef/puppet
> is needed here, right?
>
> Best regards and have a great weekend ;-)
>
> MK
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Yep, I will have to resort to either a script or an ansible playbook (in
this case) to create the configs.

Just wanted to be absolutely sure that I hadn't missed something in the
docs.

Thanks!

On 17.02.2023 17:41, Rainer Gerhards via rsyslog wrote:
> yes and yes - I suggest a simple script to run during the deployment process.
>
> Rainer
>
> El vie, 17 feb 2023 a las 16:13, Mariusz Kruk via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>> Just double-checking.
>>
>> Please confirm that I see it right and there is no way of "looping"
>> outside of the message processing mechanism. I mean - if I have several
>> dozens of lookup files I have to explicitly create those lookups one by
>> one, I can't do a "loop" over some set of names to automatically create
>> those lookups, right?
>>
>> And on a similar note - I can't dynamically create values for some
>> configuration parts like - for example - input name. Use case - same
>> config deployed over several hosts and I want to have inputs named
>> differently so that when I capture stats I can easiliy distinguish
>> between the nodes. I suppose provisioning tool like ansible/chef/puppet
>> is needed here, right?
>>
>> Best regards and have a great weekend ;-)
>>
>> MK
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

correct, the rsyslog config is not a programming language

what sort of lookups are you trying to create?

David Lang

On Fri, 17 Feb 2023, Mariusz Kruk via rsyslog wrote:

> Date: Fri, 17 Feb 2023 16:13:29 +0100
> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> To: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: [rsyslog] Non-message based looping
>
> Just double-checking.
>
> Please confirm that I see it right and there is no way of "looping" outside
> of the message processing mechanism. I mean - if I have several dozens of
> lookup files I have to explicitly create those lookups one by one, I can't do
> a "loop" over some set of names to automatically create those lookups, right?
>
> And on a similar note - I can't dynamically create values for some
> configuration parts like - for example - input name. Use case - same config
> deployed over several hosts and I want to have inputs named differently so
> that when I capture stats I can easiliy distinguish between the nodes. I
> suppose provisioning tool like ansible/chef/puppet is needed here, right?
>
> Best regards and have a great weekend ;-)
>
> MK
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

To some extent, it is ;-) But seriously, i terms of event processing it
resembles... awk, for example.

Actually I'm not trying to create lookups, I have them. I was just
thinking if there was any way to simplify managing my config a bit.

I have a multi-layered setup routing events from several environments
into different solutions. And I have lookups defining what goes where.
For now for each environment I simply define a lookup file with
additional config snippet loading the lookup definition.

So I have many small .conf files containing something like:

lookup_table(name="template-ip-dest"
file="/etc/rsyslog.d/lookups.d/template/ip-dest.json" reloadOnHUP="on")

And each file has "template" substituted with a proper environment name.

If I add a new environment, I add a new json file with the lookup
contents and a new .conf file for loading said lookup.

It's just that if I was able to do something like (pseudocode)

for A in (env1,env2,env3...) do lookup_table(name=$A & "-ip-dest"
file="/etc/rsyslog.d/lookups.d/" & $A &"/ip-dest.json" reloadOnHUP="on")

that would make my life a bit easier.

But yes, I admit that my use case is very unusual so I didn't expect
such functionality to exist really. As I already said - I just wanted to
be 100% sure I hadn't missed it in case it existed after all.

MK

On 17.02.2023 21:57, David Lang wrote:
> correct, the rsyslog config is not a programming language
>
> what sort of lookups are you trying to create?
>
> David Lang
>
> On Fri, 17 Feb 2023, Mariusz Kruk via rsyslog wrote:
>
>> Date: Fri, 17 Feb 2023 16:13:29 +0100
>> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> To: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: [rsyslog] Non-message based looping
>>
>> Just double-checking.
>>
>> Please confirm that I see it right and there is no way of "looping"
>> outside of the message processing mechanism. I mean - if I have
>> several dozens of lookup files I have to explicitly create those
>> lookups one by one, I can't do a "loop" over some set of names to
>> automatically create those lookups, right?
>>
>> And on a similar note - I can't dynamically create values for some
>> configuration parts like - for example - input name. Use case - same
>> config deployed over several hosts and I want to have inputs named
>> differently so that when I capture stats I can easiliy distinguish
>> between the nodes. I suppose provisioning tool like
>> ansible/chef/puppet is needed here, right?
>>
>> Best regards and have a great weekend ;-)
>>
>> MK
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Sat, 18 Feb 2023, Mariusz Kruk via rsyslog wrote:

> To some extent, it is ;-) But seriously, i terms of event processing it
> resembles... awk, for example.

true, but one major thing is that the config is processed in two very different
ways.

1. at startup, the actions, variable assignments, conditional logic (which is
based on variables) is all ignored, it's just looking for startup-time config
lines, everything else goes into the default ruleset (unless it's defined into
it's own ruleset explicitly), but the startup-time configs are all parsed, no
matter where they appear in relation to any ruleset definitions.

2. when processing a log messages, the startup-time parameters are completely
ignored and only what's defined in the rulesets matter (default or explicitly
defined)

> Actually I'm not trying to create lookups, I have them. I was just thinking
> if there was any way to simplify managing my config a bit.
>
> I have a multi-layered setup routing events from several environments into
> different solutions. And I have lookups defining what goes where. For now for
> each environment I simply define a lookup file with additional config snippet
> loading the lookup definition.
>
> So I have many small .conf files containing something like:
>
> lookup_table(name="template-ip-dest"
> file="/etc/rsyslog.d/lookups.d/template/ip-dest.json" reloadOnHUP="on")
>
> And each file has "template" substituted with a proper environment name.
>
> If I add a new environment, I add a new json file with the lookup contents
> and a new .conf file for loading said lookup.
>
> It's just that if I was able to do something like (pseudocode)
>
> for A in (env1,env2,env3...) do lookup_table(name=$A & "-ip-dest"
> file="/etc/rsyslog.d/lookups.d/" & $A &"/ip-dest.json" reloadOnHUP="on")
>
> that would make my life a bit easier.
>
> But yes, I admit that my use case is very unusual so I didn't expect such
> functionality to exist really. As I already said - I just wanted to be 100%
> sure I hadn't missed it in case it existed after all.

the lookup table functionality is extremely scalable (I did a test with over
1400 parsing rules at one point, and the best-to-worst case match was a 30%
lookup difference)

so it may be that you can combine the lookups into one table. you would still
need to combine the json snippets into valid json, but that's probably easier
than having to write different rsyslog configs.

if you can key the lookup by adding a prefix to the value you are looking up
that you can predict ahead of time (could be an environment variable, or the
local hostname) you can possibly do this with no changes to the rsyslog config.

David Lang

> MK
>
> On 17.02.2023 21:57, David Lang wrote:
>> correct, the rsyslog config is not a programming language
>>
>> what sort of lookups are you trying to create?
>>
>> David Lang
>>
>> On Fri, 17 Feb 2023, Mariusz Kruk via rsyslog wrote:
>>
>>> Date: Fri, 17 Feb 2023 16:13:29 +0100
>>> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>> To: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: [rsyslog] Non-message based looping
>>>
>>> Just double-checking.
>>>
>>> Please confirm that I see it right and there is no way of "looping"
>>> outside of the message processing mechanism. I mean - if I have several
>>> dozens of lookup files I have to explicitly create those lookups one by
>>> one, I can't do a "loop" over some set of names to automatically create
>>> those lookups, right?
>>>
>>> And on a similar note - I can't dynamically create values for some
>>> configuration parts like - for example - input name. Use case - same
>>> config deployed over several hosts and I want to have inputs named
>>> differently so that when I capture stats I can easiliy distinguish between
>>> the nodes. I suppose provisioning tool like ansible/chef/puppet is needed
>>> here, right?
>>>
>>> Best regards and have a great weekend ;-)
>>>
>>> MK
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 18.02.2023 11:45, David Lang wrote:
> On Sat, 18 Feb 2023, Mariusz Kruk via rsyslog wrote:
>
>> To some extent, it is ;-) But seriously, i terms of event processing
>> it resembles... awk, for example.
>
> true, but one major thing is that the config is processed in two very
> different ways.

Yes, I know. And as I already wrote - I thought so but wanted to be 110%
sure. :-)


>> Actually I'm not trying to create lookups, I have them. I was just
>> thinking if there was any way to simplify managing my config a bit.
>>
>> I have a multi-layered setup routing events from several environments
>> into different solutions. And I have lookups defining what goes
>> where. For now for each environment I simply define a lookup file
>> with additional config snippet loading the lookup definition.
>>
>> So I have many small .conf files containing something like:
>>
>> lookup_table(name="template-ip-dest"
>> file="/etc/rsyslog.d/lookups.d/template/ip-dest.json" reloadOnHUP="on")
>>
>> And each file has "template" substituted with a proper environment name.
>>
>> If I add a new environment, I add a new json file with the lookup
>> contents and a new .conf file for loading said lookup.
>>
>> It's just that if I was able to do something like (pseudocode)
>>
>> for A in (env1,env2,env3...) do lookup_table(name=$A & "-ip-dest"
>> file="/etc/rsyslog.d/lookups.d/" & $A &"/ip-dest.json" reloadOnHUP="on")
>>
>> that would make my life a bit easier.
>>
>> But yes, I admit that my use case is very unusual so I didn't expect
>> such functionality to exist really. As I already said - I just wanted
>> to be 100% sure I hadn't missed it in case it existed after all.
>
> the lookup table functionality is extremely scalable (I did a test
> with over 1400 parsing rules at one point, and the best-to-worst case
> match was a 30% lookup difference)

I have around 3800 entries in three types of lookups (almost 160 files
altogether). But it's not as much about scalability, as about
maintainability. I could rework my config to fit "everything" into a
single lookup (or at least just a few of them) but it would be much more
painful to maintain such config. So in this case I'm much more
comfortable with many small lookups. It's just that once in a while if I
add new environment I have to add this new definition of a lookup or
two. Again - it's mostly a convenience thing. It would be easier if it
"happened by itself" but I won't cry if it doesn't ;-)

MK
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.