Mailing List Archive

Effective way to cut a field from event?
I have a misbehaving source which is sending a non-compliant CEF events.
The events have an extra field where it shouldn't and therefore the
events get misinterpreted further down the pipeline.

So the question is - what is the most "rsyslog-way" to delete a single
pipe-delimited field from the middle of the event.

The event typically looks this way:

<PRI>Date time host
CEF:0|Company|Product|Version|Module|extra_field|alert|priority|extended_part

I want to cut the extra_field away.

Since the fields are pipe-delimited, it's tempting to use field() but
there are two problems with this approach:

1. As far as I remember, you can't return a set of fields with field().
Just a single field. So I'd have to iterate over some counter and
assemble the resulting event from single fields. Not very nice.

2. If the extended_part contained by any chance a pipe character, it
would cut tha part short. So I can't even make a static list of field to
iterate over but I'd have to find how many fields are there. Even uglier.

The other approach I could consider is obviously re_match(). It seems
simpler in terms of the idea but regex (and I think I'd need to call it
twice to match both sides of the field I want to cut) seems a bit heavy
performancewise for such a small task.

Any better ideas?

MK

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Effective way to cut a field from event? [ In reply to ]
Hi Mariusz!

I'd suggest you try with mmfields first (
https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmfields.html).
If no luck then I'd go for mmnormalize.

This way you can extract all the fields required and then use a template to
put just fields you need back into a message.
I guess this is the only way to do it properly w/o any side effects.

On Wed, 15 Feb 2023 at 19:36, Mariusz Kruk via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> I have a misbehaving source which is sending a non-compliant CEF events.
> The events have an extra field where it shouldn't and therefore the
> events get misinterpreted further down the pipeline.
>
> So the question is - what is the most "rsyslog-way" to delete a single
> pipe-delimited field from the middle of the event.
>
> The event typically looks this way:
>
> <PRI>Date time host
>
> CEF:0|Company|Product|Version|Module|extra_field|alert|priority|extended_part
>
> I want to cut the extra_field away.
>
> Since the fields are pipe-delimited, it's tempting to use field() but
> there are two problems with this approach:
>
> 1. As far as I remember, you can't return a set of fields with field().
> Just a single field. So I'd have to iterate over some counter and
> assemble the resulting event from single fields. Not very nice.
>
> 2. If the extended_part contained by any chance a pipe character, it
> would cut tha part short. So I can't even make a static list of field to
> iterate over but I'd have to find how many fields are there. Even uglier.
>
> The other approach I could consider is obviously re_match(). It seems
> simpler in terms of the idea but regex (and I think I'd need to call it
> twice to match both sides of the field I want to cut) seems a bit heavy
> performancewise for such a small task.
>
> Any better ideas?
>
> MK
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>


--
Yury Bushmelev
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Effective way to cut a field from event? [ In reply to ]
Nice one! Thanks for the hint since I didn't know about this module.

But it would still split my event into fields but I'd have to collect
them back one-by-one. So better than just field() alone but still not
perfect.

In this particular case I think I'll stick with re_extract() (not
re_match() as I wrote in the first mail) since I double-checked and I
have just several thousands events a day so it's not some overwhelming
volume ;-)

Thanks for the response though. Good to know about this module.

MK

On 15.02.2023 13:14, Yury Bushmelev via rsyslog wrote:
> Hi Mariusz!
>
> I'd suggest you try with mmfields first (
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmfields.html).
> If no luck then I'd go for mmnormalize.
>
> This way you can extract all the fields required and then use a template to
> put just fields you need back into a message.
> I guess this is the only way to do it properly w/o any side effects.
>
> On Wed, 15 Feb 2023 at 19:36, Mariusz Kruk via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
>> I have a misbehaving source which is sending a non-compliant CEF events.
>> The events have an extra field where it shouldn't and therefore the
>> events get misinterpreted further down the pipeline.
>>
>> So the question is - what is the most "rsyslog-way" to delete a single
>> pipe-delimited field from the middle of the event.
>>
>> The event typically looks this way:
>>
>> <PRI>Date time host
>>
>> CEF:0|Company|Product|Version|Module|extra_field|alert|priority|extended_part
>>
>> I want to cut the extra_field away.
>>
>> Since the fields are pipe-delimited, it's tempting to use field() but
>> there are two problems with this approach:
>>
>> 1. As far as I remember, you can't return a set of fields with field().
>> Just a single field. So I'd have to iterate over some counter and
>> assemble the resulting event from single fields. Not very nice.
>>
>> 2. If the extended_part contained by any chance a pipe character, it
>> would cut tha part short. So I can't even make a static list of field to
>> iterate over but I'd have to find how many fields are there. Even uglier.
>>
>> The other approach I could consider is obviously re_match(). It seems
>> simpler in terms of the idea but regex (and I think I'd need to call it
>> twice to match both sides of the field I want to cut) seems a bit heavy
>> performancewise for such a small task.
>>
>> Any better ideas?
>>
>> MK
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Effective way to cut a field from event? [ In reply to ]
the regex is going to be your best approach, it would probably be a good idea
for someone to make a field extraction that let you specify multiple fields, but
nobody has done that yet.

most of rsyslog is aimed at parsing the data into a JSON structure internally
and then using that to create the output that you want. That falls apart when
you have ordered fields like this.

David Lang

On Wed, 15 Feb 2023, Mariusz Kruk via rsyslog wrote:

> Date: Wed, 15 Feb 2023 13:38:34 +0100
> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] Effective way to cut a field from event?
>
> Nice one! Thanks for the hint since I didn't know about this module.
>
> But it would still split my event into fields but I'd have to collect them
> back one-by-one. So better than just field() alone but still not perfect.
>
> In this particular case I think I'll stick with re_extract() (not re_match()
> as I wrote in the first mail) since I double-checked and I have just several
> thousands events a day so it's not some overwhelming volume ;-)
>
> Thanks for the response though. Good to know about this module.
>
> MK
>
> On 15.02.2023 13:14, Yury Bushmelev via rsyslog wrote:
>> Hi Mariusz!
>>
>> I'd suggest you try with mmfields first (
>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmfields.html).
>> If no luck then I'd go for mmnormalize.
>>
>> This way you can extract all the fields required and then use a template to
>> put just fields you need back into a message.
>> I guess this is the only way to do it properly w/o any side effects.
>>
>> On Wed, 15 Feb 2023 at 19:36, Mariusz Kruk via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>>
>>> I have a misbehaving source which is sending a non-compliant CEF events.
>>> The events have an extra field where it shouldn't and therefore the
>>> events get misinterpreted further down the pipeline.
>>>
>>> So the question is - what is the most "rsyslog-way" to delete a single
>>> pipe-delimited field from the middle of the event.
>>>
>>> The event typically looks this way:
>>>
>>> <PRI>Date time host
>>>
>>> CEF:0|Company|Product|Version|Module|extra_field|alert|priority|extended_part
>>>
>>> I want to cut the extra_field away.
>>>
>>> Since the fields are pipe-delimited, it's tempting to use field() but
>>> there are two problems with this approach:
>>>
>>> 1. As far as I remember, you can't return a set of fields with field().
>>> Just a single field. So I'd have to iterate over some counter and
>>> assemble the resulting event from single fields. Not very nice.
>>>
>>> 2. If the extended_part contained by any chance a pipe character, it
>>> would cut tha part short. So I can't even make a static list of field to
>>> iterate over but I'd have to find how many fields are there. Even uglier.
>>>
>>> The other approach I could consider is obviously re_match(). It seems
>>> simpler in terms of the idea but regex (and I think I'd need to call it
>>> twice to match both sides of the field I want to cut) seems a bit heavy
>>> performancewise for such a small task.
>>>
>>> Any better ideas?
>>>
>>> MK
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.