Mailing List Archive

Filtering of syslog events in structured format
Hello all,


did anyone tried to do a filtering of messages in structured data
format? Seems common re_match can't be used on $structured-data
property. It doesn't show error, but I can't get any positive match. I
found there is mmpstrucdata module which can parse structured format
into JSON variable tree, but not sure if these variables could be
somehow used in RainerScript if - then declarations. Spent a lot of time
trying to find some answer in forums, but I found nothing similar to
what I need. Any ideas are welcome.


Regards,

Petr

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering of syslog events in structured format [ In reply to ]
Hello all,

finally I managed to make it work as I need. Below is snip of config I
used - just in case someone else tries to achieve something similar.


Example message:
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
[exampleSDID@32473 iut="3" eventSource="Application"
eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
/dev/pts/8

Code:
$template RFC5424-to-file,"%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
module(load="mmpstrucdata")
if $structured-data != '-' then {
    action(type="mmpstrucdata")
    if re_match($!rfc5424-sd!exampleSDID@32473!eventID, '10[0-2][0-9]')
then {
        action(type="omfile" File="/var/log/eventid_1000-1029.log"
template="RFC5424-to-file")
    }
}

Regards,

Petr

On 06.08.2019 15:34, Petr Vyhnal via rsyslog wrote:
> Hello all,
>
>
> did anyone tried to do a filtering of messages in structured data
> format? Seems common re_match can't be used on $structured-data
> property. It doesn't show error, but I can't get any positive match. I
> found there is mmpstrucdata module which can parse structured format
> into JSON variable tree, but not sure if these variables could be
> somehow used in RainerScript if - then declarations. Spent a lot of
> time trying to find some answer in forums, but I found nothing similar
> to what I need. Any ideas are welcome.
>
>
> Regards,
>
> Petr
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering of syslog events in structured format [ In reply to ]
On Wed, 7 Aug 2019, Petr Vyhnal via rsyslog wrote:

> Hello all,
>
> finally I managed to make it work as I need. Below is snip of config I
> used - just in case someone else tries to achieve something similar.
>
>
> Example message:
> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
> [exampleSDID@32473 iut="3" eventSource="Application"
> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
> /dev/pts/8
>
> Code:
> $template RFC5424-to-file,"%TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %syslogtag% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
> module(load="mmpstrucdata")
> if $structured-data != '-' then {
>     action(type="mmpstrucdata")

at this point, write the log to the template RSYSLOG_DebugFormat and you can see
how it is parsed into the $! variable namespace

>     if re_match($!rfc5424-sd!exampleSDID@32473!eventID, '10[0-2][0-9]')

you could also say

if $!rfc5424-sd!exampleSDID@32473!eventID >=10000 and $!rfc5424-sd!exampleSDID@32473!eventID <=1029

that would be more efficient than a regex

David Lang

> then {
>         action(type="omfile" File="/var/log/eventid_1000-1029.log"
> template="RFC5424-to-file")
>     }
> }
>
> Regards,
>
> Petr
>
> On 06.08.2019 15:34, Petr Vyhnal via rsyslog wrote:
>> Hello all,
>>
>>
>> did anyone tried to do a filtering of messages in structured data
>> format? Seems common re_match can't be used on $structured-data
>> property. It doesn't show error, but I can't get any positive match. I
>> found there is mmpstrucdata module which can parse structured format
>> into JSON variable tree, but not sure if these variables could be
>> somehow used in RainerScript if - then declarations. Spent a lot of
>> time trying to find some answer in forums, but I found nothing similar
>> to what I need. Any ideas are welcome.
>>
>>
>> Regards,
>>
>> Petr
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering of syslog events in structured format [ In reply to ]
Hello David,


thanks for suggestions. Actually I used RSYSLOG_DebugFormat to identify
whole CEE chain. Also thanks for that recommendation to avoiding regex,
but shared code is not exactly the one I used. I'm checking IP addresses
in my case, so this is not applicable. Maybe with some additional
functions like ipv4tonum - I'll maybe give it a try. Actually I just
found it's possible to use re_match on structured-data after all, I just
had a typo in my original regex. Anyway at least I'm a bit more familiar
with structured data processing now :-)

Regards,

Petr

On 07.08.2019 11:10, David Lang wrote:
> On Wed, 7 Aug 2019, Petr Vyhnal via rsyslog wrote:
>
>> Hello all,
>>
>> finally I managed to make it work as I need. Below is snip of config
>> I used - just in case someone else tries to achieve something similar.
>>
>>
>> Example message:
>> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
>> [exampleSDID@32473 iut="3" eventSource="Application"
>> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
>> /dev/pts/8
>>
>> Code:
>> $template RFC5424-to-file,"%TIMESTAMP:::date-rfc3339% %HOSTNAME%
>> %syslogtag% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
>> module(load="mmpstrucdata")
>> if $structured-data != '-' then {
>>     action(type="mmpstrucdata")
>
> at this point, write the log to the template RSYSLOG_DebugFormat and
> you can see how it is parsed into the $! variable namespace
>
>>     if re_match($!rfc5424-sd!exampleSDID@32473!eventID, '10[0-2][0-9]')
>
> you could also say
>
> if $!rfc5424-sd!exampleSDID@32473!eventID >=10000 and
> $!rfc5424-sd!exampleSDID@32473!eventID <=1029
>
> that would be more efficient than a regex
>
> David Lang
>
>> then {
>>         action(type="omfile" File="/var/log/eventid_1000-1029.log"
>> template="RFC5424-to-file")
>>     }
>> }
>>
>> Regards,
>>
>> Petr
>>
>> On 06.08.2019 15:34, Petr Vyhnal via rsyslog wrote:
>>> Hello all,
>>>
>>>
>>> did anyone tried to do a filtering of messages in structured data
>>> format? Seems common re_match can't be used on $structured-data
>>> property. It doesn't show error, but I can't get any positive match.
>>> I found there is mmpstrucdata module which can parse structured
>>> format into JSON variable tree, but not sure if these variables
>>> could be somehow used in RainerScript if - then declarations. Spent
>>> a lot of time trying to find some answer in forums, but I found
>>> nothing similar to what I need. Any ideas are welcome.
>>>
>>>
>>> Regards,
>>>
>>> Petr
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering of syslog events in structured format [ In reply to ]
Again I have one follow up question to anyone who might know. It's
possible somehow use "wildcards" in SD-ID declaration? Considering
example message below I can have two similar messages like:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
[exampleSDID@*32473* iut="3" eventSource="Application"
eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
/dev/pts/8

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
[exampleSDID@*32474* iut="3" eventSource="Application"
eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
/dev/pts/8


As far as SD-ID is part of the path (if
$!rfc5424-sd!exampleSDID@32473!eventID) I'd like to ask if there an
option to have a single rule which would match different SD-IDs based on
some (maybe regex or just wildcard) definition?


Regards,

Petr


On 07.08.2019 11:37, Petr Vyhnal via rsyslog wrote:
> Hello David,
>
>
> thanks for suggestions. Actually I used RSYSLOG_DebugFormat to
> identify whole CEE chain. Also thanks for that recommendation to
> avoiding regex, but shared code is not exactly the one I used. I'm
> checking IP addresses in my case, so this is not applicable. Maybe
> with some additional functions like ipv4tonum - I'll maybe give it a
> try. Actually I just found it's possible to use re_match on
> structured-data after all, I just had a typo in my original regex.
> Anyway at least I'm a bit more familiar with structured data
> processing now :-)
>
> Regards,
>
> Petr
>
> On 07.08.2019 11:10, David Lang wrote:
>> On Wed, 7 Aug 2019, Petr Vyhnal via rsyslog wrote:
>>
>>> Hello all,
>>>
>>> finally I managed to make it work as I need. Below is snip of config
>>> I used - just in case someone else tries to achieve something similar.
>>>
>>>
>>> Example message:
>>> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
>>> [exampleSDID@32473 iut="3" eventSource="Application"
>>> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
>>> /dev/pts/8
>>>
>>> Code:
>>> $template RFC5424-to-file,"%TIMESTAMP:::date-rfc3339% %HOSTNAME%
>>> %syslogtag% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
>>> module(load="mmpstrucdata")
>>> if $structured-data != '-' then {
>>>     action(type="mmpstrucdata")
>>
>> at this point, write the log to the template RSYSLOG_DebugFormat and
>> you can see how it is parsed into the $! variable namespace
>>
>>>     if re_match($!rfc5424-sd!exampleSDID@32473!eventID, '10[0-2][0-9]')
>>
>> you could also say
>>
>> if $!rfc5424-sd!exampleSDID@32473!eventID >=10000 and
>> $!rfc5424-sd!exampleSDID@32473!eventID <=1029
>>
>> that would be more efficient than a regex
>>
>> David Lang
>>
>>> then {
>>>         action(type="omfile" File="/var/log/eventid_1000-1029.log"
>>> template="RFC5424-to-file")
>>>     }
>>> }
>>>
>>> Regards,
>>>
>>> Petr
>>>
>>> On 06.08.2019 15:34, Petr Vyhnal via rsyslog wrote:
>>>> Hello all,
>>>>
>>>>
>>>> did anyone tried to do a filtering of messages in structured data
>>>> format? Seems common re_match can't be used on $structured-data
>>>> property. It doesn't show error, but I can't get any positive
>>>> match. I found there is mmpstrucdata module which can parse
>>>> structured format into JSON variable tree, but not sure if these
>>>> variables could be somehow used in RainerScript if - then
>>>> declarations. Spent a lot of time trying to find some answer in
>>>> forums, but I found nothing similar to what I need. Any ideas are
>>>> welcome.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Petr
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering of syslog events in structured format [ In reply to ]
On Wed, 7 Aug 2019, Petr Vyhnal via rsyslog wrote:

> Again I have one follow up question to anyone who might know. It's
> possible somehow use "wildcards" in SD-ID declaration? Considering
> example message below I can have two similar messages like:
>
> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
> [exampleSDID@*32473* iut="3" eventSource="Application"
> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
> /dev/pts/8
>
> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
> [exampleSDID@*32474* iut="3" eventSource="Application"
> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
> /dev/pts/8
>
>
> As far as SD-ID is part of the path (if
> $!rfc5424-sd!exampleSDID@32473!eventID) I'd like to ask if there an
> option to have a single rule which would match different SD-IDs based on
> some (maybe regex or just wildcard) definition?

tale a look at table_lookup(), and for your IP matching, look at the sparse
table type (along with ipv42num), it was designed for geoip type lookups where
you give it a table with the boundries and it can lookup any IP

David Lang

>
> Regards,
>
> Petr
>
>
> On 07.08.2019 11:37, Petr Vyhnal via rsyslog wrote:
>> Hello David,
>>
>>
>> thanks for suggestions. Actually I used RSYSLOG_DebugFormat to
>> identify whole CEE chain. Also thanks for that recommendation to
>> avoiding regex, but shared code is not exactly the one I used. I'm
>> checking IP addresses in my case, so this is not applicable. Maybe
>> with some additional functions like ipv4tonum - I'll maybe give it a
>> try. Actually I just found it's possible to use re_match on
>> structured-data after all, I just had a typo in my original regex.
>> Anyway at least I'm a bit more familiar with structured data
>> processing now :-)
>>
>> Regards,
>>
>> Petr
>>
>> On 07.08.2019 11:10, David Lang wrote:
>>> On Wed, 7 Aug 2019, Petr Vyhnal via rsyslog wrote:
>>>
>>>> Hello all,
>>>>
>>>> finally I managed to make it work as I need. Below is snip of config
>>>> I used - just in case someone else tries to achieve something similar.
>>>>
>>>>
>>>> Example message:
>>>> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
>>>> [exampleSDID@32473 iut="3" eventSource="Application"
>>>> eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on
>>>> /dev/pts/8
>>>>
>>>> Code:
>>>> $template RFC5424-to-file,"%TIMESTAMP:::date-rfc3339% %HOSTNAME%
>>>> %syslogtag% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
>>>> module(load="mmpstrucdata")
>>>> if $structured-data != '-' then {
>>>>     action(type="mmpstrucdata")
>>>
>>> at this point, write the log to the template RSYSLOG_DebugFormat and
>>> you can see how it is parsed into the $! variable namespace
>>>
>>>>     if re_match($!rfc5424-sd!exampleSDID@32473!eventID, '10[0-2][0-9]')
>>>
>>> you could also say
>>>
>>> if $!rfc5424-sd!exampleSDID@32473!eventID >=10000 and
>>> $!rfc5424-sd!exampleSDID@32473!eventID <=1029
>>>
>>> that would be more efficient than a regex
>>>
>>> David Lang
>>>
>>>> then {
>>>>         action(type="omfile" File="/var/log/eventid_1000-1029.log"
>>>> template="RFC5424-to-file")
>>>>     }
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Petr
>>>>
>>>> On 06.08.2019 15:34, Petr Vyhnal via rsyslog wrote:
>>>>> Hello all,
>>>>>
>>>>>
>>>>> did anyone tried to do a filtering of messages in structured data
>>>>> format? Seems common re_match can't be used on $structured-data
>>>>> property. It doesn't show error, but I can't get any positive
>>>>> match. I found there is mmpstrucdata module which can parse
>>>>> structured format into JSON variable tree, but not sure if these
>>>>> variables could be somehow used in RainerScript if - then
>>>>> declarations. Spent a lot of time trying to find some answer in
>>>>> forums, but I found nothing similar to what I need. Any ideas are
>>>>> welcome.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Petr
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.