Mailing List Archive

Add tag before normalization
Hi all,

We are trying to use mmnormalize to process incoming CEF messages from a
fortigate firewall. Unfortunately the fortigate doesn't seem to include a
tag and so the 'CEF:' component of the message actually gets parsed as the
tag. This means that mmnormalize is then unable to parse the CEF message.

Here is an example incoming message:

Oct 21 03:07:16 CassiniTest
CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
msg=Performance statistics: average CPU: 0, memory: 43, concurrent
sessions: 12, setup-rate: 0

Here is one that I have modified by hand that works - I just added the "-"
character to indicate a nil tag value:

Oct 21 03:07:16 CassiniTest -
CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
msg=Performance statistics: average CPU: 0, memory: 43, concurrent
sessions: 12, setup-rate: 0

This is what my test config looks like:

module(load="mmnormalize")

template(name="csv" type="list") {

# syslog properties
property(name="timegenerated")
constant(value=",")
property(name="hostname")

# CSV
constant(value=",")
property(name="$!cef!DeviceVendor" format="csv")
constant(value=",")
property(name="$!cef!DeviceProduct" format="csv")
constant(value=",")
property(name="$!cef!DeviceVersion" format="csv")
constant(value=",")
property(name="$!cef!SignatureID" format="csv")
constant(value=",")
property(name="$!cef!Name" format="csv")
constant(value=",")
property(name="$!cef!Severity" format="csv")
constant(value=",")
property(name="$!cef!Extensions!act" format="csv")
constant(value="\n")
}

ruleset(name="cef_ruleset") {
action(type="mmnormalize" ruleBase="/etc/rsyslog.d/cef.rb" useRawMsg="off")
action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
File="/var/log/fortigate/parsed.log" template="csv")
}
input(type="imtcp" port="514" ruleset="cef_ruleset")


Is there an action I can use to add the missing tag field? I did look at
the mmtaghostname module but it's not in the ubuntu or centos builds (and
looks like it won't be added).

Many thanks,
James
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Add tag before normalization [ In reply to ]
One thing you could do is tell mmnormalize to use rawmsg instead of
msg.  Then just parse the pieces apart yourself.

Regards,


On 10/21/20 5:42 PM, James Fitzsimons via rsyslog wrote:
> Hi all,
>
> We are trying to use mmnormalize to process incoming CEF messages from a
> fortigate firewall. Unfortunately the fortigate doesn't seem to include a
> tag and so the 'CEF:' component of the message actually gets parsed as the
> tag. This means that mmnormalize is then unable to parse the CEF message.
>
> Here is an example incoming message:
>
> Oct 21 03:07:16 CassiniTest
> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
> sessions: 12, setup-rate: 0
>
> Here is one that I have modified by hand that works - I just added the "-"
> character to indicate a nil tag value:
>
> Oct 21 03:07:16 CassiniTest -
> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
> sessions: 12, setup-rate: 0
>
> This is what my test config looks like:
>
> module(load="mmnormalize")
>
> template(name="csv" type="list") {
>
> # syslog properties
> property(name="timegenerated")
> constant(value=",")
> property(name="hostname")
>
> # CSV
> constant(value=",")
> property(name="$!cef!DeviceVendor" format="csv")
> constant(value=",")
> property(name="$!cef!DeviceProduct" format="csv")
> constant(value=",")
> property(name="$!cef!DeviceVersion" format="csv")
> constant(value=",")
> property(name="$!cef!SignatureID" format="csv")
> constant(value=",")
> property(name="$!cef!Name" format="csv")
> constant(value=",")
> property(name="$!cef!Severity" format="csv")
> constant(value=",")
> property(name="$!cef!Extensions!act" format="csv")
> constant(value="\n")
> }
>
> ruleset(name="cef_ruleset") {
> action(type="mmnormalize" ruleBase="/etc/rsyslog.d/cef.rb" useRawMsg="off")
> action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
> File="/var/log/fortigate/parsed.log" template="csv")
> }
> input(type="imtcp" port="514" ruleset="cef_ruleset")
>
>
> Is there an action I can use to add the missing tag field? I did look at
> the mmtaghostname module but it's not in the ubuntu or centos builds (and
> looks like it won't be added).
>
> Many thanks,
> James
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Add tag before normalization [ In reply to ]
you can also set mmnormalize to parse a variable

if $programname == "CEE:" then
set $.msg="CEE:" & $msg;
else
set $.msg = $msg;

then parse with variable="$.msg" in the mmnormalize action() statement

David Lang


On Wed, 21 Oct 2020, John Chivian via rsyslog wrote:

> Date: Wed, 21 Oct 2020 19:21:45 -0500
> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: John Chivian <jchivian@chivian.com>
> Subject: Re: [rsyslog] Add tag before normalization
>
> One thing you could do is tell mmnormalize to use rawmsg instead of
> msg.  Then just parse the pieces apart yourself.
>
> Regards,
>
>
> On 10/21/20 5:42 PM, James Fitzsimons via rsyslog wrote:
>> Hi all,
>>
>> We are trying to use mmnormalize to process incoming CEF messages from a
>> fortigate firewall. Unfortunately the fortigate doesn't seem to include a
>> tag and so the 'CEF:' component of the message actually gets parsed as the
>> tag. This means that mmnormalize is then unable to parse the CEF message.
>>
>> Here is an example incoming message:
>>
>> Oct 21 03:07:16 CassiniTest
>> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
>> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
>> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
>> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
>> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
>> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
>> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
>> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
>> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
>> sessions: 12, setup-rate: 0
>>
>> Here is one that I have modified by hand that works - I just added the "-"
>> character to indicate a nil tag value:
>>
>> Oct 21 03:07:16 CassiniTest -
>> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
>> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
>> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice FTNTFGTvd=root
>> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 FTNTFGTlogdesc=System
>> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
>> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
>> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
>> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
>> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
>> sessions: 12, setup-rate: 0
>>
>> This is what my test config looks like:
>>
>> module(load="mmnormalize")
>>
>> template(name="csv" type="list") {
>>
>> # syslog properties
>> property(name="timegenerated")
>> constant(value=",")
>> property(name="hostname")
>>
>> # CSV
>> constant(value=",")
>> property(name="$!cef!DeviceVendor" format="csv")
>> constant(value=",")
>> property(name="$!cef!DeviceProduct" format="csv")
>> constant(value=",")
>> property(name="$!cef!DeviceVersion" format="csv")
>> constant(value=",")
>> property(name="$!cef!SignatureID" format="csv")
>> constant(value=",")
>> property(name="$!cef!Name" format="csv")
>> constant(value=",")
>> property(name="$!cef!Severity" format="csv")
>> constant(value=",")
>> property(name="$!cef!Extensions!act" format="csv")
>> constant(value="\n")
>> }
>>
>> ruleset(name="cef_ruleset") {
>> action(type="mmnormalize" ruleBase="/etc/rsyslog.d/cef.rb" useRawMsg="off")
>> action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
>> File="/var/log/fortigate/parsed.log" template="csv")
>> }
>> input(type="imtcp" port="514" ruleset="cef_ruleset")
>>
>>
>> Is there an action I can use to add the missing tag field? I did look at
>> the mmtaghostname module but it's not in the ubuntu or centos builds (and
>> looks like it won't be added).
>>
>> Many thanks,
>> James
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Add tag before normalization [ In reply to ]
Hi all,

Thanks both David and John for your replies.

David I used your suggestion and I now have it working. Thank you so much
for pointing me in the right direction!

Cheers,
James

On Thu, Oct 22, 2020 at 1:46 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> you can also set mmnormalize to parse a variable
>
> if $programname == "CEE:" then
> set $.msg="CEE:" & $msg;
> else
> set $.msg = $msg;
>
> then parse with variable="$.msg" in the mmnormalize action() statement
>
> David Lang
>
>
> On Wed, 21 Oct 2020, John Chivian via rsyslog wrote:
>
> > Date: Wed, 21 Oct 2020 19:21:45 -0500
> > From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: John Chivian <jchivian@chivian.com>
> > Subject: Re: [rsyslog] Add tag before normalization
> >
> > One thing you could do is tell mmnormalize to use rawmsg instead of
> > msg. Then just parse the pieces apart yourself.
> >
> > Regards,
> >
> >
> > On 10/21/20 5:42 PM, James Fitzsimons via rsyslog wrote:
> >> Hi all,
> >>
> >> We are trying to use mmnormalize to process incoming CEF messages from a
> >> fortigate firewall. Unfortunately the fortigate doesn't seem to include
> a
> >> tag and so the 'CEF:' component of the message actually gets parsed as
> the
> >> tag. This means that mmnormalize is then unable to parse the CEF
> message.
> >>
> >> Here is an example incoming message:
> >>
> >> Oct 21 03:07:16 CassiniTest
> >> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
> >> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
> >> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice
> FTNTFGTvd=root
> >> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700
> FTNTFGTlogdesc=System
> >> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
> >> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
> >> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
> >> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
> >> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
> >> sessions: 12, setup-rate: 0
> >>
> >> Here is one that I have modified by hand that works - I just added the
> "-"
> >> character to indicate a nil tag value:
> >>
> >> Oct 21 03:07:16 CassiniTest -
> >> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system
> >> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704
> >> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice
> FTNTFGTvd=root
> >> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700
> FTNTFGTlogdesc=System
> >> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43
> >> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1
> >> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0
> >> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A
> >> msg=Performance statistics: average CPU: 0, memory: 43, concurrent
> >> sessions: 12, setup-rate: 0
> >>
> >> This is what my test config looks like:
> >>
> >> module(load="mmnormalize")
> >>
> >> template(name="csv" type="list") {
> >>
> >> # syslog properties
> >> property(name="timegenerated")
> >> constant(value=",")
> >> property(name="hostname")
> >>
> >> # CSV
> >> constant(value=",")
> >> property(name="$!cef!DeviceVendor" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!DeviceProduct" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!DeviceVersion" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!SignatureID" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!Name" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!Severity" format="csv")
> >> constant(value=",")
> >> property(name="$!cef!Extensions!act" format="csv")
> >> constant(value="\n")
> >> }
> >>
> >> ruleset(name="cef_ruleset") {
> >> action(type="mmnormalize" ruleBase="/etc/rsyslog.d/cef.rb"
> useRawMsg="off")
> >> action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
> >> File="/var/log/fortigate/parsed.log" template="csv")
> >> }
> >> input(type="imtcp" port="514" ruleset="cef_ruleset")
> >>
> >>
> >> Is there an action I can use to add the missing tag field? I did look at
> >> the mmtaghostname module but it's not in the ubuntu or centos builds
> (and
> >> looks like it won't be added).
> >>
> >> Many thanks,
> >> James
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.