Mailing List Archive: Need help with Rsyslog Parser

Need help with Rsyslog Parser

Oct 11, 2023, 6:20 AM

Post #1 of 4 (149 views)

Hi Team,

I am new to this community and hoping to get some help with my below query.

We are collecting some network logs from client devices to a Syslog collector (A) which is managed by Vendor and the next hop is on the syslog collector (B) managed by us. On Collector B, I see the events are getting parsed with the IP of Collector A, meaning I am not seeing the actual device IP on my collector instead it shows Collector A IP address as sender IP. How / What config changes I need to make on my collector B rsyslog.conf file to see the actual client device IP/Hostname (using the syslog header) ?

I tried writing some template to extract hostname from the event as below but it is not working and I do not see any errors aswel when I run rsyslogd -N1.

template(name="LB_dynaFile" type="string" string="/var/log/syslog/LB/%!extracted_LB_hostname%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log")
template(name="LB_host_extraction" type="string" string="%msg:R,ERE,1,ZERO:\"GMT(.+) 0-PPE-0\"--end%")

if $rawmsg contains "IB*******2-m**i" then {
reset $!extracted_LB_hostname = exec_template("LB_host_extraction");
action(type="omfwd" target="1xx.1x.0.1x:1xx0" protocol="tcp" template="RSYSLOG_SyslogProtocol23Format")
action(type="omfile" dynaFile="LB_dynaFile" template="RSYSLOG_SyslogProtocol23Format")
}

PS - I tested regex on rsyslog regex tester and it is working.

Sample event

<134>1 2023-10-10T18:22:07.569574+00:00 10.194.237.27 - - - - 10/10/2023:18:21:58 GMT IB*******2-m**i 0-PPE-0 : default TCP CONN_TERMINATE 287440514 0 : Source 1x.1xx.0.1xx:80 - Destination 1x.1xx.1xx.1x:3xxx7 - Start Time 10/10/2023:18:21:16 GMT - End Time 10/10/2023:18:21:58 GMT - Total_bytes_send 1 - Total_bytes_recv 1

This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Need help with Rsyslog Parser [ In reply to ]

rsyslog at lists

Oct 11, 2023, 12:05 PM

Post #2 of 4 (148 views)

Permalink

Templates are how you format messages that you are outputting, they have nothing
to do with parsing messages.

I would first suggest that you log the message with the template
RSYSLOG_DebugFormat so that you can see all the variables that get parsed out of
the message already, and what is actually arriving at the system (I would guess
you are interested in $hostname, but am not completely sure)

What I do when relaying messages is to configure the relaying system to modify
the message to be in json format with $!msg as the original message and a tree
under $!mycompanyname where I store additional information (IP I received the
message from, time it arrived on the relay, hostname of the relay so in an HA
environment I know which relay handled the message, etc)

I can then use mmnormalize on the central system to parse out these additional
fields to make the data available there.

David Lang

On Wed, 11 Oct 2023, Gundlapally, Navanitha via rsyslog
wrote:

> Hi Team,
>
> I am new to this community and hoping to get some help with my below query.
>
> We are collecting some network logs from client devices to a Syslog collector (A) which is managed by Vendor and the next hop is on the syslog collector (B) managed by us. On Collector B, I see the events are getting parsed with the IP of Collector A, meaning I am not seeing the actual device IP on my collector instead it shows Collector A IP address as sender IP. How / What config changes I need to make on my collector B rsyslog.conf file to see the actual client device IP/Hostname (using the syslog header) ?
>
> I tried writing some template to extract hostname from the event as below but it is not working and I do not see any errors aswel when I run rsyslogd -N1.
>
> template(name="LB_dynaFile" type="string" string="/var/log/syslog/LB/%!extracted_LB_hostname%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log")
> template(name="LB_host_extraction" type="string" string="%msg:R,ERE,1,ZERO:\"GMT(.+) 0-PPE-0\"--end%")
>
> if $rawmsg contains "IB*******2-m**i" then {
> reset $!extracted_LB_hostname = exec_template("LB_host_extraction");
> action(type="omfwd" target="1xx.1x.0.1x:1xx0" protocol="tcp" template="RSYSLOG_SyslogProtocol23Format")
> action(type="omfile" dynaFile="LB_dynaFile" template="RSYSLOG_SyslogProtocol23Format")
> }
>
> PS - I tested regex on rsyslog regex tester and it is working.
>
> Sample event
>
> <134>1 2023-10-10T18:22:07.569574+00:00 10.194.237.27 - - - - 10/10/2023:18:21:58 GMT IB*******2-m**i 0-PPE-0 : default TCP CONN_TERMINATE 287440514 0 : Source 1x.1xx.0.1xx:80 - Destination 1x.1xx.1xx.1x:3xxx7 - Start Time 10/10/2023:18:21:16 GMT - End Time 10/10/2023:18:21:58 GMT - Total_bytes_send 1 - Total_bytes_recv 1
>
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Need help with Rsyslog Parser [ In reply to ]

rsyslog at lists

Oct 11, 2023, 12:06 PM

Post #3 of 4 (148 views)

Permalink

P.S. this confusion of templates being useful for parsing messages seems to be a
common one, any suggestions on what we should put in the documentation to make
clear that they are for output only, not for parsing messages?

David Lang

On Wed, 11 Oct 2023, Gundlapally, Navanitha via rsyslog wrote:

> Date: Wed, 11 Oct 2023 13:20:57 +0000
> From: "Gundlapally, Navanitha via rsyslog" <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "Gundlapally, Navanitha" <Navanitha.Gundlapally@broadridge.com>
> Subject: [rsyslog] Need help with Rsyslog Parser
>
> Hi Team,
>
> I am new to this community and hoping to get some help with my below query.
>
> We are collecting some network logs from client devices to a Syslog collector (A) which is managed by Vendor and the next hop is on the syslog collector (B) managed by us. On Collector B, I see the events are getting parsed with the IP of Collector A, meaning I am not seeing the actual device IP on my collector instead it shows Collector A IP address as sender IP. How / What config changes I need to make on my collector B rsyslog.conf file to see the actual client device IP/Hostname (using the syslog header) ?
>
> I tried writing some template to extract hostname from the event as below but it is not working and I do not see any errors aswel when I run rsyslogd -N1.
>
> template(name="LB_dynaFile" type="string" string="/var/log/syslog/LB/%!extracted_LB_hostname%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log")
> template(name="LB_host_extraction" type="string" string="%msg:R,ERE,1,ZERO:\"GMT(.+) 0-PPE-0\"--end%")
>
> if $rawmsg contains "IB*******2-m**i" then {
> reset $!extracted_LB_hostname = exec_template("LB_host_extraction");
> action(type="omfwd" target="1xx.1x.0.1x:1xx0" protocol="tcp" template="RSYSLOG_SyslogProtocol23Format")
> action(type="omfile" dynaFile="LB_dynaFile" template="RSYSLOG_SyslogProtocol23Format")
> }
>
> PS - I tested regex on rsyslog regex tester and it is working.
>
> Sample event
>
> <134>1 2023-10-10T18:22:07.569574+00:00 10.194.237.27 - - - - 10/10/2023:18:21:58 GMT IB*******2-m**i 0-PPE-0 : default TCP CONN_TERMINATE 287440514 0 : Source 1x.1xx.0.1xx:80 - Destination 1x.1xx.1xx.1x:3xxx7 - Start Time 10/10/2023:18:21:16 GMT - End Time 10/10/2023:18:21:58 GMT - Total_bytes_send 1 - Total_bytes_recv 1
>
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Need help with Rsyslog Parser [ In reply to ]

rsyslog at lists

Oct 11, 2023, 12:18 PM

Post #4 of 4 (148 views)

Permalink

I'll take 'A big note saying "TEMPLATES ARE FOR OUTPUT ONLY"' for 10 points
please.

On Wed, Oct 11, 2023, 21:06 David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> P.S. this confusion of templates being useful for parsing messages seems
> to be a
> common one, any suggestions on what we should put in the documentation to
> make
> clear that they are for output only, not for parsing messages?
>
> David Lang
>
> On Wed, 11 Oct 2023, Gundlapally, Navanitha via rsyslog wrote:
>
> > Date: Wed, 11 Oct 2023 13:20:57 +0000
> > From: "Gundlapally, Navanitha via rsyslog" <rsyslog@lists.adiscon.com>
> > To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> > Cc: "Gundlapally, Navanitha" <Navanitha.Gundlapally@broadridge.com>
> > Subject: [rsyslog] Need help with Rsyslog Parser
> >
> > Hi Team,
> >
> > I am new to this community and hoping to get some help with my below
> query.
> >
> > We are collecting some network logs from client devices to a Syslog
> collector (A) which is managed by Vendor and the next hop is on the syslog
> collector (B) managed by us. On Collector B, I see the events are getting
> parsed with the IP of Collector A, meaning I am not seeing the actual
> device IP on my collector instead it shows Collector A IP address as sender
> IP. How / What config changes I need to make on my collector B
> rsyslog.conf file to see the actual client device IP/Hostname (using the
> syslog header) ?
> >
> > I tried writing some template to extract hostname from the event as
> below but it is not working and I do not see any errors aswel when I run
> rsyslogd -N1.
> >
> > template(name="LB_dynaFile" type="string"
> string="/var/log/syslog/LB/%!extracted_LB_hostname%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log")
> > template(name="LB_host_extraction" type="string"
> string="%msg:R,ERE,1,ZERO:\"GMT(.+) 0-PPE-0\"--end%")
> >
> > if $rawmsg contains "IB*******2-m**i" then {
> > reset $!extracted_LB_hostname = exec_template("LB_host_extraction");
> > action(type="omfwd" target="1xx.1x.0.1x:1xx0" protocol="tcp"
> template="RSYSLOG_SyslogProtocol23Format")
> > action(type="omfile" dynaFile="LB_dynaFile"
> template="RSYSLOG_SyslogProtocol23Format")
> > }
> >
> > PS - I tested regex on rsyslog regex tester and it is working.
> >
> > Sample event
> >
> > <134>1 2023-10-10T18:22:07.569574+00:00 10.194.237.27 - - - -
> 10/10/2023:18:21:58 GMT IB*******2-m**i 0-PPE-0 : default TCP
> CONN_TERMINATE 287440514 0 : Source 1x.1xx.0.1xx:80 - Destination
> 1x.1xx.1xx.1x:3xxx7 - Start Time 10/10/2023:18:21:16 GMT - End Time
> 10/10/2023:18:21:58 GMT - Total_bytes_send 1 - Total_bytes_recv 1
> >
> >
> > This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, please notify us immediately by
> e-mail and delete the message and any attachments from your system.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.