Mailing List Archive: Rsyslog seems to think there are duplicates

Rsyslog seems to think there are duplicates

Oct 11, 2019, 10:14 AM

Post #1 of 9 (592 views)

Hey,

We are using Checkpoint Log Exporter
(https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
I don't think that it matters, except maybe for the fact that they are
implementing RFC5424.

The logs are being written to file by Rsyslog as follows:

---------------------
module(load="imtcp")
module(load="imudp")

input(type="imtcp" port="514")
input(type="imudp" port="514")

template(name="FileNameByHost" type="string"
string="/syslog/%fromhost-ip%.log")
if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
action(type="omfile" dynaFile="FileNameByHost"
template="RSYSLOG_SyslogProtocol23Format") <-------------------
} else {
action(type="omfile" dynaFile="FileNameByHost"
template="RSYSLOG_FileFormat")
}

call asyncelk
...
---------------------

Now Rsyslog thinks some messages are duplicates. And apparently it
thinks there are _many_ duplicates. I have sniffed the traffic, and no
duplicates are being transmitted AFAICS. Here is an example of what is
logged nevertheless:

<134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 -
[.action:"Reject"; ifdir:"inbound"; ifname:"eth2.53"; logid:"0";
loguid:"{0x0,0x0,0x0,0x0}"; origin:"192.168.1.2"; originsicname:"xxxxx";
sequencenum:"32"; time:"1570811271"; version:"5"; dst:"224.0.0.18";
hll_key:"2008789500278145398"; inzone:"Internal"; layer_name:"Unified";
layer_uuid:"a-b-c-d-e"; match_id:"177"; parent_rule:"0";
rule_action:"Reject"; rule_name:"xyz"; rule_uid:"a-b-c-d-e";
product:"VPN-1 & FireWall-1"; proto:"51"; service_id:"AH";
src:"1.2.3.4"; ] message repeated 49 times: []

Other times it says "99 times" or "149 times", ... I've created somes
stats in fact :D

# tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+
times' | sort | uniq -c | sort -n | tail -10
23 repeated 44 times
26 repeated 199 times
35 repeated 159 times
37 repeated 9 times
38 repeated 4 times
101 repeated 149 times
155 repeated 109 times
408 repeated 59 times
417 repeated 99 times
2926 repeated 49 times

This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64

Any idea?

Thanks,
Marki

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 11:37 AM

Post #2 of 9 (591 views)

Permalink

repeated message reduction is not recommended. It is a feature that was useful
back when it was only people looking at the logs, but when you are analyzing the
logs with automation, the 'message repeated' logs reduce your accuracy, not
improve it.

that 'message repeated' is happening on the sender, so look to fix it there (in
rsyslog we started having the message repeated log include the log being
repeated so that it was easier to track what's happening)

David Lang

On Fri, 11 Oct 2019, Marki via rsyslog wrote:

> Date: Fri, 11 Oct 2019 19:14:27 +0200
> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Marki <rsyslog@lists.roth.lu>
> Subject: [rsyslog] Rsyslog seems to think there are duplicates
>
>
> Hey,
>
> We are using Checkpoint Log Exporter
> (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
> I don't think that it matters, except maybe for the fact that they are
> implementing RFC5424.
>
> The logs are being written to file by Rsyslog as follows:
>
> ---------------------
> module(load="imtcp")
> module(load="imudp")
>
> input(type="imtcp" port="514")
> input(type="imudp" port="514")
>
> template(name="FileNameByHost" type="string"
> string="/syslog/%fromhost-ip%.log")
> if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
> action(type="omfile" dynaFile="FileNameByHost"
> template="RSYSLOG_SyslogProtocol23Format") <-------------------
> } else {
> action(type="omfile" dynaFile="FileNameByHost"
> template="RSYSLOG_FileFormat")
> }
>
> call asyncelk
> ...
> ---------------------
>
> Now Rsyslog thinks some messages are duplicates. And apparently it thinks
> there are _many_ duplicates. I have sniffed the traffic, and no duplicates
> are being transmitted AFAICS. Here is an example of what is logged
> nevertheless:
>
> <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 - [.action:"Reject";
> ifdir:"inbound"; ifname:"eth2.53"; logid:"0"; loguid:"{0x0,0x0,0x0,0x0}";
> origin:"192.168.1.2"; originsicname:"xxxxx"; sequencenum:"32";
> time:"1570811271"; version:"5"; dst:"224.0.0.18";
> hll_key:"2008789500278145398"; inzone:"Internal"; layer_name:"Unified";
> layer_uuid:"a-b-c-d-e"; match_id:"177"; parent_rule:"0";
> rule_action:"Reject"; rule_name:"xyz"; rule_uid:"a-b-c-d-e"; product:"VPN-1 &
> FireWall-1"; proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message repeated
> 49 times: []
>
> Other times it says "99 times" or "149 times", ... I've created somes stats
> in fact :D
>
> # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+ times' |
> sort | uniq -c | sort -n | tail -10
> 23 repeated 44 times
> 26 repeated 199 times
> 35 repeated 159 times
> 37 repeated 9 times
> 38 repeated 4 times
> 101 repeated 149 times
> 155 repeated 109 times
> 408 repeated 59 times
> 417 repeated 99 times
> 2926 repeated 49 times
>
> This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64
>
> Any idea?
>
> Thanks,
> Marki
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 12:03 PM

Post #3 of 9 (591 views)

Permalink

David is bang on. This might be off topic, but i used to have to poll
Checkpoint logs via their opsec API. This worked very well:

https://github.com/blackhole-em/OPSEC2Elastic

While I know this might not help your exact question, it might be relevant
as well. Then again... maybe not?

Cheers,

JB

On Fri, Oct 11, 2019 at 2:37 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> repeated message reduction is not recommended. It is a feature that was
> useful
> back when it was only people looking at the logs, but when you are
> analyzing the
> logs with automation, the 'message repeated' logs reduce your accuracy,
> not
> improve it.
>
> that 'message repeated' is happening on the sender, so look to fix it
> there (in
> rsyslog we started having the message repeated log include the log being
> repeated so that it was easier to track what's happening)
>
> David Lang
>
> On Fri, 11 Oct 2019, Marki via rsyslog wrote:
>
> > Date: Fri, 11 Oct 2019 19:14:27 +0200
> > From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: Marki <rsyslog@lists.roth.lu>
> > Subject: [rsyslog] Rsyslog seems to think there are duplicates
> >
> >
> > Hey,
> >
> > We are using Checkpoint Log Exporter
> > (
> https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
>
> > I don't think that it matters, except maybe for the fact that they are
> > implementing RFC5424.
> >
> > The logs are being written to file by Rsyslog as follows:
> >
> > ---------------------
> > module(load="imtcp")
> > module(load="imudp")
> >
> > input(type="imtcp" port="514")
> > input(type="imudp" port="514")
> >
> > template(name="FileNameByHost" type="string"
> > string="/syslog/%fromhost-ip%.log")
> > if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
> > action(type="omfile" dynaFile="FileNameByHost"
> > template="RSYSLOG_SyslogProtocol23Format") <-------------------
> > } else {
> > action(type="omfile" dynaFile="FileNameByHost"
> > template="RSYSLOG_FileFormat")
> > }
> >
> > call asyncelk
> > ...
> > ---------------------
> >
> > Now Rsyslog thinks some messages are duplicates. And apparently it
> thinks
> > there are _many_ duplicates. I have sniffed the traffic, and no
> duplicates
> > are being transmitted AFAICS. Here is an example of what is logged
> > nevertheless:
> >
> > <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 -
> [action:"Reject";
> > ifdir:"inbound"; ifname:"eth2.53"; logid:"0";
> loguid:"{0x0,0x0,0x0,0x0}";
> > origin:"192.168.1.2"; originsicname:"xxxxx"; sequencenum:"32";
> > time:"1570811271"; version:"5"; dst:"224.0.0.18";
> > hll_key:"2008789500278145398"; inzone:"Internal"; layer_name:"Unified";
> > layer_uuid:"a-b-c-d-e"; match_id:"177"; parent_rule:"0";
> > rule_action:"Reject"; rule_name:"xyz"; rule_uid:"a-b-c-d-e";
> product:"VPN-1 &
> > FireWall-1"; proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message
> repeated
> > 49 times: []
> >
> > Other times it says "99 times" or "149 times", ... I've created somes
> stats
> > in fact :D
> >
> > # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+
> times' |
> > sort | uniq -c | sort -n | tail -10
> > 23 repeated 44 times
> > 26 repeated 199 times
> > 35 repeated 159 times
> > 37 repeated 9 times
> > 38 repeated 4 times
> > 101 repeated 149 times
> > 155 repeated 109 times
> > 408 repeated 59 times
> > 417 repeated 99 times
> > 2926 repeated 49 times
> >
> > This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64
> >
> > Any idea?
> >
> > Thanks,
> > Marki
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 12:41 PM

Post #4 of 9 (591 views)

Permalink

I have rechecked it... as noted before, this "message repeated" string
appears nowhere in the packet capture I took. Also the source is not
repeating any of those messages. It must be rsyslog thinking there are
duplicates and inserting this message.

What would the empty square brackets trailing the message mean?

The interesting thing is that it indeed only happens with the messages
from Checkpoint Smartcenter / Log Exporter. What's special about it is
that it's the only source in our setup sending RFC5424-style messages
containing a structured data field.

The general interesting thing in our setup is that we're using multiple
queues.

Was there any bug of consequence somehow related to all of this in or
after 8.33.1 (which is the latest one available in SLES)?

Thanks.

On 10/11/2019 8:37 PM, David Lang wrote:
> repeated message reduction is not recommended. It is a feature that
> was useful back when it was only people looking at the logs, but when
> you are analyzing the logs with automation, the 'message repeated'
> logs reduce your accuracy, not improve it.
>
> that 'message repeated' is happening on the sender, so look to fix it
> there (in rsyslog we started having the message repeated log include
> the log being repeated so that it was easier to track what's happening)
>
> David Lang
>
> On Fri, 11 Oct 2019, Marki via rsyslog wrote:
>
>> Date: Fri, 11 Oct 2019 19:14:27 +0200
>> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog@lists.adiscon.com
>> Cc: Marki <rsyslog@lists.roth.lu>
>> Subject: [rsyslog] Rsyslog seems to think there are duplicates
>>
>>
>> Hey,
>>
>> We are using Checkpoint Log Exporter
>> (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
>> I don't think that it matters, except maybe for the fact that they
>> are implementing RFC5424.
>>
>> The logs are being written to file by Rsyslog as follows:
>>
>> ---------------------
>> module(load="imtcp")
>> module(load="imudp")
>>
>> input(type="imtcp" port="514")
>> input(type="imudp" port="514")
>>
>> template(name="FileNameByHost" type="string"
>> string="/syslog/%fromhost-ip%.log")
>> if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
>> action(type="omfile" dynaFile="FileNameByHost"
>> template="RSYSLOG_SyslogProtocol23Format") <-------------------
>> } else {
>> action(type="omfile" dynaFile="FileNameByHost"
>> template="RSYSLOG_FileFormat")
>> }
>>
>> call asyncelk
>> ...
>> ---------------------
>>
>> Now Rsyslog thinks some messages are duplicates. And apparently it
>> thinks there are _many_ duplicates. I have sniffed the traffic, and
>> no duplicates are being transmitted AFAICS. Here is an example of
>> what is logged nevertheless:
>>
>> <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 -
>> [.action:"Reject"; ifdir:"inbound"; ifname:"eth2.53"; logid:"0";
>> loguid:"{0x0,0x0,0x0,0x0}"; origin:"192.168.1.2";
>> originsicname:"xxxxx"; sequencenum:"32"; time:"1570811271";
>> version:"5"; dst:"224.0.0.18"; hll_key:"2008789500278145398";
>> inzone:"Internal"; layer_name:"Unified"; layer_uuid:"a-b-c-d-e";
>> match_id:"177"; parent_rule:"0"; rule_action:"Reject";
>> rule_name:"xyz"; rule_uid:"a-b-c-d-e"; product:"VPN-1 & FireWall-1";
>> proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message repeated 49
>> times: []
>>
>> Other times it says "99 times" or "149 times", ... I've created somes
>> stats in fact :D
>>
>> # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+
>> times' | sort | uniq -c | sort -n | tail -10
>>     23 repeated 44 times
>>     26 repeated 199 times
>>     35 repeated 159 times
>>     37 repeated 9 times
>>     38 repeated 4 times
>>    101 repeated 149 times
>>    155 repeated 109 times
>>    408 repeated 59 times
>>    417 repeated 99 times
>> 2926 repeated 49 times
>>
>> This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64
>>
>> Any idea?
>>
>> Thanks,
>> Marki
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 12:51 PM

Post #5 of 9 (591 views)

Permalink

The legacy code for this very old style "message repeated n times" does not
consider rfc5424 structured data, so the empty MSG field makes the messages
all equal.

As David said, that feature is not useful any longer today. This is why it
is turned off by default.

HTH
Rainer

Sent from phone, thus brief.

Marki via rsyslog <rsyslog@lists.adiscon.com> schrieb am Fr., 11. Okt.
2019, 21:41:

> I have rechecked it... as noted before, this "message repeated" string
> appears nowhere in the packet capture I took. Also the source is not
> repeating any of those messages. It must be rsyslog thinking there are
> duplicates and inserting this message.
>
> What would the empty square brackets trailing the message mean?
>
> The interesting thing is that it indeed only happens with the messages
> from Checkpoint Smartcenter / Log Exporter. What's special about it is
> that it's the only source in our setup sending RFC5424-style messages
> containing a structured data field.
>
> The general interesting thing in our setup is that we're using multiple
> queues.
>
> Was there any bug of consequence somehow related to all of this in or
> after 8.33.1 (which is the latest one available in SLES)?
>
> Thanks.
>
> On 10/11/2019 8:37 PM, David Lang wrote:
> > repeated message reduction is not recommended. It is a feature that
> > was useful back when it was only people looking at the logs, but when
> > you are analyzing the logs with automation, the 'message repeated'
> > logs reduce your accuracy, not improve it.
> >
> > that 'message repeated' is happening on the sender, so look to fix it
> > there (in rsyslog we started having the message repeated log include
> > the log being repeated so that it was easier to track what's happening)
> >
> > David Lang
> >
> > On Fri, 11 Oct 2019, Marki via rsyslog wrote:
> >
> >> Date: Fri, 11 Oct 2019 19:14:27 +0200
> >> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> >> To: rsyslog@lists.adiscon.com
> >> Cc: Marki <rsyslog@lists.roth.lu>
> >> Subject: [rsyslog] Rsyslog seems to think there are duplicates
> >>
> >>
> >> Hey,
> >>
> >> We are using Checkpoint Log Exporter
> >> (
> https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
>
> >> I don't think that it matters, except maybe for the fact that they
> >> are implementing RFC5424.
> >>
> >> The logs are being written to file by Rsyslog as follows:
> >>
> >> ---------------------
> >> module(load="imtcp")
> >> module(load="imudp")
> >>
> >> input(type="imtcp" port="514")
> >> input(type="imudp" port="514")
> >>
> >> template(name="FileNameByHost" type="string"
> >> string="/syslog/%fromhost-ip%.log")
> >> if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
> >> action(type="omfile" dynaFile="FileNameByHost"
> >> template="RSYSLOG_SyslogProtocol23Format") <-------------------
> >> } else {
> >> action(type="omfile" dynaFile="FileNameByHost"
> >> template="RSYSLOG_FileFormat")
> >> }
> >>
> >> call asyncelk
> >> ...
> >> ---------------------
> >>
> >> Now Rsyslog thinks some messages are duplicates. And apparently it
> >> thinks there are _many_ duplicates. I have sniffed the traffic, and
> >> no duplicates are being transmitted AFAICS. Here is an example of
> >> what is logged nevertheless:
> >>
> >> <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 -
> >> [.action:"Reject"; ifdir:"inbound"; ifname:"eth2.53"; logid:"0";
> >> loguid:"{0x0,0x0,0x0,0x0}"; origin:"192.168.1.2";
> >> originsicname:"xxxxx"; sequencenum:"32"; time:"1570811271";
> >> version:"5"; dst:"224.0.0.18"; hll_key:"2008789500278145398";
> >> inzone:"Internal"; layer_name:"Unified"; layer_uuid:"a-b-c-d-e";
> >> match_id:"177"; parent_rule:"0"; rule_action:"Reject";
> >> rule_name:"xyz"; rule_uid:"a-b-c-d-e"; product:"VPN-1 & FireWall-1";
> >> proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message repeated 49
> >> times: []
> >>
> >> Other times it says "99 times" or "149 times", ... I've created somes
> >> stats in fact :D
> >>
> >> # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+
> >> times' | sort | uniq -c | sort -n | tail -10
> >> 23 repeated 44 times
> >> 26 repeated 199 times
> >> 35 repeated 159 times
> >> 37 repeated 9 times
> >> 38 repeated 4 times
> >> 101 repeated 149 times
> >> 155 repeated 109 times
> >> 408 repeated 59 times
> >> 417 repeated 99 times
> >> 2926 repeated 49 times
> >>
> >> This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64
> >>
> >> Any idea?
> >>
> >> Thanks,
> >> Marki
> >>
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >> POST if you DON'T LIKE THAT.
> >>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 2:17 PM

Post #6 of 9 (591 views)

Permalink

can you post your config?

David Lang

On Fri, 11 Oct 2019, Marki via rsyslog wrote:

> Date: Fri, 11 Oct 2019 21:41:04 +0200
> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> To: Marki via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Marki <rsyslog@lists.roth.lu>
> Subject: Re: [rsyslog] Rsyslog seems to think there are duplicates
>
> I have rechecked it... as noted before, this "message repeated" string
> appears nowhere in the packet capture I took. Also the source is not
> repeating any of those messages. It must be rsyslog thinking there are
> duplicates and inserting this message.
>
> What would the empty square brackets trailing the message mean?
>
> The interesting thing is that it indeed only happens with the messages
> from Checkpoint Smartcenter / Log Exporter. What's special about it is
> that it's the only source in our setup sending RFC5424-style messages
> containing a structured data field.
>
> The general interesting thing in our setup is that we're using multiple
> queues.
>
> Was there any bug of consequence somehow related to all of this in or
> after 8.33.1 (which is the latest one available in SLES)?
>
> Thanks.
>
> On 10/11/2019 8:37 PM, David Lang wrote:
>> repeated message reduction is not recommended. It is a feature that
>> was useful back when it was only people looking at the logs, but when
>> you are analyzing the logs with automation, the 'message repeated'
>> logs reduce your accuracy, not improve it.
>>
>> that 'message repeated' is happening on the sender, so look to fix it
>> there (in rsyslog we started having the message repeated log include
>> the log being repeated so that it was easier to track what's happening)
>>
>> David Lang
>>
>> On Fri, 11 Oct 2019, Marki via rsyslog wrote:
>>
>>> Date: Fri, 11 Oct 2019 19:14:27 +0200
>>> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Marki <rsyslog@lists.roth.lu>
>>> Subject: [rsyslog] Rsyslog seems to think there are duplicates
>>>
>>>
>>> Hey,
>>>
>>> We are using Checkpoint Log Exporter
>>>
> (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323)
>>> I don't think that it matters, except maybe for the fact that they
>>> are implementing RFC5424.
>>>
>>> The logs are being written to file by Rsyslog as follows:
>>>
>>> ---------------------
>>> module(load="imtcp")
>>> module(load="imudp")
>>>
>>> input(type="imtcp" port="514")
>>> input(type="imudp" port="514")
>>>
>>> template(name="FileNameByHost" type="string"
>>> string="/syslog/%fromhost-ip%.log")
>>> if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424
>>> action(type="omfile" dynaFile="FileNameByHost"
>>> template="RSYSLOG_SyslogProtocol23Format") <-------------------
>>> } else {
>>> action(type="omfile" dynaFile="FileNameByHost"
>>> template="RSYSLOG_FileFormat")
>>> }
>>>
>>> call asyncelk
>>> ...
>>> ---------------------
>>>
>>> Now Rsyslog thinks some messages are duplicates. And apparently it
>>> thinks there are _many_ duplicates. I have sniffed the traffic, and
>>> no duplicates are being transmitted AFAICS. Here is an example of
>>> what is logged nevertheless:
>>>
>>> <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 -
>>> [.action:"Reject"; ifdir:"inbound"; ifname:"eth2.53"; logid:"0";
>>> loguid:"{0x0,0x0,0x0,0x0}"; origin:"192.168.1.2";
>>> originsicname:"xxxxx"; sequencenum:"32"; time:"1570811271";
>>> version:"5"; dst:"224.0.0.18"; hll_key:"2008789500278145398";
>>> inzone:"Internal"; layer_name:"Unified"; layer_uuid:"a-b-c-d-e";
>>> match_id:"177"; parent_rule:"0"; rule_action:"Reject";
>>> rule_name:"xyz"; rule_uid:"a-b-c-d-e"; product:"VPN-1 & FireWall-1";
>>> proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message repeated 49
>>> times: []
>>>
>>> Other times it says "99 times" or "149 times", ... I've created somes
>>> stats in fact :D
>>>
>>> # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+
>>> times' | sort | uniq -c | sort -n | tail -10
>>>     23 repeated 44 times
>>>     26 repeated 199 times
>>>     35 repeated 159 times
>>>     37 repeated 9 times
>>>     38 repeated 4 times
>>>    101 repeated 149 times
>>>    155 repeated 109 times
>>>    408 repeated 59 times
>>>    417 repeated 99 times
>>> 2926 repeated 49 times
>>>
>>> This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64
>>>
>>> Any idea?
>>>
>>> Thanks,
>>> Marki
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 4:04 PM

Post #7 of 9 (591 views)

Permalink

On Fri, Oct 11, 2019 at 21:51:24 +0200, Rainer Gerhards via rsyslog wrote:
> The legacy code for this very old style "message repeated n times" does not
> consider rfc5424 structured data, so the empty MSG field makes the messages
> all equal.
>
> As David said, that feature is not useful any longer today. This is why it
> is turned off by default.

Marki, note that the configuration parameter which controls this
behavior is "$RepeatedMsgReduction".

Nathan
----------------------------------------------------------------------------
Nathan Stratton Treadway - nathanst@ontko.com - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 4:38 PM

Post #8 of 9 (591 views)

Permalink

Yep, I know :)
The default in SLES for this setting still seems to be On :-[

Thanks everyone!

On October 12, 2019 1:04:30 AM GMT+02:00, Nathan Stratton Treadway via rsyslog <rsyslog@lists.adiscon.com> wrote:
>On Fri, Oct 11, 2019 at 21:51:24 +0200, Rainer Gerhards via rsyslog
>wrote:
>> The legacy code for this very old style "message repeated n times"
>does not
>> consider rfc5424 structured data, so the empty MSG field makes the
>messages
>> all equal.
>>
>> As David said, that feature is not useful any longer today. This is
>why it
>> is turned off by default.
>
>Marki, note that the configuration parameter which controls this
>behavior is "$RepeatedMsgReduction".
>
> Nathan
>----------------------------------------------------------------------------
>Nathan Stratton Treadway - nathanst@ontko.com - Mid-Atlantic region
>Ray Ontko & Co. - Software consulting services -
>http://www.ontko.com/
>GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID:
>1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
>_______________________________________________
>rsyslog mailing list
>http://lists.adiscon.net/mailman/listinfo/rsyslog
>http://www.rsyslog.com/professional-services/
>What's up with rsyslog? Follow https://twitter.com/rgerhards
>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog seems to think there are duplicates [ In reply to ]

rsyslog at lists

Oct 11, 2019, 5:06 PM

Post #9 of 9 (591 views)

Permalink

It's a legacy thing, 20 years ago when everyone was reviewing logfiles by hand
ksyslogd was the default, and disks were small, it was made the default in
ksyslogd.

when the distros switched over to rsyslog, one of the attractive things was that
they could get equivalent functionality with almost no changes to the config
file, and they were looking to make the transition with as few changes in
behavior as possible, it was made explicitly the config default in just about
every distro's rsyslog.conf

David Lang

On Sat, 12 Oct 2019, marki via rsyslog wrote:

> Date: Sat, 12 Oct 2019 01:38:47 +0200
> From: marki via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: marki <rsyslog@lists.roth.lu>
> Subject: Re: [rsyslog] Rsyslog seems to think there are duplicates
>
> Yep, I know :)
> The default in SLES for this setting still seems to be On :-[
>
> Thanks everyone!
>
> On October 12, 2019 1:04:30 AM GMT+02:00, Nathan Stratton Treadway via rsyslog <rsyslog@lists.adiscon.com> wrote:
>> On Fri, Oct 11, 2019 at 21:51:24 +0200, Rainer Gerhards via rsyslog
>> wrote:
>>> The legacy code for this very old style "message repeated n times"
>> does not
>>> consider rfc5424 structured data, so the empty MSG field makes the
>> messages
>>> all equal.
>>>
>>> As David said, that feature is not useful any longer today. This is
>> why it
>>> is turned off by default.
>>
>> Marki, note that the configuration parameter which controls this
>> behavior is "$RepeatedMsgReduction".
>>
>> Nathan
>> ----------------------------------------------------------------------------
>> Nathan Stratton Treadway - nathanst@ontko.com - Mid-Atlantic region
>> Ray Ontko & Co. - Software consulting services -
>> http://www.ontko.com/
>> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID:
>> 1023D/ECFB6239
>> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.