Mailing List Archive: rsyslogd memory leak

rsyslogd memory leak

Feb 4, 2020, 5:32 AM

Post #1 of 7 (665 views)

I am seeing rsyslogd memory leak when remote server is unreachable.

I tried to change resumeRetryCount ="0" & queue.size ="0", so that rsyslogd
won't care much about retrying when remote server becomes reachable again.

When remote server is reachable, it stays at 1.5MB.
*/home/admin# ps_mem | grep rsyslog*
* 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*

When remote server is unreachable, it is increasing by 1 MB. And it goes
upto 2GB after 24 hours .

/home/admin# ps_mem | grep rsyslog
1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
/home/admin# ps_mem | grep rsyslog
1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
/home/admin# ps_mem | grep rsyslog
1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
/home/admin#
/home/admin# ps_mem | grep rsyslog
1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd

/home/admin# ps_mem | grep rsyslog
* 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*

/home/admin# date
Tue Feb 4 13:23:13 UTC 2020

/home/admin# ps_mem | grep rsyslog
* 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*

/home/admin# date
Tue Feb 4 13:26:32 UTC 2020
/home/admin# ps_mem | grep rsyslog
* 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*

When I made the remote server reachable again, it is reducing again

:/home/admin# date
Tue Feb 4 13:29:16 UTC 2020
:/home/admin# ps_mem | grep rsyslog
2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd

Could you please check this issue . Below is my remote configurations

*/home/admin# cat /etc/rsyslog.remote.conf*
$DefaultNetstreamDriverCAFile
/fs/security/certmgr/apps/syslog-client/syslog-client.crt
$DefaultNetstreamDriverCertFile
/fs/security/certmgr/apps/syslog-client/syslog-client.crt
$DefaultNetstreamDriverKeyFile
/fs/security/certmgr/apps/syslog-client/syslog-client.key
set $!updatepri = $syslogseverity + 184;
$template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
%APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
ruleset(name="logger1"){
action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
NetworkNamespace="swns" template="remoteFmt"Address="None"
*queue.type="LinkedList"
action.resumeRetryCount="-1" *queue.size="1000")
}

if($msg contains "Event|") then {
if $syslogseverity <= 6 then {
call logger1
}
}

--
*Regards,*
*Vinitha H*
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 4, 2020, 5:45 AM

Post #2 of 7 (665 views)

Permalink

That's probably not a memory leak, but rather rsyslog queuing the
messages in memory until the remote system is again available and the
queue can be drained.

Regards,

On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
> I am seeing rsyslogd memory leak when remote server is unreachable.
>
> I tried to change resumeRetryCount ="0" & queue.size ="0", so that rsyslogd
> won't care much about retrying when remote server becomes reachable again.
>
> When remote server is reachable, it stays at 1.5MB.
> */home/admin# ps_mem | grep rsyslog*
> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
>
>
> When remote server is unreachable, it is increasing by 1 MB. And it goes
> upto 2GB after 24 hours .
>
> /home/admin# ps_mem | grep rsyslog
> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> /home/admin# ps_mem | grep rsyslog
> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> /home/admin# ps_mem | grep rsyslog
> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> /home/admin#
> /home/admin# ps_mem | grep rsyslog
> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
>
> /home/admin# ps_mem | grep rsyslog
> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
>
> /home/admin# date
> Tue Feb 4 13:23:13 UTC 2020
>
> /home/admin# ps_mem | grep rsyslog
> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
>
>
> /home/admin# date
> Tue Feb 4 13:26:32 UTC 2020
> /home/admin# ps_mem | grep rsyslog
> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
>
>
> When I made the remote server reachable again, it is reducing again
>
> :/home/admin# date
> Tue Feb 4 13:29:16 UTC 2020
> :/home/admin# ps_mem | grep rsyslog
> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
>
>
> Could you please check this issue . Below is my remote configurations
>
> */home/admin# cat /etc/rsyslog.remote.conf*
> $DefaultNetstreamDriverCAFile
> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> $DefaultNetstreamDriverCertFile
> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> $DefaultNetstreamDriverKeyFile
> /fs/security/certmgr/apps/syslog-client/syslog-client.key
> set $!updatepri = $syslogseverity + 184;
> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
> ruleset(name="logger1"){
> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
> NetworkNamespace="swns" template="remoteFmt"Address="None"
> *queue.type="LinkedList"
> action.resumeRetryCount="-1" *queue.size="1000")
> }
>
> if($msg contains "Event|") then {
> if $syslogseverity <= 6 then {
> call logger1
> }
> }
>
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 4, 2020, 12:27 PM

Post #3 of 7 (664 views)

Permalink

enable impstats to see the queue sizes and resources consumed

David Lang

On Tue, 4 Feb 2020, John Chivian via rsyslog wrote:

> Date: Tue, 4 Feb 2020 07:45:35 -0600
> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: John Chivian <jchivian@chivian.com>
> Subject: Re: [rsyslog] rsyslogd memory leak
>
> That's probably not a memory leak, but rather rsyslog queuing the messages in
> memory until the remote system is again available and the queue can be
> drained.
>
> Regards,
>
>
> On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
>> I am seeing rsyslogd memory leak when remote server is unreachable.
>>
>> I tried to change resumeRetryCount ="0" & queue.size ="0", so that rsyslogd
>> won't care much about retrying when remote server becomes reachable again.
>>
>> When remote server is reachable, it stays at 1.5MB.
>> */home/admin# ps_mem | grep rsyslog*
>> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
>>
>>
>> When remote server is unreachable, it is increasing by 1 MB. And it goes
>> upto 2GB after 24 hours .
>>
>> /home/admin# ps_mem | grep rsyslog
>> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> /home/admin# ps_mem | grep rsyslog
>> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> /home/admin# ps_mem | grep rsyslog
>> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> /home/admin#
>> /home/admin# ps_mem | grep rsyslog
>> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
>>
>> /home/admin# ps_mem | grep rsyslog
>> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
>>
>> /home/admin# date
>> Tue Feb 4 13:23:13 UTC 2020
>>
>> /home/admin# ps_mem | grep rsyslog
>> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
>>
>>
>> /home/admin# date
>> Tue Feb 4 13:26:32 UTC 2020
>> /home/admin# ps_mem | grep rsyslog
>> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
>>
>>
>> When I made the remote server reachable again, it is reducing again
>>
>> :/home/admin# date
>> Tue Feb 4 13:29:16 UTC 2020
>> :/home/admin# ps_mem | grep rsyslog
>> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
>>
>>
>> Could you please check this issue . Below is my remote configurations
>>
>> */home/admin# cat /etc/rsyslog.remote.conf*
>> $DefaultNetstreamDriverCAFile
>> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
>> $DefaultNetstreamDriverCertFile
>> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
>> $DefaultNetstreamDriverKeyFile
>> /fs/security/certmgr/apps/syslog-client/syslog-client.key
>> set $!updatepri = $syslogseverity + 184;
>> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
>> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
>> ruleset(name="logger1"){
>> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
>> NetworkNamespace="swns" template="remoteFmt"Address="None"
>> *queue.type="LinkedList"
>> action.resumeRetryCount="-1" *queue.size="1000")
>> }
>>
>> if($msg contains "Event|") then {
>> if $syslogseverity <= 6 then {
>> call logger1
>> }
>> }
>>
>>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 4, 2020, 9:18 PM

Post #4 of 7 (664 views)

Permalink

Is this expected ?? How to l avoid buffering messages in rsyslogd ? Since
it is not acceptable that rsyslog consuming 2G + memory in 4G memory
system,
Which parameter determines this behavior ? I tried with queue size &
retrying logic, did;t seem to be working.

I have tried to load Impstats module. But it couldn't find module and logs
are below.
Is there anything I need to enable to load the module ?

Feb 05 05:16:37 rsyslogd[8064]: *could not load module
'/usr/lib/rsyslog/impstats.so', dlopen: /usr/lib/rsyslog/impstats.so: *cannot
open shared object file: No such file or directory [v8.36.0 try
http://www.rsyslog.com/e/2066 ]
Feb 05 05:16:37 systemd[1]: Started System Logging Service.
Feb 05 05:16:37 rsyslogd[8064]: imuxsock: Acquired UNIX socket
'/run/systemd/journal/syslog' (fd 3) from systemd. [v8.36.0]
Feb 05 05:16:37 rsyslogd[8064]: [origin software="*rsyslogd"
swVersion="8.36.0"* x-pid="8064" x-info="http://www.rsyslog.com"] start

On Wed, Feb 5, 2020 at 1:57 AM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> enable impstats to see the queue sizes and resources consumed
>
> David Lang
>
> On Tue, 4 Feb 2020, John Chivian via rsyslog wrote:
>
> > Date: Tue, 4 Feb 2020 07:45:35 -0600
> > From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: John Chivian <jchivian@chivian.com>
> > Subject: Re: [rsyslog] rsyslogd memory leak
> >
> > That's probably not a memory leak, but rather rsyslog queuing the
> messages in
> > memory until the remote system is again available and the queue can be
> > drained.
> >
> > Regards,
> >
> >
> > On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
> >> I am seeing rsyslogd memory leak when remote server is unreachable.
> >>
> >> I tried to change resumeRetryCount ="0" & queue.size ="0", so that
> rsyslogd
> >> won't care much about retrying when remote server becomes reachable
> again.
> >>
> >> When remote server is reachable, it stays at 1.5MB.
> >> */home/admin# ps_mem | grep rsyslog*
> >> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
> >>
> >>
> >> When remote server is unreachable, it is increasing by 1 MB. And it goes
> >> upto 2GB after 24 hours .
> >>
> >> /home/admin# ps_mem | grep rsyslog
> >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> >> /home/admin# ps_mem | grep rsyslog
> >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> >> /home/admin# ps_mem | grep rsyslog
> >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> >> /home/admin#
> >> /home/admin# ps_mem | grep rsyslog
> >> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
> >>
> >> /home/admin# ps_mem | grep rsyslog
> >> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
> >>
> >> /home/admin# date
> >> Tue Feb 4 13:23:13 UTC 2020
> >>
> >> /home/admin# ps_mem | grep rsyslog
> >> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
> >>
> >>
> >> /home/admin# date
> >> Tue Feb 4 13:26:32 UTC 2020
> >> /home/admin# ps_mem | grep rsyslog
> >> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
> >>
> >>
> >> When I made the remote server reachable again, it is reducing again
> >>
> >> :/home/admin# date
> >> Tue Feb 4 13:29:16 UTC 2020
> >> :/home/admin# ps_mem | grep rsyslog
> >> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
> >>
> >>
> >> Could you please check this issue . Below is my remote configurations
> >>
> >> */home/admin# cat /etc/rsyslog.remote.conf*
> >> $DefaultNetstreamDriverCAFile
> >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> >> $DefaultNetstreamDriverCertFile
> >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> >> $DefaultNetstreamDriverKeyFile
> >> /fs/security/certmgr/apps/syslog-client/syslog-client.key
> >> set $!updatepri = $syslogseverity + 184;
> >> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339%
> %HOSTNAME%
> >> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
> >> ruleset(name="logger1"){
> >> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
> >> NetworkNamespace="swns" template="remoteFmt"Address="None"
> >> *queue.type="LinkedList"
> >> action.resumeRetryCount="-1" *queue.size="1000")
> >> }
> >>
> >> if($msg contains "Event|") then {
> >> if $syslogseverity <= 6 then {
> >> call logger1
> >> }
> >> }
> >>
> >>
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>

--
*Regards,*
*Vinitha H*
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 5, 2020, 1:54 AM

Post #5 of 7 (664 views)

Permalink

2G+ memory is of course not a usual result, and it will not happen
with default config (we keep around 10.000 msgs in queues). I guess if
you show us your full config, we will be able to spot the problem.

Also note that 8.36 is veeeeery old - maybe there indeed is a memory
leak which is long-fixed.

HTH
Rainer

El mié., 5 feb. 2020 a las 6:19, Vini Harimoorthy via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Is this expected ?? How to l avoid buffering messages in rsyslogd ? Since
> it is not acceptable that rsyslog consuming 2G + memory in 4G memory
> system,
> Which parameter determines this behavior ? I tried with queue size &
> retrying logic, did;t seem to be working.
>
>
> I have tried to load Impstats module. But it couldn't find module and logs
> are below.
> Is there anything I need to enable to load the module ?
>
>
> Feb 05 05:16:37 rsyslogd[8064]: *could not load module
> '/usr/lib/rsyslog/impstats.so', dlopen: /usr/lib/rsyslog/impstats.so: *cannot
> open shared object file: No such file or directory [v8.36.0 try
> http://www.rsyslog.com/e/2066 ]
> Feb 05 05:16:37 systemd[1]: Started System Logging Service.
> Feb 05 05:16:37 rsyslogd[8064]: imuxsock: Acquired UNIX socket
> '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.36.0]
> Feb 05 05:16:37 rsyslogd[8064]: [origin software="*rsyslogd"
> swVersion="8.36.0"* x-pid="8064" x-info="http://www.rsyslog.com"] start
>
>
> On Wed, Feb 5, 2020 at 1:57 AM David Lang via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> > enable impstats to see the queue sizes and resources consumed
> >
> > David Lang
> >
> > On Tue, 4 Feb 2020, John Chivian via rsyslog wrote:
> >
> > > Date: Tue, 4 Feb 2020 07:45:35 -0600
> > > From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> > > To: rsyslog@lists.adiscon.com
> > > Cc: John Chivian <jchivian@chivian.com>
> > > Subject: Re: [rsyslog] rsyslogd memory leak
> > >
> > > That's probably not a memory leak, but rather rsyslog queuing the
> > messages in
> > > memory until the remote system is again available and the queue can be
> > > drained.
> > >
> > > Regards,
> > >
> > >
> > > On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
> > >> I am seeing rsyslogd memory leak when remote server is unreachable.
> > >>
> > >> I tried to change resumeRetryCount ="0" & queue.size ="0", so that
> > rsyslogd
> > >> won't care much about retrying when remote server becomes reachable
> > again.
> > >>
> > >> When remote server is reachable, it stays at 1.5MB.
> > >> */home/admin# ps_mem | grep rsyslog*
> > >> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
> > >>
> > >>
> > >> When remote server is unreachable, it is increasing by 1 MB. And it goes
> > >> upto 2GB after 24 hours .
> > >>
> > >> /home/admin# ps_mem | grep rsyslog
> > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > >> /home/admin# ps_mem | grep rsyslog
> > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > >> /home/admin# ps_mem | grep rsyslog
> > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > >> /home/admin#
> > >> /home/admin# ps_mem | grep rsyslog
> > >> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
> > >>
> > >> /home/admin# ps_mem | grep rsyslog
> > >> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
> > >>
> > >> /home/admin# date
> > >> Tue Feb 4 13:23:13 UTC 2020
> > >>
> > >> /home/admin# ps_mem | grep rsyslog
> > >> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
> > >>
> > >>
> > >> /home/admin# date
> > >> Tue Feb 4 13:26:32 UTC 2020
> > >> /home/admin# ps_mem | grep rsyslog
> > >> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
> > >>
> > >>
> > >> When I made the remote server reachable again, it is reducing again
> > >>
> > >> :/home/admin# date
> > >> Tue Feb 4 13:29:16 UTC 2020
> > >> :/home/admin# ps_mem | grep rsyslog
> > >> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
> > >>
> > >>
> > >> Could you please check this issue . Below is my remote configurations
> > >>
> > >> */home/admin# cat /etc/rsyslog.remote.conf*
> > >> $DefaultNetstreamDriverCAFile
> > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> > >> $DefaultNetstreamDriverCertFile
> > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> > >> $DefaultNetstreamDriverKeyFile
> > >> /fs/security/certmgr/apps/syslog-client/syslog-client.key
> > >> set $!updatepri = $syslogseverity + 184;
> > >> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339%
> > %HOSTNAME%
> > >> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
> > >> ruleset(name="logger1"){
> > >> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
> > >> NetworkNamespace="swns" template="remoteFmt"Address="None"
> > >> *queue.type="LinkedList"
> > >> action.resumeRetryCount="-1" *queue.size="1000")
> > >> }
> > >>
> > >> if($msg contains "Event|") then {
> > >> if $syslogseverity <= 6 then {
> > >> call logger1
> > >> }
> > >> }
> > >>
> > >>
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of
> > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T
> > > LIKE THAT.
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
>
>
> --
> *Regards,*
> *Vinitha H*
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 5, 2020, 4:03 AM

Post #6 of 7 (664 views)

Permalink

Please find the attached configurations.

On Wed, Feb 5, 2020 at 3:25 PM Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> 2G+ memory is of course not a usual result, and it will not happen
> with default config (we keep around 10.000 msgs in queues). I guess if
> you show us your full config, we will be able to spot the problem.
>
> Also note that 8.36 is veeeeery old - maybe there indeed is a memory
> leak which is long-fixed.
>
> HTH
> Rainer
>
> El mié., 5 feb. 2020 a las 6:19, Vini Harimoorthy via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > Is this expected ?? How to l avoid buffering messages in rsyslogd ? Since
> > it is not acceptable that rsyslog consuming 2G + memory in 4G memory
> > system,
> > Which parameter determines this behavior ? I tried with queue size &
> > retrying logic, did;t seem to be working.
> >
> >
> > I have tried to load Impstats module. But it couldn't find module and
> logs
> > are below.
> > Is there anything I need to enable to load the module ?
> >
> >
> > Feb 05 05:16:37 rsyslogd[8064]: *could not load module
> > '/usr/lib/rsyslog/impstats.so', dlopen: /usr/lib/rsyslog/impstats.so:
> *cannot
> > open shared object file: No such file or directory [v8.36.0 try
> > http://www.rsyslog.com/e/2066 ]
> > Feb 05 05:16:37 systemd[1]: Started System Logging Service.
> > Feb 05 05:16:37 rsyslogd[8064]: imuxsock: Acquired UNIX socket
> > '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.36.0]
> > Feb 05 05:16:37 rsyslogd[8064]: [origin software="*rsyslogd"
> > swVersion="8.36.0"* x-pid="8064" x-info="http://www.rsyslog.com"] start
> >
> >
> > On Wed, Feb 5, 2020 at 1:57 AM David Lang via rsyslog <
> > rsyslog@lists.adiscon.com> wrote:
> >
> > > enable impstats to see the queue sizes and resources consumed
> > >
> > > David Lang
> > >
> > > On Tue, 4 Feb 2020, John Chivian via rsyslog wrote:
> > >
> > > > Date: Tue, 4 Feb 2020 07:45:35 -0600
> > > > From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> > > > To: rsyslog@lists.adiscon.com
> > > > Cc: John Chivian <jchivian@chivian.com>
> > > > Subject: Re: [rsyslog] rsyslogd memory leak
> > > >
> > > > That's probably not a memory leak, but rather rsyslog queuing the
> > > messages in
> > > > memory until the remote system is again available and the queue can
> be
> > > > drained.
> > > >
> > > > Regards,
> > > >
> > > >
> > > > On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
> > > >> I am seeing rsyslogd memory leak when remote server is unreachable.
> > > >>
> > > >> I tried to change resumeRetryCount ="0" & queue.size ="0", so that
> > > rsyslogd
> > > >> won't care much about retrying when remote server becomes reachable
> > > again.
> > > >>
> > > >> When remote server is reachable, it stays at 1.5MB.
> > > >> */home/admin# ps_mem | grep rsyslog*
> > > >> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
> > > >>
> > > >>
> > > >> When remote server is unreachable, it is increasing by 1 MB. And it
> goes
> > > >> upto 2GB after 24 hours .
> > > >>
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
> > > >> /home/admin#
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
> > > >>
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
> > > >>
> > > >> /home/admin# date
> > > >> Tue Feb 4 13:23:13 UTC 2020
> > > >>
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
> > > >>
> > > >>
> > > >> /home/admin# date
> > > >> Tue Feb 4 13:26:32 UTC 2020
> > > >> /home/admin# ps_mem | grep rsyslog
> > > >> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
> > > >>
> > > >>
> > > >> When I made the remote server reachable again, it is reducing again
> > > >>
> > > >> :/home/admin# date
> > > >> Tue Feb 4 13:29:16 UTC 2020
> > > >> :/home/admin# ps_mem | grep rsyslog
> > > >> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
> > > >>
> > > >>
> > > >> Could you please check this issue . Below is my remote
> configurations
> > > >>
> > > >> */home/admin# cat /etc/rsyslog.remote.conf*
> > > >> $DefaultNetstreamDriverCAFile
> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> > > >> $DefaultNetstreamDriverCertFile
> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
> > > >> $DefaultNetstreamDriverKeyFile
> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.key
> > > >> set $!updatepri = $syslogseverity + 184;
> > > >> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339%
> > > %HOSTNAME%
> > > >> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
> > > >> ruleset(name="logger1"){
> > > >> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
> > > >> NetworkNamespace="swns" template="remoteFmt"Address="None"
> > > >> *queue.type="LinkedList"
> > > >> action.resumeRetryCount="-1" *queue.size="1000")
> > > >> }
> > > >>
> > > >> if($msg contains "Event|") then {
> > > >> if $syslogseverity <= 6 then {
> > > >> call logger1
> > > >> }
> > > >> }
> > > >>
> > > >>
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > of
> > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > > DON'T
> > > > LIKE THAT.
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > > DON'T LIKE THAT.
> > >
> >
> >
> > --
> > *Regards,*
> > *Vinitha H*
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>

--
*Regards,*
*Vinitha H*

Re: rsyslogd memory leak [ In reply to ]

rsyslog at lists

Feb 5, 2020, 4:21 AM

Post #7 of 7 (664 views)

Permalink

I don't see anything obviously wrong, except that the forwarding
action has action.resumeRetryCount="-1" which means eternal blocking.
Remove that param to make rsyslog throw away the messages when
transmission fails after a retry. But the queue sizes are low (1000)
and default for the main queue (whatever it was on that version). I do
not see anything that could consume 2GiB of main memory.

Conclusion: may actually be a bug. I suggest to move on to 8.2001.0
and see if the problem persists. I guess it will go away.

HTH
Rainer

El mié., 5 feb. 2020 a las 13:04, Vini Harimoorthy
(<vini6452@gmail.com>) escribió:
>
>
> Please find the attached configurations.
>
> On Wed, Feb 5, 2020 at 3:25 PM Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>>
>> 2G+ memory is of course not a usual result, and it will not happen
>> with default config (we keep around 10.000 msgs in queues). I guess if
>> you show us your full config, we will be able to spot the problem.
>>
>> Also note that 8.36 is veeeeery old - maybe there indeed is a memory
>> leak which is long-fixed.
>>
>> HTH
>> Rainer
>>
>> El mié., 5 feb. 2020 a las 6:19, Vini Harimoorthy via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>> >
>> > Is this expected ?? How to l avoid buffering messages in rsyslogd ? Since
>> > it is not acceptable that rsyslog consuming 2G + memory in 4G memory
>> > system,
>> > Which parameter determines this behavior ? I tried with queue size &
>> > retrying logic, did;t seem to be working.
>> >
>> >
>> > I have tried to load Impstats module. But it couldn't find module and logs
>> > are below.
>> > Is there anything I need to enable to load the module ?
>> >
>> >
>> > Feb 05 05:16:37 rsyslogd[8064]: *could not load module
>> > '/usr/lib/rsyslog/impstats.so', dlopen: /usr/lib/rsyslog/impstats.so: *cannot
>> > open shared object file: No such file or directory [v8.36.0 try
>> > http://www.rsyslog.com/e/2066 ]
>> > Feb 05 05:16:37 systemd[1]: Started System Logging Service.
>> > Feb 05 05:16:37 rsyslogd[8064]: imuxsock: Acquired UNIX socket
>> > '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.36.0]
>> > Feb 05 05:16:37 rsyslogd[8064]: [origin software="*rsyslogd"
>> > swVersion="8.36.0"* x-pid="8064" x-info="http://www.rsyslog.com"] start
>> >
>> >
>> > On Wed, Feb 5, 2020 at 1:57 AM David Lang via rsyslog <
>> > rsyslog@lists.adiscon.com> wrote:
>> >
>> > > enable impstats to see the queue sizes and resources consumed
>> > >
>> > > David Lang
>> > >
>> > > On Tue, 4 Feb 2020, John Chivian via rsyslog wrote:
>> > >
>> > > > Date: Tue, 4 Feb 2020 07:45:35 -0600
>> > > > From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>> > > > To: rsyslog@lists.adiscon.com
>> > > > Cc: John Chivian <jchivian@chivian.com>
>> > > > Subject: Re: [rsyslog] rsyslogd memory leak
>> > > >
>> > > > That's probably not a memory leak, but rather rsyslog queuing the
>> > > messages in
>> > > > memory until the remote system is again available and the queue can be
>> > > > drained.
>> > > >
>> > > > Regards,
>> > > >
>> > > >
>> > > > On 2/4/20 7:32 AM, Vini Harimoorthy via rsyslog wrote:
>> > > >> I am seeing rsyslogd memory leak when remote server is unreachable.
>> > > >>
>> > > >> I tried to change resumeRetryCount ="0" & queue.size ="0", so that
>> > > rsyslogd
>> > > >> won't care much about retrying when remote server becomes reachable
>> > > again.
>> > > >>
>> > > >> When remote server is reachable, it stays at 1.5MB.
>> > > >> */home/admin# ps_mem | grep rsyslog*
>> > > >> * 1.5 MiB + 31.5 KiB = 1.5 MiB rsyslogd*
>> > > >>
>> > > >>
>> > > >> When remote server is unreachable, it is increasing by 1 MB. And it goes
>> > > >> upto 2GB after 24 hours .
>> > > >>
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> 1.6 MiB + 31.5 KiB = 1.6 MiB rsyslogd
>> > > >> /home/admin#
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> 1.6 MiB + 31.5 KiB = 1.7 MiB rsyslogd
>> > > >>
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> * 2.4 MiB + 31.5 KiB = 2.5 MiB rsyslogd*
>> > > >>
>> > > >> /home/admin# date
>> > > >> Tue Feb 4 13:23:13 UTC 2020
>> > > >>
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> * 3.8 MiB + 31.5 KiB = 3.8 MiB rsyslogd*
>> > > >>
>> > > >>
>> > > >> /home/admin# date
>> > > >> Tue Feb 4 13:26:32 UTC 2020
>> > > >> /home/admin# ps_mem | grep rsyslog
>> > > >> * 5.2 MiB + 31.5 KiB = 5.2 MiB rsyslogd*
>> > > >>
>> > > >>
>> > > >> When I made the remote server reachable again, it is reducing again
>> > > >>
>> > > >> :/home/admin# date
>> > > >> Tue Feb 4 13:29:16 UTC 2020
>> > > >> :/home/admin# ps_mem | grep rsyslog
>> > > >> 2.6 MiB + 31.5 KiB = 2.6 MiB rsyslogd
>> > > >>
>> > > >>
>> > > >> Could you please check this issue . Below is my remote configurations
>> > > >>
>> > > >> */home/admin# cat /etc/rsyslog.remote.conf*
>> > > >> $DefaultNetstreamDriverCAFile
>> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
>> > > >> $DefaultNetstreamDriverCertFile
>> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.crt
>> > > >> $DefaultNetstreamDriverKeyFile
>> > > >> /fs/security/certmgr/apps/syslog-client/syslog-client.key
>> > > >> set $!updatepri = $syslogseverity + 184;
>> > > >> $template remoteFmt,"<%$!updatepri%>1 %TIMESTAMP:::date-rfc3339%
>> > > %HOSTNAME%
>> > > >> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%%msg%\n"
>> > > >> ruleset(name="logger1"){
>> > > >> action(type="omfwd" Target="20.20.20.2" Port="514" Protocol="tcp"
>> > > >> NetworkNamespace="swns" template="remoteFmt"Address="None"
>> > > >> *queue.type="LinkedList"
>> > > >> action.resumeRetryCount="-1" *queue.size="1000")
>> > > >> }
>> > > >>
>> > > >> if($msg contains "Event|") then {
>> > > >> if $syslogseverity <= 6 then {
>> > > >> call logger1
>> > > >> }
>> > > >> }
>> > > >>
>> > > >>
>> > > >
>> > > > _______________________________________________
>> > > > rsyslog mailing list
>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > http://www.rsyslog.com/professional-services/
>> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > > of
>> > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > > DON'T
>> > > > LIKE THAT.
>> > > >
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > > DON'T LIKE THAT.
>> > >
>> >
>> >
>> > --
>> > *Regards,*
>> > *Vinitha H*
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
>
> --
> Regards,
> Vinitha H
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Mailing List Archive

Mailing List Archive

Attached Files: