Mailing List Archive: Rate-limit: Cannot configure higher limit

Rate-limit: Cannot configure higher limit

Dec 14, 2022, 8:48 AM

Post #1 of 7 (365 views)

REF: Rsyslogd/ommysql.so: Not writing to DB intermittently

Rainer asked us to start a new post for the rate-limit issue.

A few of many hundreds of rate-limit errors and lost messages:

2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500
allowed within 5 seconds)
s

# date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
Wed Dec 14 10:35:41 CST 2022
$DebugFile /var/log/rsyslog.debug
$DebugLevel 2
module(load="imjournal" Ratelimit.Burst="30000" Ratelimit.Interval="1000"
StateFile="imjournal.state")
module(load="imklog")
module(load="immark")
module(load="impstats" interval="600" severity="7")
syslog.=debug /var/log/rsyslog-stats
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")
module(load="ommysql.so")
global(workDirectory="/var/lib/rsyslog")
authpriv.none;cron.none;*.info;mail.none /var/log/messages
authpriv.* /var/log/secure
cron.* /var/log/cron
*.emerg :omusrmsg:*
ftp.* /var/log/vsftpd.log
local7.* /var/log/boot.log
mail.* /var/log/maillog
uucp,news.crit /var/log/spooler
$ActionName Ftp
$ActionQueueFileName dbFtpQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
ftp.*
:ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
$ActionName Sftp
$ActionQueueFileName dbSftpQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
authpriv.*
:ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
$ActionName Admin
$ActionQueueFileName ZenossQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
*.* @@10.199.1.160
Wed Dec 14 10:35:41 CST 2022

Rainer asked us to setup a debug log, according to:
https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html

Initial startup here:
https://pastebin.com/DUgwmPC

No rate-limiting occurred since early yesterday (12/13) morning. This
appears to be associated with the errors and multi-line syslog entries
mentioned in the other post.

The sole intent of the database logging is tracking all incoming remote
file transfer (SFTP) activities. There is a firewall between this host and
the internet. Only "whitelisted" IP addresses can get through, and are to
be inserted into the database.

Apparently, at least one client connects in the early morning hours, and
this unusual SFTP unusual activity results in multi-line syslog entries
that come in very large numbers. One problem is, the multiple line entries
are not written to /var/log/messages, are not inserted into the database,
and rate-limiting obscures all content. Hence, this support request is our
attempt to understand what is happening, after which we can act to correct
these problems.

Interestingly, we are not aware of any missing files from this or any other
file transfer clients.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 14, 2022, 9:31 AM

Post #2 of 7 (365 views)

Permalink

I ignore the database logging issue. When you have rate-limiting
issues again, please report, together with the description of what
happens.

If you think this is related to mysql, please address that issue first.

Rainer

El mié, 14 dic 2022 a las 17:48, helices
(<mike+rsyslog@mdsresource.net>) escribió:
>
> REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
>
> Rainer asked us to start a new post for the rate-limit issue.
>
>
> A few of many hundreds of rate-limit errors and lost messages:
>
> 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]: rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500 allowed within 5 seconds)
> 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]: rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500 allowed within 5 seconds)
> 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]: rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500 allowed within 5 seconds)
> 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]: rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500 allowed within 5 seconds)
> s
>
>
> # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
> Wed Dec 14 10:35:41 CST 2022
> $DebugFile /var/log/rsyslog.debug
> $DebugLevel 2
> module(load="imjournal" Ratelimit.Burst="30000" Ratelimit.Interval="1000" StateFile="imjournal.state")
> module(load="imklog")
> module(load="immark")
> module(load="impstats" interval="600" severity="7")
> syslog.=debug /var/log/rsyslog-stats
> module(load="imtcp")
> input(type="imtcp" port="514")
> module(load="imudp")
> input(type="imudp" port="514")
> module(load="ommysql.so")
> global(workDirectory="/var/lib/rsyslog")
> authpriv.none;cron.none;*.info;mail.none /var/log/messages
> authpriv.* /var/log/secure
> cron.* /var/log/cron
> *.emerg :omusrmsg:*
> ftp.* /var/log/vsftpd.log
> local7.* /var/log/boot.log
> mail.* /var/log/maillog
> uucp,news.crit /var/log/spooler
> $ActionName Ftp
> $ActionQueueFileName dbFtpQueue # Set file name, also enables disk mode
> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> $ActionQueueType LinkedList # Use asynchronous processing
> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> ftp.* :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
> $ActionName Sftp
> $ActionQueueFileName dbSftpQueue # Set file name, also enables disk mode
> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> $ActionQueueType LinkedList # Use asynchronous processing
> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> authpriv.* :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
> $ActionName Admin
> $ActionQueueFileName ZenossQueue # Set file name, also enables disk mode
> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> $ActionQueueType LinkedList # Use asynchronous processing
> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> *.* @@10.199.1.160
> Wed Dec 14 10:35:41 CST 2022
>
>
> Rainer asked us to setup a debug log, according to:
> https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
>
> Initial startup here:
> https://pastebin.com/DUgwmPC
>
>
> No rate-limiting occurred since early yesterday (12/13) morning. This appears to be associated with the errors and multi-line syslog entries mentioned in the other post.
>
> The sole intent of the database logging is tracking all incoming remote file transfer (SFTP) activities. There is a firewall between this host and the internet. Only "whitelisted" IP addresses can get through, and are to be inserted into the database.
>
> Apparently, at least one client connects in the early morning hours, and this unusual SFTP unusual activity results in multi-line syslog entries that come in very large numbers. One problem is, the multiple line entries are not written to /var/log/messages, are not inserted into the database, and rate-limiting obscures all content. Hence, this support request is our attempt to understand what is happening, after which we can act to correct these problems.
>
> Interestingly, we are not aware of any missing files from this or any other file transfer clients.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 15, 2022, 1:38 PM

Post #3 of 7 (363 views)

Permalink

It happened again this afternoon:

2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]:
rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]:
rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]:
rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]:
rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500
allowed within 5 seconds)

On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> I ignore the database logging issue. When you have rate-limiting
> issues again, please report, together with the description of what
> happens.
>
> If you think this is related to mysql, please address that issue first.
>
> Rainer
>
> El mié, 14 dic 2022 a las 17:48, helices
> (<mike+rsyslog@mdsresource.net>) escribió:
> >
> > REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
> >
> > Rainer asked us to start a new post for the rate-limit issue.
> >
> >
> > A few of many hundreds of rate-limit errors and lost messages:
> >
> > 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> > 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> > 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> > 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> > s
> >
> >
> > # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
> > Wed Dec 14 10:35:41 CST 2022
> > $DebugFile /var/log/rsyslog.debug
> > $DebugLevel 2
> > module(load="imjournal" Ratelimit.Burst="30000"
> Ratelimit.Interval="1000" StateFile="imjournal.state")
> > module(load="imklog")
> > module(load="immark")
> > module(load="impstats" interval="600" severity="7")
> > syslog.=debug /var/log/rsyslog-stats
> > module(load="imtcp")
> > input(type="imtcp" port="514")
> > module(load="imudp")
> > input(type="imudp" port="514")
> > module(load="ommysql.so")
> > global(workDirectory="/var/lib/rsyslog")
> > authpriv.none;cron.none;*.info;mail.none /var/log/messages
> > authpriv.* /var/log/secure
> > cron.* /var/log/cron
> > *.emerg :omusrmsg:*
> > ftp.* /var/log/vsftpd.log
> > local7.* /var/log/boot.log
> > mail.* /var/log/maillog
> > uucp,news.crit /var/log/spooler
> > $ActionName Ftp
> > $ActionQueueFileName dbFtpQueue # Set file name, also enables disk mode
> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> > $ActionQueueType LinkedList # Use asynchronous processing
> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
> > ftp.*
> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
> > $ActionName Sftp
> > $ActionQueueFileName dbSftpQueue # Set file name, also enables disk
> mode
> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> > $ActionQueueType LinkedList # Use asynchronous processing
> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
> > authpriv.*
> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
> > $ActionName Admin
> > $ActionQueueFileName ZenossQueue # Set file name, also enables disk mode
> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> > $ActionQueueType LinkedList # Use asynchronous processing
> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
> > *.* @@10.199.1.160
> > Wed Dec 14 10:35:41 CST 2022
> >
> >
> > Rainer asked us to setup a debug log, according to:
> > https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
> >
> > Initial startup here:
> > https://pastebin.com/DUgwmPC
> >
> >
> > No rate-limiting occurred since early yesterday (12/13) morning. This
> appears to be associated with the errors and multi-line syslog entries
> mentioned in the other post.
> >
> > The sole intent of the database logging is tracking all incoming remote
> file transfer (SFTP) activities. There is a firewall between this host and
> the internet. Only "whitelisted" IP addresses can get through, and are to
> be inserted into the database.
> >
> > Apparently, at least one client connects in the early morning hours, and
> this unusual SFTP unusual activity results in multi-line syslog entries
> that come in very large numbers. One problem is, the multiple line entries
> are not written to /var/log/messages, are not inserted into the database,
> and rate-limiting obscures all content. Hence, this support request is our
> attempt to understand what is happening, after which we can act to correct
> these problems.
> >
> > Interestingly, we are not aware of any missing files from this or any
> other file transfer clients.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 15, 2022, 1:43 PM

Post #4 of 7 (363 views)

Permalink

did you post the full debug log at startup?

since you are attempting to set the limit higher, but this is showing the
default limit, there has to be something wrong with the config or the config
parsing.

since the trigger is only 500 logs in 5 seconds, you should be able to use
logger to generate this many messages rather than waiting for it to happen.

David Lang

On Thu, 15 Dec 2022, helices wrote:

> It happened again this afternoon:
>
> 2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500
> allowed within 5 seconds)
>
> On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
> wrote:
>
>> I ignore the database logging issue. When you have rate-limiting
>> issues again, please report, together with the description of what
>> happens.
>>
>> If you think this is related to mysql, please address that issue first.
>>
>> Rainer
>>
>> El mi?, 14 dic 2022 a las 17:48, helices
>> (<mike+rsyslog@mdsresource.net>) escribi?:
>>>
>>> REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
>>>
>>> Rainer asked us to start a new post for the rate-limit issue.
>>>
>>>
>>> A few of many hundreds of rate-limit errors and lost messages:
>>>
>>> 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>>> 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>>> 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>>> 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>>> s
>>>
>>>
>>> # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
>>> Wed Dec 14 10:35:41 CST 2022
>>> $DebugFile /var/log/rsyslog.debug
>>> $DebugLevel 2
>>> module(load="imjournal" Ratelimit.Burst="30000"
>> Ratelimit.Interval="1000" StateFile="imjournal.state")
>>> module(load="imklog")
>>> module(load="immark")
>>> module(load="impstats" interval="600" severity="7")
>>> syslog.=debug /var/log/rsyslog-stats
>>> module(load="imtcp")
>>> input(type="imtcp" port="514")
>>> module(load="imudp")
>>> input(type="imudp" port="514")
>>> module(load="ommysql.so")
>>> global(workDirectory="/var/lib/rsyslog")
>>> authpriv.none;cron.none;*.info;mail.none /var/log/messages
>>> authpriv.* /var/log/secure
>>> cron.* /var/log/cron
>>> *.emerg :omusrmsg:*
>>> ftp.* /var/log/vsftpd.log
>>> local7.* /var/log/boot.log
>>> mail.* /var/log/maillog
>>> uucp,news.crit /var/log/spooler
>>> $ActionName Ftp
>>> $ActionQueueFileName dbFtpQueue # Set file name, also enables disk mode
>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>> $ActionQueueType LinkedList # Use asynchronous processing
>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>> ftp.*
>> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
>>> $ActionName Sftp
>>> $ActionQueueFileName dbSftpQueue # Set file name, also enables disk
>> mode
>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>> $ActionQueueType LinkedList # Use asynchronous processing
>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>> authpriv.*
>> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
>>> $ActionName Admin
>>> $ActionQueueFileName ZenossQueue # Set file name, also enables disk mode
>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>> $ActionQueueType LinkedList # Use asynchronous processing
>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>> *.* @@10.199.1.160
>>> Wed Dec 14 10:35:41 CST 2022
>>>
>>>
>>> Rainer asked us to setup a debug log, according to:
>>> https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
>>>
>>> Initial startup here:
>>> https://pastebin.com/DUgwmPC
>>>
>>>
>>> No rate-limiting occurred since early yesterday (12/13) morning. This
>> appears to be associated with the errors and multi-line syslog entries
>> mentioned in the other post.
>>>
>>> The sole intent of the database logging is tracking all incoming remote
>> file transfer (SFTP) activities. There is a firewall between this host and
>> the internet. Only "whitelisted" IP addresses can get through, and are to
>> be inserted into the database.
>>>
>>> Apparently, at least one client connects in the early morning hours, and
>> this unusual SFTP unusual activity results in multi-line syslog entries
>> that come in very large numbers. One problem is, the multiple line entries
>> are not written to /var/log/messages, are not inserted into the database,
>> and rate-limiting obscures all content. Hence, this support request is our
>> attempt to understand what is happening, after which we can act to correct
>> these problems.
>>>
>>> Interestingly, we are not aware of any missing files from this or any
>> other file transfer clients.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 15, 2022, 2:05 PM

Post #5 of 7 (363 views)

Permalink

I've pared down the debug file to 18MB and bzip2 - too large for pastebin.

How an I get it to you?

On Thu, Dec 15, 2022 at 3:38 PM helices <mike+rsyslog@mdsresource.net>
wrote:

> It happened again this afternoon:
>
> 2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500
> allowed within 5 seconds)
> 2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]:
> rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500
> allowed within 5 seconds)
>
> On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
> wrote:
>
>> I ignore the database logging issue. When you have rate-limiting
>> issues again, please report, together with the description of what
>> happens.
>>
>> If you think this is related to mysql, please address that issue first.
>>
>> Rainer
>>
>> El mié, 14 dic 2022 a las 17:48, helices
>> (<mike+rsyslog@mdsresource.net>) escribió:
>> >
>> > REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
>> >
>> > Rainer asked us to start a new post for the rate-limit issue.
>> >
>> >
>> > A few of many hundreds of rate-limit errors and lost messages:
>> >
>> > 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> > 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> > 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> > 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
>> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> > s
>> >
>> >
>> > # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
>> > Wed Dec 14 10:35:41 CST 2022
>> > $DebugFile /var/log/rsyslog.debug
>> > $DebugLevel 2
>> > module(load="imjournal" Ratelimit.Burst="30000"
>> Ratelimit.Interval="1000" StateFile="imjournal.state")
>> > module(load="imklog")
>> > module(load="immark")
>> > module(load="impstats" interval="600" severity="7")
>> > syslog.=debug /var/log/rsyslog-stats
>> > module(load="imtcp")
>> > input(type="imtcp" port="514")
>> > module(load="imudp")
>> > input(type="imudp" port="514")
>> > module(load="ommysql.so")
>> > global(workDirectory="/var/lib/rsyslog")
>> > authpriv.none;cron.none;*.info;mail.none /var/log/messages
>> > authpriv.* /var/log/secure
>> > cron.* /var/log/cron
>> > *.emerg :omusrmsg:*
>> > ftp.* /var/log/vsftpd.log
>> > local7.* /var/log/boot.log
>> > mail.* /var/log/maillog
>> > uucp,news.crit /var/log/spooler
>> > $ActionName Ftp
>> > $ActionQueueFileName dbFtpQueue # Set file name, also enables disk
>> mode
>> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>> > $ActionQueueType LinkedList # Use asynchronous processing
>> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
>> > ftp.*
>> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
>> > $ActionName Sftp
>> > $ActionQueueFileName dbSftpQueue # Set file name, also enables disk
>> mode
>> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>> > $ActionQueueType LinkedList # Use asynchronous processing
>> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
>> > authpriv.*
>> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
>> > $ActionName Admin
>> > $ActionQueueFileName ZenossQueue # Set file name, also enables disk
>> mode
>> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>> > $ActionQueueType LinkedList # Use asynchronous processing
>> > $ActionResumeRetryCount -1 # Infinite retries on insert failure
>> > *.* @@10.199.1.160
>> > Wed Dec 14 10:35:41 CST 2022
>> >
>> >
>> > Rainer asked us to setup a debug log, according to:
>> > https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
>> >
>> > Initial startup here:
>> > https://pastebin.com/DUgwmPC
>> >
>> >
>> > No rate-limiting occurred since early yesterday (12/13) morning. This
>> appears to be associated with the errors and multi-line syslog entries
>> mentioned in the other post.
>> >
>> > The sole intent of the database logging is tracking all incoming remote
>> file transfer (SFTP) activities. There is a firewall between this host and
>> the internet. Only "whitelisted" IP addresses can get through, and are to
>> be inserted into the database.
>> >
>> > Apparently, at least one client connects in the early morning hours,
>> and this unusual SFTP unusual activity results in multi-line syslog entries
>> that come in very large numbers. One problem is, the multiple line entries
>> are not written to /var/log/messages, are not inserted into the database,
>> and rate-limiting obscures all content. Hence, this support request is our
>> attempt to understand what is happening, after which we can act to correct
>> these problems.
>> >
>> > Interestingly, we are not aware of any missing files from this or any
>> other file transfer clients.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 15, 2022, 2:44 PM

Post #6 of 7 (363 views)

Permalink

Yes - see pastebin link in 1st message.

On Thu, Dec 15, 2022, 15:43 David Lang <david@lang.hm> wrote:

> did you post the full debug log at startup?
>
> since you are attempting to set the limit higher, but this is showing the
> default limit, there has to be something wrong with the config or the
> config
> parsing.
>
> since the trigger is only 500 logs in 5 seconds, you should be able to use
> logger to generate this many messages rather than waiting for it to happen.
>
> David Lang
>
> On Thu, 15 Dec 2022, helices wrote:
>
> > It happened again this afternoon:
> >
> > 2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]:
> > rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500
> > allowed within 5 seconds)
> > 2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]:
> > rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500
> > allowed within 5 seconds)
> > 2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]:
> > rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500
> > allowed within 5 seconds)
> > 2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]:
> > rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500
> > allowed within 5 seconds)
> >
> > On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <
> rgerhards@hq.adiscon.com>
> > wrote:
> >
> >> I ignore the database logging issue. When you have rate-limiting
> >> issues again, please report, together with the description of what
> >> happens.
> >>
> >> If you think this is related to mysql, please address that issue first.
> >>
> >> Rainer
> >>
> >> El mié, 14 dic 2022 a las 17:48, helices
> >> (<mike+rsyslog@mdsresource.net>) escribió:
> >>>
> >>> REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
> >>>
> >>> Rainer asked us to start a new post for the rate-limit issue.
> >>>
> >>>
> >>> A few of many hundreds of rate-limit errors and lost messages:
> >>>
> >>> 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
> >> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting
> (500
> >> allowed within 5 seconds)
> >>> 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
> >> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting
> (500
> >> allowed within 5 seconds)
> >>> 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
> >> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting
> (500
> >> allowed within 5 seconds)
> >>> 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
> >> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting
> (500
> >> allowed within 5 seconds)
> >>> s
> >>>
> >>>
> >>> # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
> >>> Wed Dec 14 10:35:41 CST 2022
> >>> $DebugFile /var/log/rsyslog.debug
> >>> $DebugLevel 2
> >>> module(load="imjournal" Ratelimit.Burst="30000"
> >> Ratelimit.Interval="1000" StateFile="imjournal.state")
> >>> module(load="imklog")
> >>> module(load="immark")
> >>> module(load="impstats" interval="600" severity="7")
> >>> syslog.=debug /var/log/rsyslog-stats
> >>> module(load="imtcp")
> >>> input(type="imtcp" port="514")
> >>> module(load="imudp")
> >>> input(type="imudp" port="514")
> >>> module(load="ommysql.so")
> >>> global(workDirectory="/var/lib/rsyslog")
> >>> authpriv.none;cron.none;*.info;mail.none /var/log/messages
> >>> authpriv.* /var/log/secure
> >>> cron.* /var/log/cron
> >>> *.emerg :omusrmsg:*
> >>> ftp.* /var/log/vsftpd.log
> >>> local7.* /var/log/boot.log
> >>> mail.* /var/log/maillog
> >>> uucp,news.crit /var/log/spooler
> >>> $ActionName Ftp
> >>> $ActionQueueFileName dbFtpQueue # Set file name, also enables disk
> mode
> >>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> >>> $ActionQueueType LinkedList # Use asynchronous processing
> >>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> >>> ftp.*
> >> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
> >>> $ActionName Sftp
> >>> $ActionQueueFileName dbSftpQueue # Set file name, also enables disk
> >> mode
> >>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> >>> $ActionQueueType LinkedList # Use asynchronous processing
> >>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> >>> authpriv.*
> >> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
> >>> $ActionName Admin
> >>> $ActionQueueFileName ZenossQueue # Set file name, also enables disk
> mode
> >>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
> >>> $ActionQueueType LinkedList # Use asynchronous processing
> >>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
> >>> *.* @@10.199.1.160
> >>> Wed Dec 14 10:35:41 CST 2022
> >>>
> >>>
> >>> Rainer asked us to setup a debug log, according to:
> >>> https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
> >>>
> >>> Initial startup here:
> >>> https://pastebin.com/DUgwmPC
> >>>
> >>>
> >>> No rate-limiting occurred since early yesterday (12/13) morning. This
> >> appears to be associated with the errors and multi-line syslog entries
> >> mentioned in the other post.
> >>>
> >>> The sole intent of the database logging is tracking all incoming remote
> >> file transfer (SFTP) activities. There is a firewall between this host
> and
> >> the internet. Only "whitelisted" IP addresses can get through, and are
> to
> >> be inserted into the database.
> >>>
> >>> Apparently, at least one client connects in the early morning hours,
> and
> >> this unusual SFTP unusual activity results in multi-line syslog entries
> >> that come in very large numbers. One problem is, the multiple line
> entries
> >> are not written to /var/log/messages, are not inserted into the
> database,
> >> and rate-limiting obscures all content. Hence, this support request is
> our
> >> attempt to understand what is happening, after which we can act to
> correct
> >> these problems.
> >>>
> >>> Interestingly, we are not aware of any missing files from this or any
> >> other file transfer clients.
> >>
> >
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rate-limit: Cannot configure higher limit [ In reply to ]

rsyslog at lists

Dec 15, 2022, 3:26 PM

Post #7 of 7 (363 views)

Permalink

direct email will work for me (assuming your mail server can handle it)

the original pastebin link is no longer valid.

David Lang

On Thu, 15 Dec 2022, helices wrote:

> Date: Thu, 15 Dec 2022 16:05:11 -0600
> From: helices <mike+rsyslog@mdsresource.net>
> To: helices <mike+rsyslog@mdsresource.net>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>,
> Rainer Gerhards <rgerhards@hq.adiscon.com>, David Lang <david@lang.hm>
> Subject: Re: Rate-limit: Cannot configure higher limit
>
> I've pared down the debug file to 18MB and bzip2 - too large for pastebin.
>
> How an I get it to you?
>
> On Thu, Dec 15, 2022 at 3:38 PM helices <mike+rsyslog@mdsresource.net>
> wrote:
>
>> It happened again this afternoon:
>>
>> 2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]:
>> rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> 2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]:
>> rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> 2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]:
>> rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>> 2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]:
>> rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500
>> allowed within 5 seconds)
>>
>> On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
>> wrote:
>>
>>> I ignore the database logging issue. When you have rate-limiting
>>> issues again, please report, together with the description of what
>>> happens.
>>>
>>> If you think this is related to mysql, please address that issue first.
>>>
>>> Rainer
>>>
>>> El mi?, 14 dic 2022 a las 17:48, helices
>>> (<mike+rsyslog@mdsresource.net>) escribi?:
>>>>
>>>> REF: Rsyslogd/ommysql.so: Not writing to DB intermittently
>>>>
>>>> Rainer asked us to start a new post for the rate-limit issue.
>>>>
>>>>
>>>> A few of many hundreds of rate-limit errors and lost messages:
>>>>
>>>> 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]:
>>> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500
>>> allowed within 5 seconds)
>>>> 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]:
>>> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500
>>> allowed within 5 seconds)
>>>> 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]:
>>> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500
>>> allowed within 5 seconds)
>>>> 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]:
>>> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500
>>> allowed within 5 seconds)
>>>> s
>>>>
>>>>
>>>> # date; grep -v "^$#\|\s*$$" /etc/rsyslog.conf ;date
>>>> Wed Dec 14 10:35:41 CST 2022
>>>> $DebugFile /var/log/rsyslog.debug
>>>> $DebugLevel 2
>>>> module(load="imjournal" Ratelimit.Burst="30000"
>>> Ratelimit.Interval="1000" StateFile="imjournal.state")
>>>> module(load="imklog")
>>>> module(load="immark")
>>>> module(load="impstats" interval="600" severity="7")
>>>> syslog.=debug /var/log/rsyslog-stats
>>>> module(load="imtcp")
>>>> input(type="imtcp" port="514")
>>>> module(load="imudp")
>>>> input(type="imudp" port="514")
>>>> module(load="ommysql.so")
>>>> global(workDirectory="/var/lib/rsyslog")
>>>> authpriv.none;cron.none;*.info;mail.none /var/log/messages
>>>> authpriv.* /var/log/secure
>>>> cron.* /var/log/cron
>>>> *.emerg :omusrmsg:*
>>>> ftp.* /var/log/vsftpd.log
>>>> local7.* /var/log/boot.log
>>>> mail.* /var/log/maillog
>>>> uucp,news.crit /var/log/spooler
>>>> $ActionName Ftp
>>>> $ActionQueueFileName dbFtpQueue # Set file name, also enables disk
>>> mode
>>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>>> $ActionQueueType LinkedList # Use asynchronous processing
>>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>>> ftp.*
>>> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
>>>> $ActionName Sftp
>>>> $ActionQueueFileName dbSftpQueue # Set file name, also enables disk
>>> mode
>>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>>> $ActionQueueType LinkedList # Use asynchronous processing
>>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>>> authpriv.*
>>> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
>>>> $ActionName Admin
>>>> $ActionQueueFileName ZenossQueue # Set file name, also enables disk
>>> mode
>>>> $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
>>>> $ActionQueueType LinkedList # Use asynchronous processing
>>>> $ActionResumeRetryCount -1 # Infinite retries on insert failure
>>>> *.* @@10.199.1.160
>>>> Wed Dec 14 10:35:41 CST 2022
>>>>
>>>>
>>>> Rainer asked us to setup a debug log, according to:
>>>> https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html
>>>>
>>>> Initial startup here:
>>>> https://pastebin.com/DUgwmPC
>>>>
>>>>
>>>> No rate-limiting occurred since early yesterday (12/13) morning. This
>>> appears to be associated with the errors and multi-line syslog entries
>>> mentioned in the other post.
>>>>
>>>> The sole intent of the database logging is tracking all incoming remote
>>> file transfer (SFTP) activities. There is a firewall between this host and
>>> the internet. Only "whitelisted" IP addresses can get through, and are to
>>> be inserted into the database.
>>>>
>>>> Apparently, at least one client connects in the early morning hours,
>>> and this unusual SFTP unusual activity results in multi-line syslog entries
>>> that come in very large numbers. One problem is, the multiple line entries
>>> are not written to /var/log/messages, are not inserted into the database,
>>> and rate-limiting obscures all content. Hence, this support request is our
>>> attempt to understand what is happening, after which we can act to correct
>>> these problems.
>>>>
>>>> Interestingly, we are not aware of any missing files from this or any
>>> other file transfer clients.
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.