Hello (again)
I have a central rsyslog system which receives logs from 10 syslog relays via imptcp. After enabling impstats a few facts came to surface. It appears that my MainQ on central syslog system constantly has AVG of 150k objects.
In fact:
Messages/sec: ~45k
MainQ size: ~150k with spikes of 350k per minute (screenshot attached)
Looking at the documentation and trying to find some info, I got a bit confused on the way, so any help on the following questions are appreciated.
1. How one would interpret the fact that a MainQ size constantly fluctuates between 150k ~ 350k on a ~45k mps traffic.
Should your MainQ be always zero? Is there an issue having MainQ size of 150k?
2. I was thinking to tweak dequeue_batch_size and worker_threads parameters to bring MainQ size down, but I am not sure of what is currently set in defaults. I used as a reference the code in github for my current running version (8.2204)
https://github.com/rsyslog/rsyslog/blob/v8.2204.1/runtime/rsconf.c
globals.mainQ.iMainMsgQHighWtrMark = 80000;
globals.mainQ.iMainMsgQueDeqBatchSize = 256;
globals.mainQ.iMainMsgQueueNumWorkers = 2;
globals.mainQ.iMainMsgQLowWtrMark = 20000;
is the above correct, or rsyslog would use various factors on start-up to dynamically tweak above main q parameters?
The documentation* states that the default dequeue batch size for ruleset queues is 1024.
Which value is correct, 256 (github) or 1024 (Documentation)?
In general, what/where is the best place to get default values of rsyslog?
3. Is there a way to detect throttling
According to doc**, when queue is full , rsyslogd will throttle submitter (in my case, syslog relays). Is there a way to detect if/when my central syslog system is throttling my relays?
This is my config on Central busy syslog system
?*****************rsyslog.conf
module(
load="impstats"
interval="10"
resetCounters="on"
format="json"
ruleset="impstats"
)
module(load="omprog")
$LocalHostName central-syslog.mydomain.net
$MaxMessageSize 32k
# CIS
$umask 0000
$FileCreateMode 0640
$FileOwner root
$FileGroup logroup
$DirCreateMode 0750
$DirOwner root
$DirGroup logroup
$ModLoad imuxsock
$ModLoad imklog
$ModLoad immark
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$ModLoad imptcp
$InputPTCPServerRun 6514
$PreserveFQDN on
$MaxOpenFiles 16384
$ActionFileDefaultTemplate RSYSLOG_FileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
:msg, regex, ".* audit: .*" stop
:msg, regex, ".* kauditd_printk_skb: .*" stop
template(name="APPone_plain type="string"
string="/logs/app1-logs.log")
template(name="APPone_archive" type="string"
string="/logs/archive/app1-logs.log.gz")
if $syslogtag contains 'app-one' then {
action(type="omfile" DynaFile="APPone_plain" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
action(type="omfile" DynaFile="APPone_archive" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
stop
}
template(name="APPzero_plain type="string"
string="/logs/app_zero-logs.log")
template(name="APPone_archive" type="string"
string="/logs/archive/app_zero-logs.log.gz")
if $syslogtag contains 'app-zero' then {
action(type="omfile" DynaFile="APPzero_plain" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
action(type="omfile" DynaFile="APPzero_archive" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
stop
}
*.info;mail.none;authpriv.none;cron.none /logs/messages
# Log anything (except mail) of level info or higher.
# The authpriv file has restricted access.
authpriv.* /logs/auth
# Log all the mail messages in one place.
mail.* /logs/maillog
# Log Kernel messages
kern.* /logs/kernel
# Log cron stuff
cron.* /logs/cron
# Everybody gets emergency messages
*.emerg /logs/messages
# Save news errors of level crit and higher in a special file.
uucp,news.crit /logs/spooler
lpr.debug /logs/boot.log
# Save boot messages also to boot.log
local7.* /logs/boot.log
local6.* /logs/devs.log
local5.* /logs/various.log
local4.* /logs/session.log
local3.* /logs/messages
local2.* /logs/messages
local1.* /logs/history.log
$OMFileZipLevel 6
$template MAINARC,"/logs/archive/%$YEAR%/%$MONTH%/%$DAY%/all.log.gz"
*.* -?MAINARC
*********************************************
Thanks
D.
* https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
** https://www.rsyslog.com/doc/v8-stable/concepts/queues.html
?
I have a central rsyslog system which receives logs from 10 syslog relays via imptcp. After enabling impstats a few facts came to surface. It appears that my MainQ on central syslog system constantly has AVG of 150k objects.
In fact:
Messages/sec: ~45k
MainQ size: ~150k with spikes of 350k per minute (screenshot attached)
Looking at the documentation and trying to find some info, I got a bit confused on the way, so any help on the following questions are appreciated.
1. How one would interpret the fact that a MainQ size constantly fluctuates between 150k ~ 350k on a ~45k mps traffic.
Should your MainQ be always zero? Is there an issue having MainQ size of 150k?
2. I was thinking to tweak dequeue_batch_size and worker_threads parameters to bring MainQ size down, but I am not sure of what is currently set in defaults. I used as a reference the code in github for my current running version (8.2204)
https://github.com/rsyslog/rsyslog/blob/v8.2204.1/runtime/rsconf.c
globals.mainQ.iMainMsgQHighWtrMark = 80000;
globals.mainQ.iMainMsgQueDeqBatchSize = 256;
globals.mainQ.iMainMsgQueueNumWorkers = 2;
globals.mainQ.iMainMsgQLowWtrMark = 20000;
is the above correct, or rsyslog would use various factors on start-up to dynamically tweak above main q parameters?
The documentation* states that the default dequeue batch size for ruleset queues is 1024.
Which value is correct, 256 (github) or 1024 (Documentation)?
In general, what/where is the best place to get default values of rsyslog?
3. Is there a way to detect throttling
According to doc**, when queue is full , rsyslogd will throttle submitter (in my case, syslog relays). Is there a way to detect if/when my central syslog system is throttling my relays?
This is my config on Central busy syslog system
?*****************rsyslog.conf
module(
load="impstats"
interval="10"
resetCounters="on"
format="json"
ruleset="impstats"
)
module(load="omprog")
$LocalHostName central-syslog.mydomain.net
$MaxMessageSize 32k
# CIS
$umask 0000
$FileCreateMode 0640
$FileOwner root
$FileGroup logroup
$DirCreateMode 0750
$DirOwner root
$DirGroup logroup
$ModLoad imuxsock
$ModLoad imklog
$ModLoad immark
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$ModLoad imptcp
$InputPTCPServerRun 6514
$PreserveFQDN on
$MaxOpenFiles 16384
$ActionFileDefaultTemplate RSYSLOG_FileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
:msg, regex, ".* audit: .*" stop
:msg, regex, ".* kauditd_printk_skb: .*" stop
template(name="APPone_plain type="string"
string="/logs/app1-logs.log")
template(name="APPone_archive" type="string"
string="/logs/archive/app1-logs.log.gz")
if $syslogtag contains 'app-one' then {
action(type="omfile" DynaFile="APPone_plain" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
action(type="omfile" DynaFile="APPone_archive" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
stop
}
template(name="APPzero_plain type="string"
string="/logs/app_zero-logs.log")
template(name="APPone_archive" type="string"
string="/logs/archive/app_zero-logs.log.gz")
if $syslogtag contains 'app-zero' then {
action(type="omfile" DynaFile="APPzero_plain" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
action(type="omfile" DynaFile="APPzero_archive" Template="RSYSLOG_SyslogProtocol23Format"
DirCreateMode="0750" dirOwner="root" dirGroup="logroup"
FileCreateMode="0640" fileOwner="root" fileGroup="logroup")
stop
}
*.info;mail.none;authpriv.none;cron.none /logs/messages
# Log anything (except mail) of level info or higher.
# The authpriv file has restricted access.
authpriv.* /logs/auth
# Log all the mail messages in one place.
mail.* /logs/maillog
# Log Kernel messages
kern.* /logs/kernel
# Log cron stuff
cron.* /logs/cron
# Everybody gets emergency messages
*.emerg /logs/messages
# Save news errors of level crit and higher in a special file.
uucp,news.crit /logs/spooler
lpr.debug /logs/boot.log
# Save boot messages also to boot.log
local7.* /logs/boot.log
local6.* /logs/devs.log
local5.* /logs/various.log
local4.* /logs/session.log
local3.* /logs/messages
local2.* /logs/messages
local1.* /logs/history.log
$OMFileZipLevel 6
$template MAINARC,"/logs/archive/%$YEAR%/%$MONTH%/%$DAY%/all.log.gz"
*.* -?MAINARC
*********************************************
Thanks
D.
* https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
** https://www.rsyslog.com/doc/v8-stable/concepts/queues.html
?