Mailing List Archive

rsyslog Performance Tuning - Dropped UDP Events
Hey folks,

Hoping someone has the expertise to help me out here. We have a Syslog server running CentOS 7.9, kernel 5.18.5. It's acting as the centralized point for Syslog (TCP + UDP) ingestion for 100+ Syslog devices. Something is causing a what I think is the kernel (udp_queue_rcv_one_skb) to drop 10k+ packets in a single go, intermittently. I expect given the nature of UDP in general to lose some packets, but 5-10% of inbound packets is excessive, and I think, avoidable. This is our only server experiencing issues - though it's also the highest processing one by a significant margin. Dropwatch's output occurred in the span of 1-2 seconds. The security-config-omsagent.conf file is the rsyslog sub configuration for Forwarding to Microsoft Sentinel. That config has been slightly modified to remove actual subnet ranges. I've removed all local disk logging.

From my research online, this appears to be a queue issue. I've tried fixing it using the rsyslog documentation, but evidently haven't made it very far. Any thoughts or obvious mistakes?

Cheers,
Mike
System Config
CPUs: 8
RAM: 16GB
Disk Space: 2x 128GB disks (OS + DATA)
net.core.rmem_default = 33554432
net.core.rmem_max = 268435456
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.udp_rmem_min = 4096

Dropwatch -l kas
[cid:image001.png@01D8F80B.B4A87FE0]

/etc/rsyslog.conf
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
global(net.enableDNS="off")


#### MODULES ####
module(
load="impstats"
interval="1"
severity="7"
resetCounters="on"
log.file="/var/syslog/impstats.log"
log.syslog="off"
)

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
module(load="imudp" threads="16")
input(type="imudp" port="514" rcvbufSize="256m")

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### Templates ####
$template RemoteIP,"/var/syslog/%FROMHOST-IP%.log"
#$template NetApp,"%timestamp% %fromhost-ip% %msg%\n"
#$template Ubiquiti,"%msg% %fromhost-ip% Mystic-Ubiquiti\n"

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

# Performance Tuning #
$ActionQueueWorkerThreads 2000
$ActionQueueWorkerThreadMinimumMessages 1000
$ActionQueueSize 1000000
$ActionQueueDiscardMark 800000
$ActionQueueHighWaterMark 600000

#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail authpriv, cron)
# Dont log private authentication messages!
#*.*;mail.none;authpriv.none;cron.none ?RemoteIP

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
# local7.* /var/syslog/boot.log

/etc/rsyslog.d/security-config-omsagent.conf
# [Firewall Log Filtering] #
:msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
:msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
:msg, ereregex, "(1.8.68.[0-9]+)" stop
:msg, ereregex, "(1.9.69.[0-9]+)" stop
:msg, ereregex, "(1.10.82.[0-9]+)" stop
:msg, ereregex, "(IP multicast routing failed)" stop
:msg, ereregex, "(TCP_7680)" stop
if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
& stop
if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
& stop
local0.info @127.0.0.1:25224
& stop
local1.info @127.0.0.1:25224
& stop
local2.info @127.0.0.1:25224
& stop
local3.info @127.0.0.1:25224
& stop
local4.info @127.0.0.1:25224
& stop
local5.info @127.0.0.1:25224
& stop
local6.info @127.0.0.1:25224
& stop
local7.info @127.0.0.1:25224
& stop
auth.* @127.0.0.1:25224
& stop
authpriv.* @127.0.0.1:25224
& stop
daemon.info @127.0.0.1:25224
& stop
syslog.* @127.0.0.1:25224
& stop
ftp.*<ftp://ftp.*> @127.0.0.1:25224
& stop
user.* @127.0.0.1:25224
& stop



[cid:image002.jpg@01D8F80B.B4A87FE0]

Michael Redbourne (BCS)
Pronouns: he/him
Senior Security Analyst
Main (C): 1-506-230-3071
VOIP : 1-506-606-0384
Service Desk: 1-877-274-2349
www.bulletproofsi.com<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2Fwww.bulletproofsi.com%2F__%3B!!NFWRZ6kECLqu!pxjMi1iqf_xCGX9L_kgynzgYHwxIacBtBbFWe63ZKWM7Mwo1M0-12T9CrPB_dAqlKGcyX6wHnrGaPpRliSwsShf6YtO2onamhERr%24&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7C9ebf1cab2a5041e45f2108da8ffc7cd0%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C637980611840534929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k5y21GBCF2DAfVgRxUUfVsTifx7G036ytfI0%2BPolIgw%3D&reserved=0>


________________________________________
This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.

Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
16 threads for UDP receive is very incorrect. Rsyslog should only need a single
thread, even when receiving messages at a rate of hundreds of thousands of
messages/sec. too many threads will slow rsyslog down and it will use
recvmmesg() to pull multiple udp messages from the OS buffers in a single
syscall.

I would set the impstats to something a bit longer than a dump every second, go
to every 10 to every 60 seconds.

you may want to adjust the OS buffers for UDP to be much larger (sysctl)

can you show us the pstats output for a time under high load where it's losing
some messages? It's important to check that you are not filling the queues and
dropping messages because you aren't processing them fast enough.

Also run top and look at the per-thread data (hit 'H' to see the threads) and
see if any threads are hitting 100% cpu. If they are, then it's an indication to
possibly use additional threads somewhere or refactor the config a bit.

David Lang
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Apologies for the lack of response. Your response got thrown into my spam folder. You've been moved out of it.

- Impstats has been set to 30s intervals.
- Threads has been removed and changed back to default. That was from when I was initially research and assumed (incorrectly) that it was responsible for moving messages from the queue.

Section Modified in /etc/rsyslog.conf
#### MODULES ####
module(
load="impstats"
interval="30"
severity="7"
resetCounters="on"
log.file="/var/syslog/impstats.log"
log.syslog="off"
)

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
module(load="imudp")
input(type="imudp" port="514" rcvbufSize="256m")

=============

When you talk about setting sysctl for UDP - which settings are you referring too? These three?
net.ipv4.udp_mem = 381225 508301 762450
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096

I had previously set - which I believe are the master ones for general buffers. I can set the UDP ones too larger sizes as well. Would increasing the buffer size in rsyslog from 256m to 512m have any impact?
net.core.rmem_max = 256m
net.core.rmem_default = 32m

The syslog server is outside of it's high load times right now. It'll be ~12h until it's back under a full load with the various syslog clients reporting. What do you mean by pstats?

Thread Information
in:udp and in:tcp @ 1-2%.
In_syslog.rb (omsagent) spikes between 5-25%
output.rb (omsagent) is usually low, but will also spike to 25% when dumping logs back to Microsoft.


Michael Redbourne



-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Monday, November 14, 2022 9:52 AM
To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

16 threads for UDP receive is very incorrect. Rsyslog should only need a single thread, even when receiving messages at a rate of hundreds of thousands of messages/sec. too many threads will slow rsyslog down and it will use
recvmmesg() to pull multiple udp messages from the OS buffers in a single syscall.

I would set the impstats to something a bit longer than a dump every second, go to every 10 to every 60 seconds.

you may want to adjust the OS buffers for UDP to be much larger (sysctl)

can you show us the pstats output for a time under high load where it's losing some messages? It's important to check that you are not filling the queues and dropping messages because you aren't processing them fast enough.

Also run top and look at the per-thread data (hit 'H' to see the threads) and see if any threads are hitting 100% cpu. If they are, then it's an indication to possibly use additional threads somewhere or refactor the config a bit.

David Lang
________________________________________
This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.

Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Apologies for the lack of response. Your response got thrown into my spam folder. You've been moved out of it.
>
> - Impstats has been set to 30s intervals.
> - Threads has been removed and changed back to default. That was from when I was initially research and assumed (incorrectly) that it was responsible for moving messages from the queue.
>
> Section Modified in /etc/rsyslog.conf
> #### MODULES ####
> module(
> load="impstats"
> interval="30"
> severity="7"
> resetCounters="on"
> log.file="/var/syslog/impstats.log"
> log.syslog="off"
> )
>
> # The imjournal module bellow is now used as a message source instead of imuxsock.
> $ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
> $ModLoad imjournal # provides access to the systemd journal
> #$ModLoad imklog # reads kernel messages (the same are read from journald)
> #$ModLoad immark # provides --MARK-- message capability
>
> # Provides UDP syslog reception
> #$ModLoad imudp
> #$UDPServerRun 514
> module(load="imudp")
> input(type="imudp" port="514" rcvbufSize="256m")
>
> =============
>
> When you talk about setting sysctl for UDP - which settings are you referring too? These three?
> net.ipv4.udp_mem = 381225 508301 762450
> net.ipv4.udp_rmem_min = 4096
> net.ipv4.udp_wmem_min = 4096

increase udp_rmem should improve things. There is a stat somewhere that records
how many udp packets are dropped at the OS level (I know I've seen it in the
output of monitoring tools, so it has to be under /proc somewhere, but I'm not
sure where)

> I had previously set - which I believe are the master ones for general buffers. I can set the UDP ones too larger sizes as well. Would increasing the buffer size in rsyslog from 256m to 512m have any impact?
> net.core.rmem_max = 256m
> net.core.rmem_default = 32m
>
> The syslog server is outside of it's high load times right now. It'll be ~12h until it's back under a full load with the various syslog clients reporting. What do you mean by pstats?
>
> Thread Information
> in:udp and in:tcp @ 1-2%.

I expect these to stay fairly low, which would indicate that the problem isn't
in receiving the messages.

> In_syslog.rb (omsagent) spikes between 5-25%
> output.rb (omsagent) is usually low, but will also spike to 25% when dumping logs back to Microsoft.

I expect that as load climbs, this is going to be where you run into grief,
first this maxing out the cpu, then the queue size climbing as it can't process
messages fast enough, and when the queue is full, then the OS buffers will fill
up until they are full, at which point packets (logs) get dropped.

let's see if this is the case before doing more tinkering. Depending on how long
the peaks last, increasing the queue/buffer sizes may let you ride out the peak,
but if the peak lasts too long, you won't have enough ram to do so.

David Lang

>
> Michael Redbourne
>
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, November 14, 2022 9:52 AM
> To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> 16 threads for UDP receive is very incorrect. Rsyslog should only need a single thread, even when receiving messages at a rate of hundreds of thousands of messages/sec. too many threads will slow rsyslog down and it will use
> recvmmesg() to pull multiple udp messages from the OS buffers in a single syscall.
>
> I would set the impstats to something a bit longer than a dump every second, go to every 10 to every 60 seconds.
>
> you may want to adjust the OS buffers for UDP to be much larger (sysctl)
>
> can you show us the pstats output for a time under high load where it's losing some messages? It's important to check that you are not filling the queues and dropping messages because you aren't processing them fast enough.
>
> Also run top and look at the per-thread data (hit 'H' to see the threads) and see if any threads are hitting 100% cpu. If they are, then it's an indication to possibly use additional threads somewhere or refactor the config a bit.
>
> David Lang
> ________________________________________
> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>
> Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> ________________________________________
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Some additional comments on the config



These action queue configs probably don't do what you intend them to do

the first thing is that they only affect the next action, which is authpriv.* to
/var/log/secure and you configure 2000 threads to write these logs out. That
will create a HUGE amount of contention for the queue lock and under load you
should see it maxing out quite quickly

what is it that you are attempting to do here?



# Performance Tuning #
$ActionQueueWorkerThreads 2000
$ActionQueueWorkerThreadMinimumMessages 1000
$ActionQueueSize 1000000
$ActionQueueDiscardMark 800000
$ActionQueueHighWaterMark 600000

#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail authpriv, cron)
# Dont log private authentication messages!
#*.*;mail.none;authpriv.none;cron.none ?RemoteIP

# The authpriv file has restricted access.
authpriv.* /var/log/secure


since the queue only applied to the next action with this config, everything
below this is operating from the main queue again as if there was no action
queue configuration




# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
# local7.* /var/syslog/boot.log



ereregex is a fairly expensive filter to apply, it's much better to figure out a
non-regex approach to filtering these. Can you post some examples of what you
are trying to filter? mmnormalize to parse the logs and then make decisions on
the parsed results id probably much faster.


/etc/rsyslog.d/security-config-omsagent.conf
# [Firewall Log Filtering] #
:msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
:msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
:msg, ereregex, "(1.8.68.[0-9]+)" stop
:msg, ereregex, "(1.9.69.[0-9]+)" stop
:msg, ereregex, "(1.10.82.[0-9]+)" stop
:msg, ereregex, "(IP multicast routing failed)" stop
:msg, ereregex, "(TCP_7680)" stop


check the messages to see where CEF: and ASA- are in the message, can you filter
on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
instead of 'contains'?, again mmnormalize may be much faster

if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
& stop
if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
& stop


combining multiple filters into one action, or having the filters call a ruleset
can be far more efficient than all of them writing things out independently.

the if..then filter structure lets you easily combine filters

local0.info @127.0.0.1:25224
& stop
local1.info @127.0.0.1:25224
& stop
local2.info @127.0.0.1:25224
& stop
local3.info @127.0.0.1:25224
& stop
local4.info @127.0.0.1:25224
& stop
local5.info @127.0.0.1:25224
& stop
local6.info @127.0.0.1:25224
& stop
local7.info @127.0.0.1:25224
& stop
auth.* @127.0.0.1:25224
& stop
authpriv.* @127.0.0.1:25224
& stop
daemon.info @127.0.0.1:25224
& stop
syslog.* @127.0.0.1:25224
& stop
ftp.*<ftp://ftp.*> @127.0.0.1:25224
& stop
user.* @127.0.0.1:25224
& stop
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
let me add: look into setting imudp to realtime priority. Doc:

https://www.rsyslog.com/doc/master/configuration/modules/imudp.html

Rainer

El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Some additional comments on the config
>
>
>
> These action queue configs probably don't do what you intend them to do
>
> the first thing is that they only affect the next action, which is authpriv.* to
> /var/log/secure and you configure 2000 threads to write these logs out. That
> will create a HUGE amount of contention for the queue lock and under load you
> should see it maxing out quite quickly
>
> what is it that you are attempting to do here?
>
>
>
> # Performance Tuning #
> $ActionQueueWorkerThreads 2000
> $ActionQueueWorkerThreadMinimumMessages 1000
> $ActionQueueSize 1000000
> $ActionQueueDiscardMark 800000
> $ActionQueueHighWaterMark 600000
>
> #### RULES ####
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail authpriv, cron)
> # Dont log private authentication messages!
> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
>
> since the queue only applied to the next action with this config, everything
> below this is operating from the main queue again as if there was no action
> queue configuration
>
>
>
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg :omusrmsg:*
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> # local7.* /var/syslog/boot.log
>
>
>
> ereregex is a fairly expensive filter to apply, it's much better to figure out a
> non-regex approach to filtering these. Can you post some examples of what you
> are trying to filter? mmnormalize to parse the logs and then make decisions on
> the parsed results id probably much faster.
>
>
> /etc/rsyslog.d/security-config-omsagent.conf
> # [Firewall Log Filtering] #
> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
> :msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
> :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
> :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
> :msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
> :msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
> :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
> :msg, ereregex, "(1.8.68.[0-9]+)" stop
> :msg, ereregex, "(1.9.69.[0-9]+)" stop
> :msg, ereregex, "(1.10.82.[0-9]+)" stop
> :msg, ereregex, "(IP multicast routing failed)" stop
> :msg, ereregex, "(TCP_7680)" stop
>
>
> check the messages to see where CEF: and ASA- are in the message, can you filter
> on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> instead of 'contains'?, again mmnormalize may be much faster
>
> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
> & stop
> if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
> & stop
>
>
> combining multiple filters into one action, or having the filters call a ruleset
> can be far more efficient than all of them writing things out independently.
>
> the if..then filter structure lets you easily combine filters
>
> local0.info @127.0.0.1:25224
> & stop
> local1.info @127.0.0.1:25224
> & stop
> local2.info @127.0.0.1:25224
> & stop
> local3.info @127.0.0.1:25224
> & stop
> local4.info @127.0.0.1:25224
> & stop
> local5.info @127.0.0.1:25224
> & stop
> local6.info @127.0.0.1:25224
> & stop
> local7.info @127.0.0.1:25224
> & stop
> auth.* @127.0.0.1:25224
> & stop
> authpriv.* @127.0.0.1:25224
> & stop
> daemon.info @127.0.0.1:25224
> & stop
> syslog.* @127.0.0.1:25224
> & stop
> ftp.*<ftp://ftp.*> @127.0.0.1:25224
> & stop
> user.* @127.0.0.1:25224
> & stop
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
I haven't needed to do that to handle 300k messages/sec on UDP input (usually I
run into bottlenecks in processing the messages long before I have problems
accepting them)

David Lang

On Tue, 15 Nov 2022, Rainer Gerhards wrote:

> let me add: look into setting imudp to realtime priority. Doc:
>
> https://www.rsyslog.com/doc/master/configuration/modules/imudp.html
>
> Rainer
>
> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> Some additional comments on the config
>>
>>
>>
>> These action queue configs probably don't do what you intend them to do
>>
>> the first thing is that they only affect the next action, which is authpriv.* to
>> /var/log/secure and you configure 2000 threads to write these logs out. That
>> will create a HUGE amount of contention for the queue lock and under load you
>> should see it maxing out quite quickly
>>
>> what is it that you are attempting to do here?
>>
>>
>>
>> # Performance Tuning #
>> $ActionQueueWorkerThreads 2000
>> $ActionQueueWorkerThreadMinimumMessages 1000
>> $ActionQueueSize 1000000
>> $ActionQueueDiscardMark 800000
>> $ActionQueueHighWaterMark 600000
>>
>> #### RULES ####
>> # Log all kernel messages to the console.
>> # Logging much else clutters up the screen.
>> #kern.* /dev/console
>>
>> # Log anything (except mail authpriv, cron)
>> # Dont log private authentication messages!
>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>
>> # The authpriv file has restricted access.
>> authpriv.* /var/log/secure
>>
>>
>> since the queue only applied to the next action with this config, everything
>> below this is operating from the main queue again as if there was no action
>> queue configuration
>>
>>
>>
>>
>> # Log all the mail messages in one place.
>> mail.* -/var/log/maillog
>>
>> # Log cron stuff
>> cron.* /var/log/cron
>>
>> # Everybody gets emergency messages
>> *.emerg :omusrmsg:*
>>
>> # Save news errors of level crit and higher in a special file.
>> uucp,news.crit /var/log/spooler
>>
>> # Save boot messages also to boot.log
>> # local7.* /var/syslog/boot.log
>>
>>
>>
>> ereregex is a fairly expensive filter to apply, it's much better to figure out a
>> non-regex approach to filtering these. Can you post some examples of what you
>> are trying to filter? mmnormalize to parse the logs and then make decisions on
>> the parsed results id probably much faster.
>>
>>
>> /etc/rsyslog.d/security-config-omsagent.conf
>> # [Firewall Log Filtering] #
>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
>> :msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
>> :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
>> :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
>> :msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
>> :msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
>> :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
>> :msg, ereregex, "(1.8.68.[0-9]+)" stop
>> :msg, ereregex, "(1.9.69.[0-9]+)" stop
>> :msg, ereregex, "(1.10.82.[0-9]+)" stop
>> :msg, ereregex, "(IP multicast routing failed)" stop
>> :msg, ereregex, "(TCP_7680)" stop
>>
>>
>> check the messages to see where CEF: and ASA- are in the message, can you filter
>> on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>> instead of 'contains'?, again mmnormalize may be much faster
>>
>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
>> & stop
>> if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
>> & stop
>>
>>
>> combining multiple filters into one action, or having the filters call a ruleset
>> can be far more efficient than all of them writing things out independently.
>>
>> the if..then filter structure lets you easily combine filters
>>
>> local0.info @127.0.0.1:25224
>> & stop
>> local1.info @127.0.0.1:25224
>> & stop
>> local2.info @127.0.0.1:25224
>> & stop
>> local3.info @127.0.0.1:25224
>> & stop
>> local4.info @127.0.0.1:25224
>> & stop
>> local5.info @127.0.0.1:25224
>> & stop
>> local6.info @127.0.0.1:25224
>> & stop
>> local7.info @127.0.0.1:25224
>> & stop
>> auth.* @127.0.0.1:25224
>> & stop
>> authpriv.* @127.0.0.1:25224
>> & stop
>> daemon.info @127.0.0.1:25224
>> & stop
>> syslog.* @127.0.0.1:25224
>> & stop
>> ftp.*<ftp://ftp.*> @127.0.0.1:25224
>> & stop
>> user.* @127.0.0.1:25224
>> & stop
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Just wanted to make sure awareness of that option. Agree that it is
not often needed.

Rainer

El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribió:
>
> I haven't needed to do that to handle 300k messages/sec on UDP input (usually I
> run into bottlenecks in processing the messages long before I have problems
> accepting them)
>
> David Lang
>
> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>
> > let me add: look into setting imudp to realtime priority. Doc:
> >
> > https://www.rsyslog.com/doc/master/configuration/modules/imudp.html
> >
> > Rainer
> >
> > El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> >>
> >> Some additional comments on the config
> >>
> >>
> >>
> >> These action queue configs probably don't do what you intend them to do
> >>
> >> the first thing is that they only affect the next action, which is authpriv.* to
> >> /var/log/secure and you configure 2000 threads to write these logs out. That
> >> will create a HUGE amount of contention for the queue lock and under load you
> >> should see it maxing out quite quickly
> >>
> >> what is it that you are attempting to do here?
> >>
> >>
> >>
> >> # Performance Tuning #
> >> $ActionQueueWorkerThreads 2000
> >> $ActionQueueWorkerThreadMinimumMessages 1000
> >> $ActionQueueSize 1000000
> >> $ActionQueueDiscardMark 800000
> >> $ActionQueueHighWaterMark 600000
> >>
> >> #### RULES ####
> >> # Log all kernel messages to the console.
> >> # Logging much else clutters up the screen.
> >> #kern.* /dev/console
> >>
> >> # Log anything (except mail authpriv, cron)
> >> # Dont log private authentication messages!
> >> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>
> >> # The authpriv file has restricted access.
> >> authpriv.* /var/log/secure
> >>
> >>
> >> since the queue only applied to the next action with this config, everything
> >> below this is operating from the main queue again as if there was no action
> >> queue configuration
> >>
> >>
> >>
> >>
> >> # Log all the mail messages in one place.
> >> mail.* -/var/log/maillog
> >>
> >> # Log cron stuff
> >> cron.* /var/log/cron
> >>
> >> # Everybody gets emergency messages
> >> *.emerg :omusrmsg:*
> >>
> >> # Save news errors of level crit and higher in a special file.
> >> uucp,news.crit /var/log/spooler
> >>
> >> # Save boot messages also to boot.log
> >> # local7.* /var/syslog/boot.log
> >>
> >>
> >>
> >> ereregex is a fairly expensive filter to apply, it's much better to figure out a
> >> non-regex approach to filtering these. Can you post some examples of what you
> >> are trying to filter? mmnormalize to parse the logs and then make decisions on
> >> the parsed results id probably much faster.
> >>
> >>
> >> /etc/rsyslog.d/security-config-omsagent.conf
> >> # [Firewall Log Filtering] #
> >> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
> >> :msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
> >> :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
> >> :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
> >> :msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
> >> :msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
> >> :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
> >> :msg, ereregex, "(1.8.68.[0-9]+)" stop
> >> :msg, ereregex, "(1.9.69.[0-9]+)" stop
> >> :msg, ereregex, "(1.10.82.[0-9]+)" stop
> >> :msg, ereregex, "(IP multicast routing failed)" stop
> >> :msg, ereregex, "(TCP_7680)" stop
> >>
> >>
> >> check the messages to see where CEF: and ASA- are in the message, can you filter
> >> on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> >> instead of 'contains'?, again mmnormalize may be much faster
> >>
> >> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
> >> & stop
> >> if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
> >> & stop
> >>
> >>
> >> combining multiple filters into one action, or having the filters call a ruleset
> >> can be far more efficient than all of them writing things out independently.
> >>
> >> the if..then filter structure lets you easily combine filters
> >>
> >> local0.info @127.0.0.1:25224
> >> & stop
> >> local1.info @127.0.0.1:25224
> >> & stop
> >> local2.info @127.0.0.1:25224
> >> & stop
> >> local3.info @127.0.0.1:25224
> >> & stop
> >> local4.info @127.0.0.1:25224
> >> & stop
> >> local5.info @127.0.0.1:25224
> >> & stop
> >> local6.info @127.0.0.1:25224
> >> & stop
> >> local7.info @127.0.0.1:25224
> >> & stop
> >> auth.* @127.0.0.1:25224
> >> & stop
> >> authpriv.* @127.0.0.1:25224
> >> & stop
> >> daemon.info @127.0.0.1:25224
> >> & stop
> >> syslog.* @127.0.0.1:25224
> >> & stop
> >> ftp.*<ftp://ftp.*> @127.0.0.1:25224
> >> & stop
> >> user.* @127.0.0.1:25224
> >> & stop
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
Udp:
5820820 packets received
1504 packets to unknown port received.
798900 packet receive errors
3338814 packets sent
798900 receive buffer errors
0 send buffer errors

I have doubled the values in net.ipv4.udp_mem.

The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).

The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop

Example Checkpoint Log:
CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> dvc=<dvc_ip_addr>

That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.

Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer Gerhards via rsyslog
Sent: Tuesday, November 15, 2022 5:11 AM
To: David Lang <david@lang.hm>
Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

Just wanted to make sure awareness of that option. Agree that it is not often needed.

Rainer

El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>
> I haven't needed to do that to handle 300k messages/sec on UDP input
> (usually I run into bottlenecks in processing the messages long before
> I have problems accepting them)
>
> David Lang
>
> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>
> > let me add: look into setting imudp to realtime priority. Doc:
> >
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> > w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
> > amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
> > 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
> > C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
> > =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
> >
> > Rainer
> >
> > El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribi?:
> >>
> >> Some additional comments on the config
> >>
> >>
> >>
> >> These action queue configs probably don't do what you intend them
> >> to do
> >>
> >> the first thing is that they only affect the next action, which is
> >> authpriv.* to /var/log/secure and you configure 2000 threads to
> >> write these logs out. That will create a HUGE amount of contention
> >> for the queue lock and under load you should see it maxing out
> >> quite quickly
> >>
> >> what is it that you are attempting to do here?
> >>
> >>
> >>
> >> # Performance Tuning #
> >> $ActionQueueWorkerThreads 2000
> >> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
> >> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
> >> 600000
> >>
> >> #### RULES ####
> >> # Log all kernel messages to the console.
> >> # Logging much else clutters up the screen.
> >> #kern.* /dev/console
> >>
> >> # Log anything (except mail authpriv, cron) # Dont log private
> >> authentication messages!
> >> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>
> >> # The authpriv file has restricted access.
> >> authpriv.* /var/log/secure
> >>
> >>
> >> since the queue only applied to the next action with this config,
> >> everything below this is operating from the main queue again as if
> >> there was no action queue configuration
> >>
> >>
> >>
> >>
> >> # Log all the mail messages in one place.
> >> mail.* -/var/log/maillog
> >>
> >> # Log cron stuff
> >> cron.* /var/log/cron
> >>
> >> # Everybody gets emergency messages
> >> *.emerg :omusrmsg:*
> >>
> >> # Save news errors of level crit and higher in a special file.
> >> uucp,news.crit /var/log/spooler
> >>
> >> # Save boot messages also to boot.log
> >> # local7.* /var/syslog/boot.log
> >>
> >>
> >>
> >> ereregex is a fairly expensive filter to apply, it's much better to
> >> figure out a non-regex approach to filtering these. Can you post
> >> some examples of what you are trying to filter? mmnormalize to
> >> parse the logs and then make decisions on the parsed results id probably much faster.
> >>
> >>
> >> /etc/rsyslog.d/security-config-omsagent.conf
> >> # [Firewall Log Filtering] #
> >> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
> >> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
> >> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
> >> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
> >> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
> >> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
> >>
> >>
> >> check the messages to see where CEF: and ASA- are in the message,
> >> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> >> instead of 'contains'?, again mmnormalize may be much faster
> >>
> >> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
> >> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
> >> then @127.0.0.1:25224 & stop
> >>
> >>
> >> combining multiple filters into one action, or having the filters
> >> call a ruleset can be far more efficient than all of them writing things out independently.
> >>
> >> the if..then filter structure lets you easily combine filters
> >>
> >> local0.info @127.0.0.1:25224
> >> & stop
> >> local1.info @127.0.0.1:25224
> >> & stop
> >> local2.info @127.0.0.1:25224
> >> & stop
> >> local3.info @127.0.0.1:25224
> >> & stop
> >> local4.info @127.0.0.1:25224
> >> & stop
> >> local5.info @127.0.0.1:25224
> >> & stop
> >> local6.info @127.0.0.1:25224
> >> & stop
> >> local7.info @127.0.0.1:25224
> >> & stop
> >> auth.* @127.0.0.1:25224
> >> & stop
> >> authpriv.* @127.0.0.1:25224
> >> & stop
> >> daemon.info @127.0.0.1:25224
> >> & stop
> >> syslog.* @127.0.0.1:25224
> >> & stop
> >> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
> >> user.* @127.0.0.1:25224
> >> & stop
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> >> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
> >> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
> >> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
> >> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> >> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
> >> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
> >> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
> >> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
> >> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
> >> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
> >> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> >> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
> >> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
> >> What's up with rsyslog? Follow
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
> >> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
> >> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
> >> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
> >> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
> >> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
> >> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQynhVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6IonLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OYkHGY%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
________________________________________
This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.

Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Building on this -

When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
In:imdup spikes to ~10%
in_syslog.rb spikes to 90-100% usage
rs:main Q:Reg spikes to 25% usage.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Redbourne,Michael via rsyslog
Sent: Tuesday, November 15, 2022 8:42 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
Udp:
5820820 packets received
1504 packets to unknown port received.
798900 packet receive errors
3338814 packets sent
798900 receive buffer errors
0 send buffer errors

I have doubled the values in net.ipv4.udp_mem.

The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).

The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop

Example Checkpoint Log:
CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> dvc=<dvc_ip_addr>

That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.

Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer Gerhards via rsyslog
Sent: Tuesday, November 15, 2022 5:11 AM
To: David Lang <david@lang.hm>
Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

Just wanted to make sure awareness of that option. Agree that it is not often needed.

Rainer

El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>
> I haven't needed to do that to handle 300k messages/sec on UDP input
> (usually I run into bottlenecks in processing the messages long before
> I have problems accepting them)
>
> David Lang
>
> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>
> > let me add: look into setting imudp to realtime priority. Doc:
> >
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> > w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
> > amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
> > 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
> > C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
> > =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
> >
> > Rainer
> >
> > El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribi?:
> >>
> >> Some additional comments on the config
> >>
> >>
> >>
> >> These action queue configs probably don't do what you intend them
> >> to do
> >>
> >> the first thing is that they only affect the next action, which is
> >> authpriv.* to /var/log/secure and you configure 2000 threads to
> >> write these logs out. That will create a HUGE amount of contention
> >> for the queue lock and under load you should see it maxing out
> >> quite quickly
> >>
> >> what is it that you are attempting to do here?
> >>
> >>
> >>
> >> # Performance Tuning #
> >> $ActionQueueWorkerThreads 2000
> >> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
> >> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
> >> 600000
> >>
> >> #### RULES ####
> >> # Log all kernel messages to the console.
> >> # Logging much else clutters up the screen.
> >> #kern.* /dev/console
> >>
> >> # Log anything (except mail authpriv, cron) # Dont log private
> >> authentication messages!
> >> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>
> >> # The authpriv file has restricted access.
> >> authpriv.* /var/log/secure
> >>
> >>
> >> since the queue only applied to the next action with this config,
> >> everything below this is operating from the main queue again as if
> >> there was no action queue configuration
> >>
> >>
> >>
> >>
> >> # Log all the mail messages in one place.
> >> mail.* -/var/log/maillog
> >>
> >> # Log cron stuff
> >> cron.* /var/log/cron
> >>
> >> # Everybody gets emergency messages
> >> *.emerg :omusrmsg:*
> >>
> >> # Save news errors of level crit and higher in a special file.
> >> uucp,news.crit /var/log/spooler
> >>
> >> # Save boot messages also to boot.log
> >> # local7.* /var/syslog/boot.log
> >>
> >>
> >>
> >> ereregex is a fairly expensive filter to apply, it's much better to
> >> figure out a non-regex approach to filtering these. Can you post
> >> some examples of what you are trying to filter? mmnormalize to
> >> parse the logs and then make decisions on the parsed results id probably much faster.
> >>
> >>
> >> /etc/rsyslog.d/security-config-omsagent.conf
> >> # [Firewall Log Filtering] #
> >> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
> >> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
> >> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
> >> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
> >> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
> >> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
> >>
> >>
> >> check the messages to see where CEF: and ASA- are in the message,
> >> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> >> instead of 'contains'?, again mmnormalize may be much faster
> >>
> >> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
> >> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
> >> then @127.0.0.1:25224 & stop
> >>
> >>
> >> combining multiple filters into one action, or having the filters
> >> call a ruleset can be far more efficient than all of them writing things out independently.
> >>
> >> the if..then filter structure lets you easily combine filters
> >>
> >> local0.info @127.0.0.1:25224
> >> & stop
> >> local1.info @127.0.0.1:25224
> >> & stop
> >> local2.info @127.0.0.1:25224
> >> & stop
> >> local3.info @127.0.0.1:25224
> >> & stop
> >> local4.info @127.0.0.1:25224
> >> & stop
> >> local5.info @127.0.0.1:25224
> >> & stop
> >> local6.info @127.0.0.1:25224
> >> & stop
> >> local7.info @127.0.0.1:25224
> >> & stop
> >> auth.* @127.0.0.1:25224
> >> & stop
> >> authpriv.* @127.0.0.1:25224
> >> & stop
> >> daemon.info @127.0.0.1:25224
> >> & stop
> >> syslog.* @127.0.0.1:25224
> >> & stop
> >> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
> >> user.* @127.0.0.1:25224
> >> & stop
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> >> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
> >> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
> >> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
> >> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> >> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
> >> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
> >> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
> >> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
> >> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
> >> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
> >> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> >> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
> >> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
> >> What's up with rsyslog? Follow
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
> >> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
> >> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
> >> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
> >> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
> >> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
> >> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&amp;reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&amp;reserved=0
What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
________________________________________
This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.

Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&amp;reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&amp;reserved=0
What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
> Udp:
> 5820820 packets received
> 1504 packets to unknown port received.
> 798900 packet receive errors
> 3338814 packets sent
> 798900 receive buffer errors
> 0 send buffer errors
>
> I have doubled the values in net.ipv4.udp_mem.
>
> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).

why do you need a queue for writing to /var/log/secure? omfile is fast enough
that the locking needed for a queue (let alone multiple threads writing to the
same file) will only slow things down)

> The ereregex filters are set to remove information from being forwarded to
> Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant
> for monitoring purpose. They mostly target /16s, /22s and /24s. I could change
> this to (pseudo):

> If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop
>
> Example Checkpoint Log:
> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> dvc=<dvc_ip_addr>

write the log out as RSYSLOG_DebugFormat and look to see if you can do something
other than 'contains'. 'startswith' takes far less processing time.

if you have messages in a known format, using mmnormalize to parse the logs and
then make your tests check the parsed variable instead of having a lot of
'contains' tests that each have to scan the entire message

> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.

if..then tests can be nested so that you don't run every test on every log

> Checking for CEF: is not something I could easily remove. It controls event
> ingestion and separation from other log source types in Microsoft's system.
> I'll remove the ASA section though, it's not necessary for this collector. I
> can probably move the Infoblox setting to a syslog tag by source ip.

the problem isn't checking for CEF/ASA, it's that you are doing so via
'contains'

David Lang


> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer Gerhards via rsyslog
> Sent: Tuesday, November 15, 2022 5:11 AM
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>
> Rainer
>
> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>
>> I haven't needed to do that to handle 300k messages/sec on UDP input
>> (usually I run into bottlenecks in processing the messages long before
>> I have problems accepting them)
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>
>>> let me add: look into setting imudp to realtime priority. Doc:
>>>
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>
>>>> Some additional comments on the config
>>>>
>>>>
>>>>
>>>> These action queue configs probably don't do what you intend them
>>>> to do
>>>>
>>>> the first thing is that they only affect the next action, which is
>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>> write these logs out. That will create a HUGE amount of contention
>>>> for the queue lock and under load you should see it maxing out
>>>> quite quickly
>>>>
>>>> what is it that you are attempting to do here?
>>>>
>>>>
>>>>
>>>> # Performance Tuning #
>>>> $ActionQueueWorkerThreads 2000
>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>> 600000
>>>>
>>>> #### RULES ####
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>> authentication messages!
>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>>
>>>> since the queue only applied to the next action with this config,
>>>> everything below this is operating from the main queue again as if
>>>> there was no action queue configuration
>>>>
>>>>
>>>>
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg :omusrmsg:*
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> # local7.* /var/syslog/boot.log
>>>>
>>>>
>>>>
>>>> ereregex is a fairly expensive filter to apply, it's much better to
>>>> figure out a non-regex approach to filtering these. Can you post
>>>> some examples of what you are trying to filter? mmnormalize to
>>>> parse the logs and then make decisions on the parsed results id probably much faster.
>>>>
>>>>
>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>> # [Firewall Log Filtering] #
>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
>>>> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
>>>> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>
>>>>
>>>> check the messages to see where CEF: and ASA- are in the message,
>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>
>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>> then @127.0.0.1:25224 & stop
>>>>
>>>>
>>>> combining multiple filters into one action, or having the filters
>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>
>>>> the if..then filter structure lets you easily combine filters
>>>>
>>>> local0.info @127.0.0.1:25224
>>>> & stop
>>>> local1.info @127.0.0.1:25224
>>>> & stop
>>>> local2.info @127.0.0.1:25224
>>>> & stop
>>>> local3.info @127.0.0.1:25224
>>>> & stop
>>>> local4.info @127.0.0.1:25224
>>>> & stop
>>>> local5.info @127.0.0.1:25224
>>>> & stop
>>>> local6.info @127.0.0.1:25224
>>>> & stop
>>>> local7.info @127.0.0.1:25224
>>>> & stop
>>>> auth.* @127.0.0.1:25224
>>>> & stop
>>>> authpriv.* @127.0.0.1:25224
>>>> & stop
>>>> daemon.info @127.0.0.1:25224
>>>> & stop
>>>> syslog.* @127.0.0.1:25224
>>>> & stop
>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>> user.* @127.0.0.1:25224
>>>> & stop
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>> What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQynhVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6IonLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
> What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OYkHGY%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> ________________________________________
> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>
> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> ________________________________________
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
what does the pstats output look like when it's dropping messages? (give a
couple cycles please)

did you try to eliminate the action queue for /var/log/secure?

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 13:01:02 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Building on this -
>
> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
> In:imdup spikes to ~10%
> in_syslog.rb spikes to 90-100% usage
> rs:main Q:Reg spikes to 25% usage.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Redbourne,Michael via rsyslog
> Sent: Tuesday, November 15, 2022 8:42 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
> Udp:
> 5820820 packets received
> 1504 packets to unknown port received.
> 798900 packet receive errors
> 3338814 packets sent
> 798900 receive buffer errors
> 0 send buffer errors
>
> I have doubled the values in net.ipv4.udp_mem.
>
> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>
> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
> If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop
>
> Example Checkpoint Log:
> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> dvc=<dvc_ip_addr>
>
> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>
> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer Gerhards via rsyslog
> Sent: Tuesday, November 15, 2022 5:11 AM
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>
> Rainer
>
> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>
>> I haven't needed to do that to handle 300k messages/sec on UDP input
>> (usually I run into bottlenecks in processing the messages long before
>> I have problems accepting them)
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>
>>> let me add: look into setting imudp to realtime priority. Doc:
>>>
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>
>>>> Some additional comments on the config
>>>>
>>>>
>>>>
>>>> These action queue configs probably don't do what you intend them
>>>> to do
>>>>
>>>> the first thing is that they only affect the next action, which is
>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>> write these logs out. That will create a HUGE amount of contention
>>>> for the queue lock and under load you should see it maxing out
>>>> quite quickly
>>>>
>>>> what is it that you are attempting to do here?
>>>>
>>>>
>>>>
>>>> # Performance Tuning #
>>>> $ActionQueueWorkerThreads 2000
>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>> 600000
>>>>
>>>> #### RULES ####
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>> authentication messages!
>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>>
>>>> since the queue only applied to the next action with this config,
>>>> everything below this is operating from the main queue again as if
>>>> there was no action queue configuration
>>>>
>>>>
>>>>
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg :omusrmsg:*
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> # local7.* /var/syslog/boot.log
>>>>
>>>>
>>>>
>>>> ereregex is a fairly expensive filter to apply, it's much better to
>>>> figure out a non-regex approach to filtering these. Can you post
>>>> some examples of what you are trying to filter? mmnormalize to
>>>> parse the logs and then make decisions on the parsed results id probably much faster.
>>>>
>>>>
>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>> # [Firewall Log Filtering] #
>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
>>>> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
>>>> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>
>>>>
>>>> check the messages to see where CEF: and ASA- are in the message,
>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>
>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>> then @127.0.0.1:25224 & stop
>>>>
>>>>
>>>> combining multiple filters into one action, or having the filters
>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>
>>>> the if..then filter structure lets you easily combine filters
>>>>
>>>> local0.info @127.0.0.1:25224
>>>> & stop
>>>> local1.info @127.0.0.1:25224
>>>> & stop
>>>> local2.info @127.0.0.1:25224
>>>> & stop
>>>> local3.info @127.0.0.1:25224
>>>> & stop
>>>> local4.info @127.0.0.1:25224
>>>> & stop
>>>> local5.info @127.0.0.1:25224
>>>> & stop
>>>> local6.info @127.0.0.1:25224
>>>> & stop
>>>> local7.info @127.0.0.1:25224
>>>> & stop
>>>> auth.* @127.0.0.1:25224
>>>> & stop
>>>> authpriv.* @127.0.0.1:25224
>>>> & stop
>>>> daemon.info @127.0.0.1:25224
>>>> & stop
>>>> syslog.* @127.0.0.1:25224
>>>> & stop
>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>> user.* @127.0.0.1:25224
>>>> & stop
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>> What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&amp;reserved=0
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&amp;reserved=0
> What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> ________________________________________
> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>
> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> ________________________________________
> _______________________________________________
> rsyslog mailing list
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&amp;reserved=0
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&amp;reserved=0
> What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
I'm still not understanding what you mean by pstats - it's not a package or command available to me. It's apart of Unix from what I can tell. I've placed below the unparsed information form /proc/net/netstat and /proc/net/udp

/proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRecvSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0

/proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0

The ActionQueue for /var/log/secure has been commented out from the configuration.

Dropwatch (Cycle 1):
6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
7 drops at skb_release_data+12b (0xffffffff9a7f8cab)

Dropwatch (Cycle 2):
12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
11 drops at skb_release_data+12b (0xffffffff937f8cab)
1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
14 drops at skb_release_data+12b (0xffffffff937f8cab)
7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
6 drops at skb_release_data+12b (0xffffffff937f8cab)
19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
11 drops at skb_release_data+12b (0xffffffff937f8cab)
1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
10 drops at skb_release_data+12b (0xffffffff937f8cab)
6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
5 drops at skb_release_data+12b (0xffffffff937f8cab)
161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
1 drops at nf_hook_slow+9d (0xffffffff938ae98d)

Top -H
1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp

/etc/sysctl.conf
net.core.rmem_default = 33554432
net.core.rmem_max = 268435456
net.core.wmem_default = 33554432
net.core.wmem_max = 268435456
net.ipv4.tcp_mem = 190611 254150 381222
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.udp_mem = 762450 1524900 3049800
net.ipv4.udp_rmem_min = 33554432
net.ipv4.udp_wmem_min = 33554432


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, November 15, 2022 2:30 PM
To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

what does the pstats output look like when it's dropping messages? (give a couple cycles please)

did you try to eliminate the action queue for /var/log/secure?

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 13:01:02 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
> <david@lang.hm>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Building on this -
>
> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
> In:imdup spikes to ~10%
> in_syslog.rb spikes to 90-100% usage
> rs:main Q:Reg spikes to 25% usage.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> Redbourne,Michael via rsyslog
> Sent: Tuesday, November 15, 2022 8:42 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> <david@lang.hm>
> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
> Udp:
> 5820820 packets received
> 1504 packets to unknown port received.
> 798900 packet receive errors
> 3338814 packets sent
> 798900 receive buffer errors
> 0 send buffer errors
>
> I have doubled the values in net.ipv4.udp_mem.
>
> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>
> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
> "<subnet>" stop
>
> Example Checkpoint Log:
> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
> Vendor-specific SQL Injection|Very-High| eventId=882492844392
> msg=Application Intelligence mrt=1599552618944 in=-2147483648
> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
> deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX
> src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX
> sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
> dvc=<dvc_ip_addr>
>
> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>
> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer
> Gerhards via rsyslog
> Sent: Tuesday, November 15, 2022 5:11 AM
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
> <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>
> Rainer
>
> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>
>> I haven't needed to do that to handle 300k messages/sec on UDP input
>> (usually I run into bottlenecks in processing the messages long
>> before I have problems accepting them)
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>
>>> let me add: look into setting imudp to realtime priority. Doc:
>>>
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>
>>>> Some additional comments on the config
>>>>
>>>>
>>>>
>>>> These action queue configs probably don't do what you intend them
>>>> to do
>>>>
>>>> the first thing is that they only affect the next action, which is
>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>> write these logs out. That will create a HUGE amount of contention
>>>> for the queue lock and under load you should see it maxing out
>>>> quite quickly
>>>>
>>>> what is it that you are attempting to do here?
>>>>
>>>>
>>>>
>>>> # Performance Tuning #
>>>> $ActionQueueWorkerThreads 2000
>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>> 600000
>>>>
>>>> #### RULES ####
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>> authentication messages!
>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>>
>>>> since the queue only applied to the next action with this config,
>>>> everything below this is operating from the main queue again as if
>>>> there was no action queue configuration
>>>>
>>>>
>>>>
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg :omusrmsg:*
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> # local7.* /var/syslog/boot.log
>>>>
>>>>
>>>>
>>>> ereregex is a fairly expensive filter to apply, it's much better to
>>>> figure out a non-regex approach to filtering these. Can you post
>>>> some examples of what you are trying to filter? mmnormalize to
>>>> parse the logs and then make decisions on the parsed results id probably much faster.
>>>>
>>>>
>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>> # [Firewall Log Filtering] #
>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
>>>> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
>>>> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>
>>>>
>>>> check the messages to see where CEF: and ASA- are in the message,
>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>
>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>> then @127.0.0.1:25224 & stop
>>>>
>>>>
>>>> combining multiple filters into one action, or having the filters
>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>
>>>> the if..then filter structure lets you easily combine filters
>>>>
>>>> local0.info @127.0.0.1:25224
>>>> & stop
>>>> local1.info @127.0.0.1:25224
>>>> & stop
>>>> local2.info @127.0.0.1:25224
>>>> & stop
>>>> local3.info @127.0.0.1:25224
>>>> & stop
>>>> local4.info @127.0.0.1:25224
>>>> & stop
>>>> local5.info @127.0.0.1:25224
>>>> & stop
>>>> local6.info @127.0.0.1:25224
>>>> & stop
>>>> local7.info @127.0.0.1:25224
>>>> & stop
>>>> auth.* @127.0.0.1:25224
>>>> & stop
>>>> authpriv.* @127.0.0.1:25224
>>>> & stop
>>>> daemon.info @127.0.0.1:25224
>>>> & stop
>>>> syslog.* @127.0.0.1:25224
>>>> & stop
>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>> user.* @127.0.0.1:25224
>>>> & stop
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>> What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
> _______________________________________________
> rsyslog mailing list
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichae
> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
> BRRguRnpir9RsV8%3D&amp;reserved=0
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbo
> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproof
> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;rese
> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> ________________________________________
> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>
> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> ________________________________________
> _______________________________________________
> rsyslog mailing list
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichae
> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
> BRRguRnpir9RsV8%3D&amp;reserved=0
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbo
> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproof
> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;rese
> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
you have the impstats module loaded in your config and writing stats out, please
post the output of this.

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 19:38:27 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I'm still not understanding what you mean by pstats - it's not a package or command available to me. It's apart of Unix from what I can tell. I've placed below the unparsed information form /proc/net/netstat and /proc/net/udp
>
> /proc/net/netstat
> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure TCPSackShifted T
CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRec
vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>
> /proc/net/udp
> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>
> The ActionQueue for /var/log/secure has been commented out from the configuration.
>
> Dropwatch (Cycle 1):
> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>
> Dropwatch (Cycle 2):
> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>
> Top -H
> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>
> /etc/sysctl.conf
> net.core.rmem_default = 33554432
> net.core.rmem_max = 268435456
> net.core.wmem_default = 33554432
> net.core.wmem_max = 268435456
> net.ipv4.tcp_mem = 190611 254150 381222
> net.ipv4.tcp_rmem = 4096 131072 6291456
> net.ipv4.tcp_wmem = 4096 16384 4194304
> net.ipv4.udp_mem = 762450 1524900 3049800
> net.ipv4.udp_rmem_min = 33554432
> net.ipv4.udp_wmem_min = 33554432
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 2:30 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> what does the pstats output look like when it's dropping messages? (give a couple cycles please)
>
> did you try to eliminate the action queue for /var/log/secure?
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>> <david@lang.hm>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> Building on this -
>>
>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>> In:imdup spikes to ~10%
>> in_syslog.rb spikes to 90-100% usage
>> rs:main Q:Reg spikes to 25% usage.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Redbourne,Michael via rsyslog
>> Sent: Tuesday, November 15, 2022 8:42 AM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>> <david@lang.hm>
>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>> Udp:
>> 5820820 packets received
>> 1504 packets to unknown port received.
>> 798900 packet receive errors
>> 3338814 packets sent
>> 798900 receive buffer errors
>> 0 send buffer errors
>>
>> I have doubled the values in net.ipv4.udp_mem.
>>
>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>
>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>> "<subnet>" stop
>>
>> Example Checkpoint Log:
>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
>> deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX
>> src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX
>> sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>> dvc=<dvc_ip_addr>
>>
>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>
>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer
>> Gerhards via rsyslog
>> Sent: Tuesday, November 15, 2022 5:11 AM
>> To: David Lang <david@lang.hm>
>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>
>> Rainer
>>
>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>
>>> I haven't needed to do that to handle 300k messages/sec on UDP input
>>> (usually I run into bottlenecks in processing the messages long
>>> before I have problems accepting them)
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>
>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata
>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>
>>>> Rainer
>>>>
>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>
>>>>> Some additional comments on the config
>>>>>
>>>>>
>>>>>
>>>>> These action queue configs probably don't do what you intend them
>>>>> to do
>>>>>
>>>>> the first thing is that they only affect the next action, which is
>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>> write these logs out. That will create a HUGE amount of contention
>>>>> for the queue lock and under load you should see it maxing out
>>>>> quite quickly
>>>>>
>>>>> what is it that you are attempting to do here?
>>>>>
>>>>>
>>>>>
>>>>> # Performance Tuning #
>>>>> $ActionQueueWorkerThreads 2000
>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>> 600000
>>>>>
>>>>> #### RULES ####
>>>>> # Log all kernel messages to the console.
>>>>> # Logging much else clutters up the screen.
>>>>> #kern.* /dev/console
>>>>>
>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>> authentication messages!
>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>
>>>>> # The authpriv file has restricted access.
>>>>> authpriv.* /var/log/secure
>>>>>
>>>>>
>>>>> since the queue only applied to the next action with this config,
>>>>> everything below this is operating from the main queue again as if
>>>>> there was no action queue configuration
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> # Log all the mail messages in one place.
>>>>> mail.* -/var/log/maillog
>>>>>
>>>>> # Log cron stuff
>>>>> cron.* /var/log/cron
>>>>>
>>>>> # Everybody gets emergency messages
>>>>> *.emerg :omusrmsg:*
>>>>>
>>>>> # Save news errors of level crit and higher in a special file.
>>>>> uucp,news.crit /var/log/spooler
>>>>>
>>>>> # Save boot messages also to boot.log
>>>>> # local7.* /var/syslog/boot.log
>>>>>
>>>>>
>>>>>
>>>>> ereregex is a fairly expensive filter to apply, it's much better to
>>>>> figure out a non-regex approach to filtering these. Can you post
>>>>> some examples of what you are trying to filter? mmnormalize to
>>>>> parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>
>>>>>
>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>> # [Firewall Log Filtering] #
>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
>>>>> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
>>>>> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>
>>>>>
>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>
>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>> then @127.0.0.1:25224 & stop
>>>>>
>>>>>
>>>>> combining multiple filters into one action, or having the filters
>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>
>>>>> the if..then filter structure lets you easily combine filters
>>>>>
>>>>> local0.info @127.0.0.1:25224
>>>>> & stop
>>>>> local1.info @127.0.0.1:25224
>>>>> & stop
>>>>> local2.info @127.0.0.1:25224
>>>>> & stop
>>>>> local3.info @127.0.0.1:25224
>>>>> & stop
>>>>> local4.info @127.0.0.1:25224
>>>>> & stop
>>>>> local5.info @127.0.0.1:25224
>>>>> & stop
>>>>> local6.info @127.0.0.1:25224
>>>>> & stop
>>>>> local7.info @127.0.0.1:25224
>>>>> & stop
>>>>> auth.* @127.0.0.1:25224
>>>>> & stop
>>>>> authpriv.* @127.0.0.1:25224
>>>>> & stop
>>>>> daemon.info @127.0.0.1:25224
>>>>> & stop
>>>>> syslog.* @127.0.0.1:25224
>>>>> & stop
>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>> user.* @127.0.0.1:25224
>>>>> & stop
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7C
>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQyn
>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael
>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6Io
>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>> What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulle
>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichae
>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
>> BRRguRnpir9RsV8%3D&amp;reserved=0
>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbo
>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproof
>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;rese
>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> ________________________________________
>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>
>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>> ________________________________________
>> _______________________________________________
>> rsyslog mailing list
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmichae
>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
>> BRRguRnpir9RsV8%3D&amp;reserved=0
>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redbo
>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproof
>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;rese
>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
Tue Nov 15 00:22:18 2022: global: origin=dynstats
Tue Nov 15 00:22:18 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Nov 15 00:22:18 2022: action 0: origin=core.action processed=1628 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 1: origin=core.action processed=66 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 2: origin=core.action processed=37449 failed=37448 suspended=1 suspended.duration=30 resumed=0
Tue Nov 15 00:22:18 2022: action 3: origin=core.action processed=4582 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 4: origin=core.action processed=5039 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 8: origin=core.action processed=862 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 10: origin=core.action processed=397 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 14: origin=core.action processed=766 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 15: origin=core.action processed=11058 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 17: origin=core.action processed=1099 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 18: origin=core.action processed=12 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=24227
Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=0
Tue Nov 15 00:22:18 2022: imtcp(514): origin=imtcp submitted=68069
Tue Nov 15 00:22:18 2022: resource-usage: origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469 majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702
Tue Nov 15 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0 discarded.full=0 discarded.nf=0 maxqsize=7033
Tue Nov 15 00:22:18 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0 msgs.received=24227


Tue Nov 15 08:41:43 2022: global: origin=dynstats
Tue Nov 15 08:41:43 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Nov 15 08:41:43 2022: action 0: origin=core.action processed=1728 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 1: origin=core.action processed=47 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 2: origin=core.action processed=35778 failed=35777 suspended=1 suspended.duration=30 resumed=0
Tue Nov 15 08:41:43 2022: action 3: origin=core.action processed=9249 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 4: origin=core.action processed=2296 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 8: origin=core.action processed=1009 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 10: origin=core.action processed=558 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 11: origin=core.action processed=40 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 14: origin=core.action processed=720 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 15: origin=core.action processed=16593 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 17: origin=core.action processed=359 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 18: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=31257
Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=0
Tue Nov 15 08:41:43 2022: imtcp(514): origin=imtcp submitted=60875
Tue Nov 15 08:41:43 2022: resource-usage: origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884 majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479
Tue Nov 15 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
Tue Nov 15 08:41:43 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0 msgs.received=31257


Tue Nov 15 08:42:13 2022: global: origin=dynstats
Tue Nov 15 08:42:13 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Nov 15 08:42:13 2022: action 0: origin=core.action processed=2181 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 1: origin=core.action processed=57 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 2: origin=core.action processed=20793 failed=137 suspended=0 suspended.duration=30 resumed=0
Tue Nov 15 08:42:13 2022: action 3: origin=core.action processed=7599 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 4: origin=core.action processed=5540 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 8: origin=core.action processed=1120 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 10: origin=core.action processed=157807 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 12: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 13: origin=core.action processed=3 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 14: origin=core.action processed=293 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 15: origin=core.action processed=20243 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 17: origin=core.action processed=866 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 18: origin=core.action processed=10 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=194836
Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=0
Tue Nov 15 08:42:13 2022: imtcp(514): origin=imtcp submitted=34848
Tue Nov 15 08:42:13 2022: resource-usage: origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196 majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993
Tue Nov 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
Tue Nov 15 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595 called.recvmsg=0 msgs.received=194836


Tue Nov 15 10:15:04 2022: global: origin=dynstats
Tue Nov 15 10:15:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Nov 15 10:15:04 2022: action 0: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 1: origin=core.action processed=151 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 2: origin=core.action processed=52035 failed=52034 suspended=1 suspended.duration=60 resumed=0
Tue Nov 15 10:15:04 2022: action 3: origin=core.action processed=19442 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 4: origin=core.action processed=4480 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 8: origin=core.action processed=1125 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 10: origin=core.action processed=415 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 11: origin=core.action processed=144 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 14: origin=core.action processed=745 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 15: origin=core.action processed=23626 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 17: origin=core.action processed=682 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 18: origin=core.action processed=45 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=51446
Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=0
Tue Nov 15 10:15:04 2022: imtcp(514): origin=imtcp submitted=81918
Tue Nov 15 10:15:04 2022: resource-usage: origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979 majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668
Tue Nov 15 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0 discarded.full=0 discarded.nf=0 maxqsize=6830
Tue Nov 15 10:15:04 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0 msgs.received=51446
-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, November 15, 2022 3:43 PM
To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

you have the impstats module loaded in your config and writing stats out, please post the output of this.

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 19:38:27 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I'm still not understanding what you mean by pstats - it's not a
> package or command available to me. It's apart of Unix from what I can
> tell. I've placed below the unparsed information form
> /proc/net/netstat and /proc/net/udp
>
> /proc/net/netstat
> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent
> TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData
> TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger
> TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono
> TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo
> TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure
> TCPSackShifted T
CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRec vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts
> OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets
> InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts
> InECT0Pkts InCEPkts ReasmOverlaps
> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>
> /proc/net/udp
> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>
> The ActionQueue for /var/log/secure has been commented out from the configuration.
>
> Dropwatch (Cycle 1):
> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>
> Dropwatch (Cycle 2):
> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>
> Top -H
> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>
> /etc/sysctl.conf
> net.core.rmem_default = 33554432
> net.core.rmem_max = 268435456
> net.core.wmem_default = 33554432
> net.core.wmem_max = 268435456
> net.ipv4.tcp_mem = 190611 254150 381222
> net.ipv4.tcp_rmem = 4096 131072 6291456
> net.ipv4.tcp_wmem = 4096 16384 4194304
> net.ipv4.udp_mem = 762450 1524900 3049800
> net.ipv4.udp_rmem_min = 33554432
> net.ipv4.udp_wmem_min = 33554432
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 2:30 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> <david@lang.hm>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> what does the pstats output look like when it's dropping messages?
> (give a couple cycles please)
>
> did you try to eliminate the action queue for /var/log/secure?
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>> <david@lang.hm>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> Building on this -
>>
>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>> In:imdup spikes to ~10%
>> in_syslog.rb spikes to 90-100% usage
>> rs:main Q:Reg spikes to 25% usage.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Redbourne,Michael via rsyslog
>> Sent: Tuesday, November 15, 2022 8:42 AM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>> <david@lang.hm>
>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>> Udp:
>> 5820820 packets received
>> 1504 packets to unknown port received.
>> 798900 packet receive errors
>> 3338814 packets sent
>> 798900 receive buffer errors
>> 0 send buffer errors
>>
>> I have doubled the values in net.ipv4.udp_mem.
>>
>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>
>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>> "<subnet>" stop
>>
>> Example Checkpoint Log:
>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
>> deviceSeverity=Very-High rt=1599552617058 deviceDirection=0
>> shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>> dvc=<dvc_ip_addr>
>>
>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>
>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer
>> Gerhards via rsyslog
>> Sent: Tuesday, November 15, 2022 5:11 AM
>> To: David Lang <david@lang.hm>
>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>
>> Rainer
>>
>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>
>>> I haven't needed to do that to handle 300k messages/sec on UDP input
>>> (usually I run into bottlenecks in processing the messages long
>>> before I have problems accepting them)
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>
>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>> w
>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html
>>>> &
>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616
>>>> 2
>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%
>>>> 7
>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>>>> Q
>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>>> a
>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>
>>>> Rainer
>>>>
>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>
>>>>> Some additional comments on the config
>>>>>
>>>>>
>>>>>
>>>>> These action queue configs probably don't do what you intend them
>>>>> to do
>>>>>
>>>>> the first thing is that they only affect the next action, which is
>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>> write these logs out. That will create a HUGE amount of contention
>>>>> for the queue lock and under load you should see it maxing out
>>>>> quite quickly
>>>>>
>>>>> what is it that you are attempting to do here?
>>>>>
>>>>>
>>>>>
>>>>> # Performance Tuning #
>>>>> $ActionQueueWorkerThreads 2000
>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>> 600000
>>>>>
>>>>> #### RULES ####
>>>>> # Log all kernel messages to the console.
>>>>> # Logging much else clutters up the screen.
>>>>> #kern.* /dev/console
>>>>>
>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>> authentication messages!
>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>
>>>>> # The authpriv file has restricted access.
>>>>> authpriv.* /var/log/secure
>>>>>
>>>>>
>>>>> since the queue only applied to the next action with this config,
>>>>> everything below this is operating from the main queue again as if
>>>>> there was no action queue configuration
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> # Log all the mail messages in one place.
>>>>> mail.* -/var/log/maillog
>>>>>
>>>>> # Log cron stuff
>>>>> cron.* /var/log/cron
>>>>>
>>>>> # Everybody gets emergency messages
>>>>> *.emerg :omusrmsg:*
>>>>>
>>>>> # Save news errors of level crit and higher in a special file.
>>>>> uucp,news.crit /var/log/spooler
>>>>>
>>>>> # Save boot messages also to boot.log
>>>>> # local7.* /var/syslog/boot.log
>>>>>
>>>>>
>>>>>
>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>
>>>>>
>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>> # [Firewall Log Filtering] #
>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>
>>>>>
>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>
>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>> then @127.0.0.1:25224 & stop
>>>>>
>>>>>
>>>>> combining multiple filters into one action, or having the filters
>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>
>>>>> the if..then filter structure lets you easily combine filters
>>>>>
>>>>> local0.info @127.0.0.1:25224
>>>>> & stop
>>>>> local1.info @127.0.0.1:25224
>>>>> & stop
>>>>> local2.info @127.0.0.1:25224
>>>>> & stop
>>>>> local3.info @127.0.0.1:25224
>>>>> & stop
>>>>> local4.info @127.0.0.1:25224
>>>>> & stop
>>>>> local5.info @127.0.0.1:25224
>>>>> & stop
>>>>> local6.info @127.0.0.1:25224
>>>>> & stop
>>>>> local7.info @127.0.0.1:25224
>>>>> & stop
>>>>> auth.* @127.0.0.1:25224
>>>>> & stop
>>>>> authpriv.* @127.0.0.1:25224
>>>>> & stop
>>>>> daemon.info @127.0.0.1:25224
>>>>> & stop
>>>>> syslog.* @127.0.0.1:25224
>>>>> & stop
>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>> user.* @127.0.0.1:25224
>>>>> & stop
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> l
>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7
>>>>> C
>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6
>>>>> e
>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031
>>>>> 5
>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
>>>>> B
>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQy
>>>>> n
>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fw
>>>>> w
>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichae
>>>>> l
>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%
>>>>> 7
>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CU
>>>>> n
>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik
>>>>> 1
>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6I
>>>>> o
>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>> What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> t
>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bull
>>>>> e
>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411b
>>>>> b
>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb
>>>>> 3
>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
>>>>> 3
>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3O
>>>>> Y
>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>> t
>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>> e
>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>> 9
>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>> n
>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>> J
>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>> 2
>> BRRguRnpir9RsV8%3D&amp;reserved=0
>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>> o
>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>> 8
>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>> p
>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>> n
>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>> N
>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>> t
>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>> f
>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>> 2
>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>> C
>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>> %
>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>> e
>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> ________________________________________
>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>
>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>> ________________________________________
>> _______________________________________________
>> rsyslog mailing list
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>> t
>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>> e
>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>> 9
>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>> n
>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>> J
>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>> 2
>> BRRguRnpir9RsV8%3D&amp;reserved=0
>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>> o
>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>> 8
>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>> p
>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>> n
>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>> N
>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>> t
>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>> f
>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>> 2
>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>> C
>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>> %
>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>> e
>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
the fact that action2 is failingh frequently needs to be addressed, that will
cause you all sorts of grief

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 20:01:52 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Tue Nov 15 00:22:18 2022: global: origin=dynstats
> Tue Nov 15 00:22:18 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 00:22:18 2022: action 0: origin=core.action processed=1628 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 1: origin=core.action processed=66 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 2: origin=core.action processed=37449 failed=37448 suspended=1 suspended.duration=30 resumed=0
> Tue Nov 15 00:22:18 2022: action 3: origin=core.action processed=4582 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 4: origin=core.action processed=5039 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 8: origin=core.action processed=862 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 10: origin=core.action processed=397 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 14: origin=core.action processed=766 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 15: origin=core.action processed=11058 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 17: origin=core.action processed=1099 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 18: origin=core.action processed=12 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=24227
> Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 00:22:18 2022: imtcp(514): origin=imtcp submitted=68069
> Tue Nov 15 00:22:18 2022: resource-usage: origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469 majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702
> Tue Nov 15 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0 discarded.full=0 discarded.nf=0 maxqsize=7033
> Tue Nov 15 00:22:18 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0 msgs.received=24227
>
>
> Tue Nov 15 08:41:43 2022: global: origin=dynstats
> Tue Nov 15 08:41:43 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 08:41:43 2022: action 0: origin=core.action processed=1728 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 1: origin=core.action processed=47 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 2: origin=core.action processed=35778 failed=35777 suspended=1 suspended.duration=30 resumed=0
> Tue Nov 15 08:41:43 2022: action 3: origin=core.action processed=9249 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 4: origin=core.action processed=2296 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 8: origin=core.action processed=1009 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 10: origin=core.action processed=558 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 11: origin=core.action processed=40 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 14: origin=core.action processed=720 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 15: origin=core.action processed=16593 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 17: origin=core.action processed=359 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 18: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=31257
> Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 08:41:43 2022: imtcp(514): origin=imtcp submitted=60875
> Tue Nov 15 08:41:43 2022: resource-usage: origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884 majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479
> Tue Nov 15 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
> Tue Nov 15 08:41:43 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0 msgs.received=31257
>
>
> Tue Nov 15 08:42:13 2022: global: origin=dynstats
> Tue Nov 15 08:42:13 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 08:42:13 2022: action 0: origin=core.action processed=2181 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 1: origin=core.action processed=57 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 2: origin=core.action processed=20793 failed=137 suspended=0 suspended.duration=30 resumed=0
> Tue Nov 15 08:42:13 2022: action 3: origin=core.action processed=7599 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 4: origin=core.action processed=5540 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 8: origin=core.action processed=1120 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 10: origin=core.action processed=157807 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 12: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 13: origin=core.action processed=3 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 14: origin=core.action processed=293 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 15: origin=core.action processed=20243 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 17: origin=core.action processed=866 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 18: origin=core.action processed=10 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=194836
> Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 08:42:13 2022: imtcp(514): origin=imtcp submitted=34848
> Tue Nov 15 08:42:13 2022: resource-usage: origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196 majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993
> Tue Nov 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
> Tue Nov 15 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595 called.recvmsg=0 msgs.received=194836
>
>
> Tue Nov 15 10:15:04 2022: global: origin=dynstats
> Tue Nov 15 10:15:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 10:15:04 2022: action 0: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 1: origin=core.action processed=151 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 2: origin=core.action processed=52035 failed=52034 suspended=1 suspended.duration=60 resumed=0
> Tue Nov 15 10:15:04 2022: action 3: origin=core.action processed=19442 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 4: origin=core.action processed=4480 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 8: origin=core.action processed=1125 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 10: origin=core.action processed=415 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 11: origin=core.action processed=144 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 14: origin=core.action processed=745 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 15: origin=core.action processed=23626 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 17: origin=core.action processed=682 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 18: origin=core.action processed=45 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=51446
> Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 10:15:04 2022: imtcp(514): origin=imtcp submitted=81918
> Tue Nov 15 10:15:04 2022: resource-usage: origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979 majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668
> Tue Nov 15 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0 discarded.full=0 discarded.nf=0 maxqsize=6830
> Tue Nov 15 10:15:04 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0 msgs.received=51446
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 3:43 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> I'm still not understanding what you mean by pstats - it's not a
>> package or command available to me. It's apart of Unix from what I can
>> tell. I've placed below the unparsed information form
>> /proc/net/netstat and /proc/net/udp
>>
>> /proc/net/netstat
>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent
>> TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData
>> TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger
>> TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono
>> TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo
>> TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure
>> TCPSackShifted T
> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRe
c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts
>> OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets
>> InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts
>> InECT0Pkts InCEPkts ReasmOverlaps
>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>
>> /proc/net/udp
>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>
>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>
>> Dropwatch (Cycle 1):
>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>
>> Dropwatch (Cycle 2):
>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>
>> Top -H
>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>
>> /etc/sysctl.conf
>> net.core.rmem_default = 33554432
>> net.core.rmem_max = 268435456
>> net.core.wmem_default = 33554432
>> net.core.wmem_max = 268435456
>> net.ipv4.tcp_mem = 190611 254150 381222
>> net.ipv4.tcp_rmem = 4096 131072 6291456
>> net.ipv4.tcp_wmem = 4096 16384 4194304
>> net.ipv4.udp_mem = 762450 1524900 3049800
>> net.ipv4.udp_rmem_min = 33554432
>> net.ipv4.udp_wmem_min = 33554432
>>
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 2:30 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>> <david@lang.hm>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> what does the pstats output look like when it's dropping messages?
>> (give a couple cycles please)
>>
>> did you try to eliminate the action queue for /var/log/secure?
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>> <david@lang.hm>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Building on this -
>>>
>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>> In:imdup spikes to ~10%
>>> in_syslog.rb spikes to 90-100% usage
>>> rs:main Q:Reg spikes to 25% usage.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Redbourne,Michael via rsyslog
>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>> <david@lang.hm>
>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>> Udp:
>>> 5820820 packets received
>>> 1504 packets to unknown port received.
>>> 798900 packet receive errors
>>> 3338814 packets sent
>>> 798900 receive buffer errors
>>> 0 send buffer errors
>>>
>>> I have doubled the values in net.ipv4.udp_mem.
>>>
>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>
>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>> "<subnet>" stop
>>>
>>> Example Checkpoint Log:
>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
>>> deviceSeverity=Very-High rt=1599552617058 deviceDirection=0
>>> shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>> dvc=<dvc_ip_addr>
>>>
>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>
>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer
>>> Gerhards via rsyslog
>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>> To: David Lang <david@lang.hm>
>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>
>>>> I haven't needed to do that to handle 300k messages/sec on UDP input
>>>> (usually I run into bottlenecks in processing the messages long
>>>> before I have problems accepting them)
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>
>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>> w
>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html
>>>>> &
>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616
>>>>> 2
>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%
>>>>> 7
>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>>>>> Q
>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>>>> a
>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>
>>>>> Rainer
>>>>>
>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>
>>>>>> Some additional comments on the config
>>>>>>
>>>>>>
>>>>>>
>>>>>> These action queue configs probably don't do what you intend them
>>>>>> to do
>>>>>>
>>>>>> the first thing is that they only affect the next action, which is
>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>> write these logs out. That will create a HUGE amount of contention
>>>>>> for the queue lock and under load you should see it maxing out
>>>>>> quite quickly
>>>>>>
>>>>>> what is it that you are attempting to do here?
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Performance Tuning #
>>>>>> $ActionQueueWorkerThreads 2000
>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>> 600000
>>>>>>
>>>>>> #### RULES ####
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> #kern.* /dev/console
>>>>>>
>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>> authentication messages!
>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>
>>>>>> # The authpriv file has restricted access.
>>>>>> authpriv.* /var/log/secure
>>>>>>
>>>>>>
>>>>>> since the queue only applied to the next action with this config,
>>>>>> everything below this is operating from the main queue again as if
>>>>>> there was no action queue configuration
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.* -/var/log/maillog
>>>>>>
>>>>>> # Log cron stuff
>>>>>> cron.* /var/log/cron
>>>>>>
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg :omusrmsg:*
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>> uucp,news.crit /var/log/spooler
>>>>>>
>>>>>> # Save boot messages also to boot.log
>>>>>> # local7.* /var/syslog/boot.log
>>>>>>
>>>>>>
>>>>>>
>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>
>>>>>>
>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>> # [Firewall Log Filtering] #
>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>
>>>>>>
>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>
>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>> then @127.0.0.1:25224 & stop
>>>>>>
>>>>>>
>>>>>> combining multiple filters into one action, or having the filters
>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>
>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>
>>>>>> local0.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local1.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local2.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local3.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local4.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local5.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local6.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local7.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> auth.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> authpriv.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> daemon.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> syslog.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>> user.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> l
>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7
>>>>>> C
>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6
>>>>>> e
>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031
>>>>>> 5
>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
>>>>>> B
>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQy
>>>>>> n
>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fw
>>>>>> w
>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichae
>>>>>> l
>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%
>>>>>> 7
>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CU
>>>>>> n
>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik
>>>>>> 1
>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6I
>>>>>> o
>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>> What's up with rsyslog? Follow
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> t
>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bull
>>>>>> e
>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411b
>>>>>> b
>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb
>>>>>> 3
>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
>>>>>> 3
>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3O
>>>>>> Y
>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> ________________________________________
>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>
>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>> ________________________________________
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
using the new action() syntax, you can name the actions so they aren't just
numbered.

starting rsyslog with -o /path/to/file will generate a config file that is the
combination of all the included files (as rsyslog actually processes the
config), which will make it easier to figure out which action is which.

that could be the tcp forwarding of messages, which can stall if the device you
are forwarding to can't keep up (that is a good place to have an action queue,
but please use the action() syntax so that it's clear what the queue is doing)

David Lang


On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 20:01:52 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Tue Nov 15 00:22:18 2022: global: origin=dynstats
> Tue Nov 15 00:22:18 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 00:22:18 2022: action 0: origin=core.action processed=1628 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 1: origin=core.action processed=66 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 2: origin=core.action processed=37449 failed=37448 suspended=1 suspended.duration=30 resumed=0
> Tue Nov 15 00:22:18 2022: action 3: origin=core.action processed=4582 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 4: origin=core.action processed=5039 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 8: origin=core.action processed=862 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 10: origin=core.action processed=397 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 14: origin=core.action processed=766 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 15: origin=core.action processed=11058 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 17: origin=core.action processed=1099 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 18: origin=core.action processed=12 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=24227
> Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 00:22:18 2022: imtcp(514): origin=imtcp submitted=68069
> Tue Nov 15 00:22:18 2022: resource-usage: origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469 majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702
> Tue Nov 15 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0 discarded.full=0 discarded.nf=0 maxqsize=7033
> Tue Nov 15 00:22:18 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0 msgs.received=24227
>
>
> Tue Nov 15 08:41:43 2022: global: origin=dynstats
> Tue Nov 15 08:41:43 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 08:41:43 2022: action 0: origin=core.action processed=1728 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 1: origin=core.action processed=47 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 2: origin=core.action processed=35778 failed=35777 suspended=1 suspended.duration=30 resumed=0
> Tue Nov 15 08:41:43 2022: action 3: origin=core.action processed=9249 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 4: origin=core.action processed=2296 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 8: origin=core.action processed=1009 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 10: origin=core.action processed=558 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 11: origin=core.action processed=40 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 14: origin=core.action processed=720 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 15: origin=core.action processed=16593 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 17: origin=core.action processed=359 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 18: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=31257
> Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 08:41:43 2022: imtcp(514): origin=imtcp submitted=60875
> Tue Nov 15 08:41:43 2022: resource-usage: origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884 majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479
> Tue Nov 15 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
> Tue Nov 15 08:41:43 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0 msgs.received=31257
>
>
> Tue Nov 15 08:42:13 2022: global: origin=dynstats
> Tue Nov 15 08:42:13 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 08:42:13 2022: action 0: origin=core.action processed=2181 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 1: origin=core.action processed=57 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 2: origin=core.action processed=20793 failed=137 suspended=0 suspended.duration=30 resumed=0
> Tue Nov 15 08:42:13 2022: action 3: origin=core.action processed=7599 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 4: origin=core.action processed=5540 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 8: origin=core.action processed=1120 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 10: origin=core.action processed=157807 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 12: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 13: origin=core.action processed=3 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 14: origin=core.action processed=293 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 15: origin=core.action processed=20243 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 17: origin=core.action processed=866 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 18: origin=core.action processed=10 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=194836
> Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 08:42:13 2022: imtcp(514): origin=imtcp submitted=34848
> Tue Nov 15 08:42:13 2022: resource-usage: origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196 majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993
> Tue Nov 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893
> Tue Nov 15 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595 called.recvmsg=0 msgs.received=194836
>
>
> Tue Nov 15 10:15:04 2022: global: origin=dynstats
> Tue Nov 15 10:15:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Tue Nov 15 10:15:04 2022: action 0: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 1: origin=core.action processed=151 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 2: origin=core.action processed=52035 failed=52034 suspended=1 suspended.duration=60 resumed=0
> Tue Nov 15 10:15:04 2022: action 3: origin=core.action processed=19442 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 4: origin=core.action processed=4480 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 8: origin=core.action processed=1125 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 10: origin=core.action processed=415 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 11: origin=core.action processed=144 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 14: origin=core.action processed=745 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 15: origin=core.action processed=23626 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 17: origin=core.action processed=682 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 18: origin=core.action processed=45 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=51446
> Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=0
> Tue Nov 15 10:15:04 2022: imtcp(514): origin=imtcp submitted=81918
> Tue Nov 15 10:15:04 2022: resource-usage: origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979 majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668
> Tue Nov 15 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0 discarded.full=0 discarded.nf=0 maxqsize=6830
> Tue Nov 15 10:15:04 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0 msgs.received=51446
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 3:43 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> I'm still not understanding what you mean by pstats - it's not a
>> package or command available to me. It's apart of Unix from what I can
>> tell. I've placed below the unparsed information form
>> /proc/net/netstat and /proc/net/udp
>>
>> /proc/net/netstat
>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent
>> TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData
>> TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger
>> TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono
>> TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo
>> TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure
>> TCPSackShifted T
> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRe
c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts
>> OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets
>> InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts
>> InECT0Pkts InCEPkts ReasmOverlaps
>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>
>> /proc/net/udp
>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>
>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>
>> Dropwatch (Cycle 1):
>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>
>> Dropwatch (Cycle 2):
>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>
>> Top -H
>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>
>> /etc/sysctl.conf
>> net.core.rmem_default = 33554432
>> net.core.rmem_max = 268435456
>> net.core.wmem_default = 33554432
>> net.core.wmem_max = 268435456
>> net.ipv4.tcp_mem = 190611 254150 381222
>> net.ipv4.tcp_rmem = 4096 131072 6291456
>> net.ipv4.tcp_wmem = 4096 16384 4194304
>> net.ipv4.udp_mem = 762450 1524900 3049800
>> net.ipv4.udp_rmem_min = 33554432
>> net.ipv4.udp_wmem_min = 33554432
>>
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 2:30 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>> <david@lang.hm>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> what does the pstats output look like when it's dropping messages?
>> (give a couple cycles please)
>>
>> did you try to eliminate the action queue for /var/log/secure?
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>> <david@lang.hm>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Building on this -
>>>
>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>> In:imdup spikes to ~10%
>>> in_syslog.rb spikes to 90-100% usage
>>> rs:main Q:Reg spikes to 25% usage.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Redbourne,Michael via rsyslog
>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>> <david@lang.hm>
>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>> Udp:
>>> 5820820 packets received
>>> 1504 packets to unknown port received.
>>> 798900 packet receive errors
>>> 3338814 packets sent
>>> 798900 receive buffer errors
>>> 0 send buffer errors
>>>
>>> I have doubled the values in net.ipv4.udp_mem.
>>>
>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>
>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>> "<subnet>" stop
>>>
>>> Example Checkpoint Log:
>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
>>> deviceSeverity=Very-High rt=1599552617058 deviceDirection=0
>>> shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>> dvc=<dvc_ip_addr>
>>>
>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>
>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Rainer
>>> Gerhards via rsyslog
>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>> To: David Lang <david@lang.hm>
>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>
>>>> I haven't needed to do that to handle 300k messages/sec on UDP input
>>>> (usually I run into bottlenecks in processing the messages long
>>>> before I have problems accepting them)
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>
>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>> w
>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html
>>>>> &
>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc616
>>>>> 2
>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%
>>>>> 7
>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>>>>> Q
>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>>>> a
>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>
>>>>> Rainer
>>>>>
>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>
>>>>>> Some additional comments on the config
>>>>>>
>>>>>>
>>>>>>
>>>>>> These action queue configs probably don't do what you intend them
>>>>>> to do
>>>>>>
>>>>>> the first thing is that they only affect the next action, which is
>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>> write these logs out. That will create a HUGE amount of contention
>>>>>> for the queue lock and under load you should see it maxing out
>>>>>> quite quickly
>>>>>>
>>>>>> what is it that you are attempting to do here?
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Performance Tuning #
>>>>>> $ActionQueueWorkerThreads 2000
>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>> 600000
>>>>>>
>>>>>> #### RULES ####
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> #kern.* /dev/console
>>>>>>
>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>> authentication messages!
>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>
>>>>>> # The authpriv file has restricted access.
>>>>>> authpriv.* /var/log/secure
>>>>>>
>>>>>>
>>>>>> since the queue only applied to the next action with this config,
>>>>>> everything below this is operating from the main queue again as if
>>>>>> there was no action queue configuration
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.* -/var/log/maillog
>>>>>>
>>>>>> # Log cron stuff
>>>>>> cron.* /var/log/cron
>>>>>>
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg :omusrmsg:*
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>> uucp,news.crit /var/log/spooler
>>>>>>
>>>>>> # Save boot messages also to boot.log
>>>>>> # local7.* /var/syslog/boot.log
>>>>>>
>>>>>>
>>>>>>
>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>
>>>>>>
>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>> # [Firewall Log Filtering] #
>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>
>>>>>>
>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>
>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>> then @127.0.0.1:25224 & stop
>>>>>>
>>>>>>
>>>>>> combining multiple filters into one action, or having the filters
>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>
>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>
>>>>>> local0.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local1.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local2.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local3.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local4.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local5.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local6.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local7.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> auth.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> authpriv.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> daemon.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> syslog.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>> user.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> l
>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7
>>>>>> C
>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6
>>>>>> e
>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031
>>>>>> 5
>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
>>>>>> B
>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQy
>>>>>> n
>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fw
>>>>>> w
>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichae
>>>>>> l
>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%
>>>>>> 7
>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CU
>>>>>> n
>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik
>>>>>> 1
>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6I
>>>>>> o
>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>> What's up with rsyslog? Follow
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> t
>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bull
>>>>>> e
>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411b
>>>>>> b
>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb
>>>>>> 3
>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
>>>>>> 3
>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3O
>>>>>> Y
>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> ________________________________________
>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>
>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>> ________________________________________
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmicha
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknow
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdat
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.redb
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWF
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwi
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletproo
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;res
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.

If I'm still havcing issues after that, I'll respond back.


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, November 15, 2022 4:19 PM
To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

using the new action() syntax, you can name the actions so they aren't just numbered.

starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.

that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)

David Lang


On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 20:01:52 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
> origin=core.action processed=1628 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
> origin=core.action processed=66 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
> origin=core.action processed=37449 failed=37448 suspended=1
> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
> origin=core.action processed=4582 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
> origin=core.action processed=5039 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
> origin=core.action processed=862 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
> origin=core.action processed=397 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
> origin=core.action processed=50 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
> origin=core.action processed=2 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
> origin=core.action processed=13 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
> origin=core.action processed=766 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
> origin=core.action processed=11058 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
> origin=core.action processed=1099 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
> origin=core.action processed=12 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
> msgs.received=24227
>
>
> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
> origin=core.action processed=1728 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
> origin=core.action processed=47 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
> origin=core.action processed=35778 failed=35777 suspended=1
> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
> origin=core.action processed=9249 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
> origin=core.action processed=2296 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
> origin=core.action processed=1009 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
> origin=core.action processed=558 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
> origin=core.action processed=40 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
> origin=core.action processed=2 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
> origin=core.action processed=13 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
> origin=core.action processed=720 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
> origin=core.action processed=16593 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
> origin=core.action processed=359 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
> origin=core.action processed=2 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
> msgs.received=31257
>
>
> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
> origin=core.action processed=2181 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
> origin=core.action processed=57 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
> origin=core.action processed=20793 failed=137 suspended=0
> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
> origin=core.action processed=7599 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
> origin=core.action processed=5540 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
> origin=core.action processed=1120 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
> origin=core.action processed=157807 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
> origin=core.action processed=50 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
> origin=core.action processed=3 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
> origin=core.action processed=293 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
> origin=core.action processed=20243 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
> origin=core.action processed=866 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
> origin=core.action processed=10 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
> called.recvmsg=0 msgs.received=194836
>
>
> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
> origin=core.action processed=2009 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
> origin=core.action processed=151 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
> origin=core.action processed=52035 failed=52034 suspended=1
> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
> origin=core.action processed=19442 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
> origin=core.action processed=4480 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
> origin=core.action processed=1125 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
> origin=core.action processed=415 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
> origin=core.action processed=144 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
> origin=core.action processed=2 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
> origin=core.action processed=13 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
> origin=core.action processed=745 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
> origin=core.action processed=23626 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
> origin=core.action processed=682 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
> origin=core.action processed=45 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
> origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
> msgs.received=51446 -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 3:43 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users
> <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> I'm still not understanding what you mean by pstats - it's not a
>> package or command available to me. It's apart of Unix from what I
>> can tell. I've placed below the unparsed information form
>> /proc/net/netstat and /proc/net/udp
>>
>> /proc/net/netstat
>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
>> TCPMD5Failure TCPSackShifted T
> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
> TcpDuplicateDataRehash TCPDSACKRe
c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>
>> /proc/net/udp
>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>
>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>
>> Dropwatch (Cycle 1):
>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>
>> Dropwatch (Cycle 2):
>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>
>> Top -H
>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>
>> /etc/sysctl.conf
>> net.core.rmem_default = 33554432
>> net.core.rmem_max = 268435456
>> net.core.wmem_default = 33554432
>> net.core.wmem_max = 268435456
>> net.ipv4.tcp_mem = 190611 254150 381222
>> net.ipv4.tcp_rmem = 4096 131072 6291456
>> net.ipv4.tcp_wmem = 4096 16384 4194304
>> net.ipv4.udp_mem = 762450 1524900 3049800
>> net.ipv4.udp_rmem_min = 33554432
>> net.ipv4.udp_wmem_min = 33554432
>>
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 2:30 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>> <david@lang.hm>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> what does the pstats output look like when it's dropping messages?
>> (give a couple cycles please)
>>
>> did you try to eliminate the action queue for /var/log/secure?
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>> <david@lang.hm>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Building on this -
>>>
>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>> In:imdup spikes to ~10%
>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>> usage.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Redbourne,Michael via rsyslog
>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>> <david@lang.hm>
>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>> Udp:
>>> 5820820 packets received
>>> 1504 packets to unknown port received.
>>> 798900 packet receive errors
>>> 3338814 packets sent
>>> 798900 receive buffer errors
>>> 0 send buffer errors
>>>
>>> I have doubled the values in net.ipv4.udp_mem.
>>>
>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>
>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>> "<subnet>" stop
>>>
>>> Example Checkpoint Log:
>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>> dvc=<dvc_ip_addr>
>>>
>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>
>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Rainer Gerhards via rsyslog
>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>> To: David Lang <david@lang.hm>
>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>
>>> Rainer
>>>
>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>
>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>> input (usually I run into bottlenecks in processing the messages
>>>> long before I have problems accepting them)
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>
>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> w
>>>>> w
>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
>>>>> l
>>>>> &
>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
>>>>> 6
>>>>> 2
>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
>>>>> %
>>>>> 7
>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>>>> J
>>>>> Q
>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>> t
>>>>> a
>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>
>>>>> Rainer
>>>>>
>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>
>>>>>> Some additional comments on the config
>>>>>>
>>>>>>
>>>>>>
>>>>>> These action queue configs probably don't do what you intend them
>>>>>> to do
>>>>>>
>>>>>> the first thing is that they only affect the next action, which
>>>>>> is
>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>> write these logs out. That will create a HUGE amount of
>>>>>> contention for the queue lock and under load you should see it
>>>>>> maxing out quite quickly
>>>>>>
>>>>>> what is it that you are attempting to do here?
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Performance Tuning #
>>>>>> $ActionQueueWorkerThreads 2000
>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>> 600000
>>>>>>
>>>>>> #### RULES ####
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> #kern.* /dev/console
>>>>>>
>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>> authentication messages!
>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>
>>>>>> # The authpriv file has restricted access.
>>>>>> authpriv.* /var/log/secure
>>>>>>
>>>>>>
>>>>>> since the queue only applied to the next action with this config,
>>>>>> everything below this is operating from the main queue again as
>>>>>> if there was no action queue configuration
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.* -/var/log/maillog
>>>>>>
>>>>>> # Log cron stuff
>>>>>> cron.* /var/log/cron
>>>>>>
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg :omusrmsg:*
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>> uucp,news.crit /var/log/spooler
>>>>>>
>>>>>> # Save boot messages also to boot.log
>>>>>> # local7.* /var/syslog/boot.log
>>>>>>
>>>>>>
>>>>>>
>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>
>>>>>>
>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>> # [Firewall Log Filtering] #
>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>
>>>>>>
>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>
>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>> then @127.0.0.1:25224 & stop
>>>>>>
>>>>>>
>>>>>> combining multiple filters into one action, or having the filters
>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>
>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>
>>>>>> local0.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local1.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local2.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local3.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local4.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local5.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local6.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> local7.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> auth.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> authpriv.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> daemon.info @127.0.0.1:25224
>>>>>> & stop
>>>>>> syslog.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>> user.* @127.0.0.1:25224
>>>>>> & stop
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>> F
>>>>>> l
>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
>>>>>> 7
>>>>>> C
>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
>>>>>> 6
>>>>>> e
>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
>>>>>> 1
>>>>>> 5
>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
>>>>>> J
>>>>>> B
>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
>>>>>> y
>>>>>> n
>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
>>>>>> w
>>>>>> w
>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
>>>>>> e
>>>>>> l
>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
>>>>>> %
>>>>>> 7
>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
>>>>>> U
>>>>>> n
>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
>>>>>> k
>>>>>> 1
>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
>>>>>> I
>>>>>> o
>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>> What's up with rsyslog? Follow
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>> F
>>>>>> t
>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
>>>>>> l
>>>>>> e
>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
>>>>>> b
>>>>>> b
>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
>>>>>> b
>>>>>> 3
>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
>>>>>> %
>>>>>> 3
>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
>>>>>> O
>>>>>> Y
>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> s
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>> a
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>> C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>> w
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>> C
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>> %
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>> e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>> C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>> C
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>> t
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>> b
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>> 3
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>> F
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>> M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>> w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>> i
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>> o
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>> e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>> M
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>> C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>> s
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> ________________________________________
>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>
>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>> ________________________________________
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> s
>>> t
>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>> a
>>> e
>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>> C
>>> 9
>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>> w
>>> n
>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>> C
>>> J
>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>> %
>>> 2
>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>> e
>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>> C
>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>> C
>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>> t
>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>> b
>>> o
>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>> 3
>>> 8
>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>> F
>>> p
>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>> M
>>> n
>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>> w
>>> N
>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>> i
>>> t
>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>> o
>>> f
>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>> e
>>> 2
>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>> M
>>> C
>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>> C
>>> %
>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>> s
>>> e
>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
As I said before, log some of the messages with the template RSYSLOG_DebugFormat
and see what you have and how you can filter more efficiently.

Rsyslog is very efficient at processing messages, but regex and contains are
about the most expensive tests that you can do

If you really do need multiple threads to process the log messages, you need to
set the queue parameters on the main queue with the global() stanza.

I'm not convinced that we've caught the full problem during our sampling yet.

As I said, it's probably a good idea to put a queue on the TCP forwarding that
you have configured to isolate any issues there from the general processing.


@rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?

David Lang


On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 21:39:45 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
>
> If I'm still havcing issues after that, I'll respond back.
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 4:19 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> using the new action() syntax, you can name the actions so they aren't just numbered.
>
> starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
>
> that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)
>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 20:01:52 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
>> origin=core.action processed=1628 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
>> origin=core.action processed=66 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
>> origin=core.action processed=37449 failed=37448 suspended=1
>> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
>> origin=core.action processed=4582 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
>> origin=core.action processed=5039 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
>> origin=core.action processed=862 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
>> origin=core.action processed=397 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
>> origin=core.action processed=50 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
>> origin=core.action processed=766 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
>> origin=core.action processed=11058 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
>> origin=core.action processed=1099 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
>> origin=core.action processed=12 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
>> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
>> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
>> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
>> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
>> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
>> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
>> msgs.received=24227
>>
>>
>> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
>> origin=core.action processed=1728 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
>> origin=core.action processed=47 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
>> origin=core.action processed=35778 failed=35777 suspended=1
>> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
>> origin=core.action processed=9249 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
>> origin=core.action processed=2296 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
>> origin=core.action processed=1009 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
>> origin=core.action processed=558 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
>> origin=core.action processed=40 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
>> origin=core.action processed=720 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
>> origin=core.action processed=16593 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
>> origin=core.action processed=359 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
>> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
>> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
>> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
>> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
>> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
>> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
>> msgs.received=31257
>>
>>
>> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
>> origin=core.action processed=2181 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
>> origin=core.action processed=57 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
>> origin=core.action processed=20793 failed=137 suspended=0
>> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
>> origin=core.action processed=7599 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
>> origin=core.action processed=5540 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
>> origin=core.action processed=1120 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
>> origin=core.action processed=157807 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
>> origin=core.action processed=50 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
>> origin=core.action processed=3 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
>> origin=core.action processed=293 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
>> origin=core.action processed=20243 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
>> origin=core.action processed=866 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
>> origin=core.action processed=10 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
>> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
>> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
>> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
>> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
>> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
>> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
>> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
>> called.recvmsg=0 msgs.received=194836
>>
>>
>> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
>> origin=core.action processed=2009 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
>> origin=core.action processed=151 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
>> origin=core.action processed=52035 failed=52034 suspended=1
>> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
>> origin=core.action processed=19442 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
>> origin=core.action processed=4480 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
>> origin=core.action processed=1125 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
>> origin=core.action processed=415 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
>> origin=core.action processed=144 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
>> origin=core.action processed=745 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
>> origin=core.action processed=23626 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
>> origin=core.action processed=682 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
>> origin=core.action processed=45 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
>> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
>> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
>> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
>> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
>> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
>> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
>> msgs.received=51446 -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 3:43 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> I'm still not understanding what you mean by pstats - it's not a
>>> package or command available to me. It's apart of Unix from what I
>>> can tell. I've placed below the unparsed information form
>>> /proc/net/netstat and /proc/net/udp
>>>
>>> /proc/net/netstat
>>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
>>> TCPMD5Failure TCPSackShifted T
>> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
>> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
>> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
>> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
>> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
>> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
>> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
>> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
>> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
>> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
>> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
>> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
>> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
>> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
>> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
>> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
>> TcpDuplicateDataRehash TCPDSACKRe
> c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
>>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>>
>>> /proc/net/udp
>>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>>
>>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>>
>>> Dropwatch (Cycle 1):
>>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>
>>> Dropwatch (Cycle 2):
>>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>
>>> Top -H
>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>>
>>> /etc/sysctl.conf
>>> net.core.rmem_default = 33554432
>>> net.core.rmem_max = 268435456
>>> net.core.wmem_default = 33554432
>>> net.core.wmem_max = 268435456
>>> net.ipv4.tcp_mem = 190611 254150 381222
>>> net.ipv4.tcp_rmem = 4096 131072 6291456
>>> net.ipv4.tcp_wmem = 4096 16384 4194304
>>> net.ipv4.udp_mem = 762450 1524900 3049800
>>> net.ipv4.udp_rmem_min = 33554432
>>> net.ipv4.udp_wmem_min = 33554432
>>>
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, November 15, 2022 2:30 PM
>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>> <david@lang.hm>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> what does the pstats output look like when it's dropping messages?
>>> (give a couple cycles please)
>>>
>>> did you try to eliminate the action queue for /var/log/secure?
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>
>>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>>> <david@lang.hm>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Building on this -
>>>>
>>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>>> In:imdup spikes to ~10%
>>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>>> usage.
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Redbourne,Michael via rsyslog
>>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>> <david@lang.hm>
>>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>>> Udp:
>>>> 5820820 packets received
>>>> 1504 packets to unknown port received.
>>>> 798900 packet receive errors
>>>> 3338814 packets sent
>>>> 798900 receive buffer errors
>>>> 0 send buffer errors
>>>>
>>>> I have doubled the values in net.ipv4.udp_mem.
>>>>
>>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>>
>>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>>> "<subnet>" stop
>>>>
>>>> Example Checkpoint Log:
>>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>>> dvc=<dvc_ip_addr>
>>>>
>>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>>
>>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Rainer Gerhards via rsyslog
>>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>>> To: David Lang <david@lang.hm>
>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>>> <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>>
>>>> Rainer
>>>>
>>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>>
>>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>>> input (usually I run into bottlenecks in processing the messages
>>>>> long before I have problems accepting them)
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>>
>>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>>
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> w
>>>>>> w
>>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
>>>>>> l
>>>>>> &
>>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
>>>>>> 6
>>>>>> 2
>>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
>>>>>> %
>>>>>> 7
>>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>>>>> J
>>>>>> Q
>>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>>> t
>>>>>> a
>>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>>
>>>>>>> Some additional comments on the config
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> These action queue configs probably don't do what you intend them
>>>>>>> to do
>>>>>>>
>>>>>>> the first thing is that they only affect the next action, which
>>>>>>> is
>>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>>> write these logs out. That will create a HUGE amount of
>>>>>>> contention for the queue lock and under load you should see it
>>>>>>> maxing out quite quickly
>>>>>>>
>>>>>>> what is it that you are attempting to do here?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Performance Tuning #
>>>>>>> $ActionQueueWorkerThreads 2000
>>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>>> 600000
>>>>>>>
>>>>>>> #### RULES ####
>>>>>>> # Log all kernel messages to the console.
>>>>>>> # Logging much else clutters up the screen.
>>>>>>> #kern.* /dev/console
>>>>>>>
>>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>>> authentication messages!
>>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>>
>>>>>>> # The authpriv file has restricted access.
>>>>>>> authpriv.* /var/log/secure
>>>>>>>
>>>>>>>
>>>>>>> since the queue only applied to the next action with this config,
>>>>>>> everything below this is operating from the main queue again as
>>>>>>> if there was no action queue configuration
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Log all the mail messages in one place.
>>>>>>> mail.* -/var/log/maillog
>>>>>>>
>>>>>>> # Log cron stuff
>>>>>>> cron.* /var/log/cron
>>>>>>>
>>>>>>> # Everybody gets emergency messages
>>>>>>> *.emerg :omusrmsg:*
>>>>>>>
>>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>>> uucp,news.crit /var/log/spooler
>>>>>>>
>>>>>>> # Save boot messages also to boot.log
>>>>>>> # local7.* /var/syslog/boot.log
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>>
>>>>>>>
>>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>>> # [Firewall Log Filtering] #
>>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>>
>>>>>>>
>>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>>
>>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>>> then @127.0.0.1:25224 & stop
>>>>>>>
>>>>>>>
>>>>>>> combining multiple filters into one action, or having the filters
>>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>>
>>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>>
>>>>>>> local0.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local1.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local2.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local3.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local4.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local5.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local6.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local7.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> auth.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> authpriv.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> daemon.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> syslog.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>>> user.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>> F
>>>>>>> l
>>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
>>>>>>> 7
>>>>>>> C
>>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
>>>>>>> 6
>>>>>>> e
>>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
>>>>>>> 1
>>>>>>> 5
>>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
>>>>>>> J
>>>>>>> B
>>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
>>>>>>> y
>>>>>>> n
>>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
>>>>>>> w
>>>>>>> w
>>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
>>>>>>> e
>>>>>>> l
>>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
>>>>>>> %
>>>>>>> 7
>>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
>>>>>>> U
>>>>>>> n
>>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
>>>>>>> k
>>>>>>> 1
>>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
>>>>>>> I
>>>>>>> o
>>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>>> What's up with rsyslog? Follow
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>> F
>>>>>>> t
>>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
>>>>>>> l
>>>>>>> e
>>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
>>>>>>> b
>>>>>>> b
>>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
>>>>>>> b
>>>>>>> 3
>>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
>>>>>>> %
>>>>>>> 3
>>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
>>>>>>> O
>>>>>>> Y
>>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>> s
>>>> t
>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>> a
>>>> e
>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>> C
>>>> 9
>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>> w
>>>> n
>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>> C
>>>> J
>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>> %
>>>> 2
>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>> e
>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>> C
>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>> C
>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>> t
>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>> b
>>>> o
>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>> 3
>>>> 8
>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>> F
>>>> p
>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>> M
>>>> n
>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>> w
>>>> N
>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>> i
>>>> t
>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>> o
>>>> f
>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>> e
>>>> 2
>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>> M
>>>> C
>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>> C
>>>> %
>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>> s
>>>> e
>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> ________________________________________
>>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>>
>>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>>> ________________________________________
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>> s
>>>> t
>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>> a
>>>> e
>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>> C
>>>> 9
>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>> w
>>>> n
>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>> C
>>>> J
>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>> %
>>>> 2
>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>> e
>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>> C
>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>> C
>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>> t
>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>> b
>>>> o
>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>> 3
>>>> 8
>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>> F
>>>> p
>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>> M
>>>> n
>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>> w
>>>> N
>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>> i
>>>> t
>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>> o
>>>> f
>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>> e
>>>> 2
>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>> M
>>>> C
>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>> C
>>>> %
>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>> s
>>>> e
>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and might help clear up what the actual issue is.

If you're referencing this:

>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*

It's owned by a completely different process (and package) from rsyslog. It's the Microsoft OMS Agent for Linux Package, available here: https://github.com/microsoft/OMS-Agent-for-Linux. Unfortunately, it's not something I can get rid of... yet. I'm waiting on Microsoft to support the newer Azure ARC agent for new builds of CentOS/RHEL 9.

Cheers,
Mike

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, November 15, 2022 5:49 PM
To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently.

Rsyslog is very efficient at processing messages, but regex and contains are about the most expensive tests that you can do

If you really do need multiple threads to process the log messages, you need to set the queue parameters on the main queue with the global() stanza.

I'm not convinced that we've caught the full problem during our sampling yet.

As I said, it's probably a good idea to put a queue on the TCP forwarding that you have configured to isolate any issues there from the general processing.


@rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?

David Lang


On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 21:39:45 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
>
> If I'm still havcing issues after that, I'll respond back.
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 4:19 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> using the new action() syntax, you can name the actions so they aren't just numbered.
>
> starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
>
> that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)
>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 20:01:52 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
>> origin=core.action processed=1628 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
>> origin=core.action processed=66 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
>> origin=core.action processed=37449 failed=37448 suspended=1
>> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
>> origin=core.action processed=4582 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
>> origin=core.action processed=5039 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
>> origin=core.action processed=862 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
>> origin=core.action processed=397 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
>> origin=core.action processed=50 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
>> origin=core.action processed=766 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
>> origin=core.action processed=11058 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
>> origin=core.action processed=1099 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
>> origin=core.action processed=12 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
>> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
>> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
>> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
>> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
>> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
>> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
>> msgs.received=24227
>>
>>
>> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
>> origin=core.action processed=1728 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
>> origin=core.action processed=47 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
>> origin=core.action processed=35778 failed=35777 suspended=1
>> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
>> origin=core.action processed=9249 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
>> origin=core.action processed=2296 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
>> origin=core.action processed=1009 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
>> origin=core.action processed=558 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
>> origin=core.action processed=40 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
>> origin=core.action processed=720 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
>> origin=core.action processed=16593 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
>> origin=core.action processed=359 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
>> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
>> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
>> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
>> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
>> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
>> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
>> msgs.received=31257
>>
>>
>> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
>> origin=core.action processed=2181 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
>> origin=core.action processed=57 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
>> origin=core.action processed=20793 failed=137 suspended=0
>> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
>> origin=core.action processed=7599 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
>> origin=core.action processed=5540 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
>> origin=core.action processed=1120 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
>> origin=core.action processed=157807 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
>> origin=core.action processed=50 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
>> origin=core.action processed=3 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
>> origin=core.action processed=293 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
>> origin=core.action processed=20243 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
>> origin=core.action processed=866 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
>> origin=core.action processed=10 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
>> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
>> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
>> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
>> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
>> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
>> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
>> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
>> called.recvmsg=0 msgs.received=194836
>>
>>
>> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
>> origin=core.action processed=2009 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
>> origin=core.action processed=151 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
>> origin=core.action processed=52035 failed=52034 suspended=1
>> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
>> origin=core.action processed=19442 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
>> origin=core.action processed=4480 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
>> origin=core.action processed=1125 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
>> origin=core.action processed=415 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
>> origin=core.action processed=144 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
>> origin=core.action processed=2 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
>> origin=core.action processed=13 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
>> origin=core.action processed=745 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
>> origin=core.action processed=23626 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
>> origin=core.action processed=682 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
>> origin=core.action processed=45 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
>> origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
>> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
>> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
>> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
>> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
>> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
>> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
>> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
>> msgs.received=51446 -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 3:43 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>>
>> David Lang
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> I'm still not understanding what you mean by pstats - it's not a
>>> package or command available to me. It's apart of Unix from what I
>>> can tell. I've placed below the unparsed information form
>>> /proc/net/netstat and /proc/net/udp
>>>
>>> /proc/net/netstat
>>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
>>> TCPMD5Failure TCPSackShifted T
>> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
>> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
>> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
>> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
>> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
>> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
>> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
>> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
>> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
>> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
>> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
>> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
>> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
>> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
>> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
>> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
>> TcpDuplicateDataRehash TCPDSACKRe
> c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
>>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>>
>>> /proc/net/udp
>>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>>
>>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>>
>>> Dropwatch (Cycle 1):
>>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>
>>> Dropwatch (Cycle 2):
>>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>
>>> Top -H
>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>>
>>> /etc/sysctl.conf
>>> net.core.rmem_default = 33554432
>>> net.core.rmem_max = 268435456
>>> net.core.wmem_default = 33554432
>>> net.core.wmem_max = 268435456
>>> net.ipv4.tcp_mem = 190611 254150 381222
>>> net.ipv4.tcp_rmem = 4096 131072 6291456
>>> net.ipv4.tcp_wmem = 4096 16384 4194304
>>> net.ipv4.udp_mem = 762450 1524900 3049800
>>> net.ipv4.udp_rmem_min = 33554432
>>> net.ipv4.udp_wmem_min = 33554432
>>>
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, November 15, 2022 2:30 PM
>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>> <david@lang.hm>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> what does the pstats output look like when it's dropping messages?
>>> (give a couple cycles please)
>>>
>>> did you try to eliminate the action queue for /var/log/secure?
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>
>>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>>> <david@lang.hm>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Building on this -
>>>>
>>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>>> In:imdup spikes to ~10%
>>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>>> usage.
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Redbourne,Michael via rsyslog
>>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>> <david@lang.hm>
>>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>>> Udp:
>>>> 5820820 packets received
>>>> 1504 packets to unknown port received.
>>>> 798900 packet receive errors
>>>> 3338814 packets sent
>>>> 798900 receive buffer errors
>>>> 0 send buffer errors
>>>>
>>>> I have doubled the values in net.ipv4.udp_mem.
>>>>
>>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>>
>>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>>> "<subnet>" stop
>>>>
>>>> Example Checkpoint Log:
>>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>>> dvc=<dvc_ip_addr>
>>>>
>>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>>
>>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Rainer Gerhards via rsyslog
>>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>>> To: David Lang <david@lang.hm>
>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>>> <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>>
>>>> Rainer
>>>>
>>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>>
>>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>>> input (usually I run into bottlenecks in processing the messages
>>>>> long before I have problems accepting them)
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>>
>>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>>
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> w
>>>>>> w
>>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
>>>>>> l
>>>>>> &
>>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
>>>>>> 6
>>>>>> 2
>>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
>>>>>> %
>>>>>> 7
>>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>>>>> J
>>>>>> Q
>>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>>> t
>>>>>> a
>>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>>
>>>>>>> Some additional comments on the config
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> These action queue configs probably don't do what you intend them
>>>>>>> to do
>>>>>>>
>>>>>>> the first thing is that they only affect the next action, which
>>>>>>> is
>>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>>> write these logs out. That will create a HUGE amount of
>>>>>>> contention for the queue lock and under load you should see it
>>>>>>> maxing out quite quickly
>>>>>>>
>>>>>>> what is it that you are attempting to do here?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Performance Tuning #
>>>>>>> $ActionQueueWorkerThreads 2000
>>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>>> 600000
>>>>>>>
>>>>>>> #### RULES ####
>>>>>>> # Log all kernel messages to the console.
>>>>>>> # Logging much else clutters up the screen.
>>>>>>> #kern.* /dev/console
>>>>>>>
>>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>>> authentication messages!
>>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>>
>>>>>>> # The authpriv file has restricted access.
>>>>>>> authpriv.* /var/log/secure
>>>>>>>
>>>>>>>
>>>>>>> since the queue only applied to the next action with this config,
>>>>>>> everything below this is operating from the main queue again as
>>>>>>> if there was no action queue configuration
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Log all the mail messages in one place.
>>>>>>> mail.* -/var/log/maillog
>>>>>>>
>>>>>>> # Log cron stuff
>>>>>>> cron.* /var/log/cron
>>>>>>>
>>>>>>> # Everybody gets emergency messages
>>>>>>> *.emerg :omusrmsg:*
>>>>>>>
>>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>>> uucp,news.crit /var/log/spooler
>>>>>>>
>>>>>>> # Save boot messages also to boot.log
>>>>>>> # local7.* /var/syslog/boot.log
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>>
>>>>>>>
>>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>>> # [Firewall Log Filtering] #
>>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>>
>>>>>>>
>>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>>
>>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>>> then @127.0.0.1:25224 & stop
>>>>>>>
>>>>>>>
>>>>>>> combining multiple filters into one action, or having the filters
>>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>>
>>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>>
>>>>>>> local0.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local1.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local2.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local3.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local4.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local5.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local6.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> local7.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> auth.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> authpriv.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> daemon.info @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> syslog.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>>> user.* @127.0.0.1:25224
>>>>>>> & stop
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>> F
>>>>>>> l
>>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
>>>>>>> 7
>>>>>>> C
>>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
>>>>>>> 6
>>>>>>> e
>>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
>>>>>>> 1
>>>>>>> 5
>>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
>>>>>>> J
>>>>>>> B
>>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
>>>>>>> y
>>>>>>> n
>>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
>>>>>>> w
>>>>>>> w
>>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
>>>>>>> e
>>>>>>> l
>>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
>>>>>>> %
>>>>>>> 7
>>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
>>>>>>> U
>>>>>>> n
>>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
>>>>>>> k
>>>>>>> 1
>>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
>>>>>>> I
>>>>>>> o
>>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>>> What's up with rsyslog? Follow
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>> F
>>>>>>> t
>>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
>>>>>>> l
>>>>>>> e
>>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
>>>>>>> b
>>>>>>> b
>>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
>>>>>>> b
>>>>>>> 3
>>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
>>>>>>> %
>>>>>>> 3
>>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
>>>>>>> O
>>>>>>> Y
>>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>> s
>>>> t
>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>> a
>>>> e
>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>> C
>>>> 9
>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>> w
>>>> n
>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>> C
>>>> J
>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>> %
>>>> 2
>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>> e
>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>> C
>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>> C
>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>> t
>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>> b
>>>> o
>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>> 3
>>>> 8
>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>> F
>>>> p
>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>> M
>>>> n
>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>> w
>>>> N
>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>> i
>>>> t
>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>> o
>>>> f
>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>> e
>>>> 2
>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>> M
>>>> C
>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>> C
>>>> %
>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>> s
>>>> e
>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> ________________________________________
>>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>>
>>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>>> ________________________________________
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>> s
>>>> t
>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>> a
>>>> e
>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>> C
>>>> 9
>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>> w
>>>> n
>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>> C
>>>> J
>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>> %
>>>> 2
>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>> e
>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>> C
>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>> C
>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>> t
>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>> b
>>>> o
>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>> 3
>>>> 8
>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>> F
>>>> p
>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>> M
>>>> n
>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>> w
>>>> N
>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>> i
>>>> t
>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>> o
>>>> f
>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>> e
>>>> 2
>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>> M
>>>> C
>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>> C
>>>> %
>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>> s
>>>> e
>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
ahh, I thought that was a rsyslog thread that could be maxing out a core.

my logging strategy is that everything should get sent to the central syslog
server, and only there should it get thrown away (and before you throw it away,
consider counting it, the number of times that an uninteresting thing happens
can be an interesting thing to track)

per the impstats output you included, you are hitting up to ~240k messages/30
seconds, or around 8k messages/sec. That should not be challenging to rsyslog to
receive (I've done over 300k messages/sec), we just need to watch the cost of
processing them.

you don't show any times when the queue actually filled up, so messages should
not be dropped because of that.

with the new configs, are you sure you are still dropping syslog messages? the
proc stats you show below show drops on port hex 6288 (25224 decimal), not on
hex 202 (514 decimal)

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 21:55:48 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and might help clear up what the actual issue is.
>
> If you're referencing this:
>
>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>
> It's owned by a completely different process (and package) from rsyslog. It's the Microsoft OMS Agent for Linux Package, available here: https://github.com/microsoft/OMS-Agent-for-Linux. Unfortunately, it's not something I can get rid of... yet. I'm waiting on Microsoft to support the newer Azure ARC agent for new builds of CentOS/RHEL 9.
>
> Cheers,
> Mike
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 5:49 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently.
>
> Rsyslog is very efficient at processing messages, but regex and contains are about the most expensive tests that you can do
>
> If you really do need multiple threads to process the log messages, you need to set the queue parameters on the main queue with the global() stanza.
>
> I'm not convinced that we've caught the full problem during our sampling yet.
>
> As I said, it's probably a good idea to put a queue on the TCP forwarding that you have configured to isolate any issues there from the general processing.
>
>
> @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?
>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 21:39:45 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
>>
>> If I'm still havcing issues after that, I'll respond back.
>>
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 4:19 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> using the new action() syntax, you can name the actions so they aren't just numbered.
>>
>> starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
>>
>> that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)
>>
>> David Lang
>>
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 20:01:52 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>>
>>> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
>>> origin=core.action processed=1628 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
>>> origin=core.action processed=66 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
>>> origin=core.action processed=37449 failed=37448 suspended=1
>>> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
>>> origin=core.action processed=4582 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
>>> origin=core.action processed=5039 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
>>> origin=core.action processed=862 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
>>> origin=core.action processed=397 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
>>> origin=core.action processed=50 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
>>> origin=core.action processed=766 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
>>> origin=core.action processed=11058 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
>>> origin=core.action processed=1099 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
>>> origin=core.action processed=12 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
>>> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
>>> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
>>> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
>>> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
>>> msgs.received=24227
>>>
>>>
>>> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
>>> origin=core.action processed=1728 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
>>> origin=core.action processed=47 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
>>> origin=core.action processed=35778 failed=35777 suspended=1
>>> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
>>> origin=core.action processed=9249 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
>>> origin=core.action processed=2296 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
>>> origin=core.action processed=1009 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
>>> origin=core.action processed=558 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
>>> origin=core.action processed=40 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
>>> origin=core.action processed=720 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
>>> origin=core.action processed=16593 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
>>> origin=core.action processed=359 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
>>> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
>>> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
>>> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
>>> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
>>> msgs.received=31257
>>>
>>>
>>> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
>>> origin=core.action processed=2181 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
>>> origin=core.action processed=57 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
>>> origin=core.action processed=20793 failed=137 suspended=0
>>> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
>>> origin=core.action processed=7599 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
>>> origin=core.action processed=5540 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
>>> origin=core.action processed=1120 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
>>> origin=core.action processed=157807 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
>>> origin=core.action processed=50 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
>>> origin=core.action processed=3 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
>>> origin=core.action processed=293 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
>>> origin=core.action processed=20243 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
>>> origin=core.action processed=866 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
>>> origin=core.action processed=10 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
>>> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
>>> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
>>> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
>>> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
>>> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
>>> called.recvmsg=0 msgs.received=194836
>>>
>>>
>>> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
>>> origin=core.action processed=2009 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
>>> origin=core.action processed=151 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
>>> origin=core.action processed=52035 failed=52034 suspended=1
>>> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
>>> origin=core.action processed=19442 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
>>> origin=core.action processed=4480 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
>>> origin=core.action processed=1125 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
>>> origin=core.action processed=415 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
>>> origin=core.action processed=144 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
>>> origin=core.action processed=745 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
>>> origin=core.action processed=23626 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
>>> origin=core.action processed=682 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
>>> origin=core.action processed=45 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
>>> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
>>> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
>>> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
>>> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
>>> msgs.received=51446 -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, November 15, 2022 3:43 PM
>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>>
>>> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>
>>>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>> To: David Lang <david@lang.hm>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> I'm still not understanding what you mean by pstats - it's not a
>>>> package or command available to me. It's apart of Unix from what I
>>>> can tell. I've placed below the unparsed information form
>>>> /proc/net/netstat and /proc/net/udp
>>>>
>>>> /proc/net/netstat
>>>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
>>>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
>>>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
>>>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
>>>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
>>>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
>>>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
>>>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
>>>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>>>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>>>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>>>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>>>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>>>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>>>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
>>>> TCPMD5Failure TCPSackShifted T
>>> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
>>> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
>>> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
>>> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
>>> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
>>> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
>>> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
>>> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
>>> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
>>> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
>>> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
>>> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
>>> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
>>> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
>>> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
>>> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
>>> TcpDuplicateDataRehash TCPDSACKRe
>> c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
>>>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
>>>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
>>>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
>>>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
>>>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>>>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>>>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>>>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
>>>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>>>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
>>>>
>>>> /proc/net/udp
>>>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>>>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>>>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>>>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>>>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>>>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>>>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>>>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>>>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>>>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>>>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>>>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>>>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>>>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>>>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>>>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>>>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>>>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>>>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>>>
>>>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>>>
>>>> Dropwatch (Cycle 1):
>>>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>>>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>>>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>
>>>> Dropwatch (Cycle 2):
>>>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>>>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>>>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>>
>>>> Top -H
>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>>>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>>>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>>>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>>>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>>>
>>>> /etc/sysctl.conf
>>>> net.core.rmem_default = 33554432
>>>> net.core.rmem_max = 268435456
>>>> net.core.wmem_default = 33554432
>>>> net.core.wmem_max = 268435456
>>>> net.ipv4.tcp_mem = 190611 254150 381222
>>>> net.ipv4.tcp_rmem = 4096 131072 6291456
>>>> net.ipv4.tcp_wmem = 4096 16384 4194304
>>>> net.ipv4.udp_mem = 762450 1524900 3049800
>>>> net.ipv4.udp_rmem_min = 33554432
>>>> net.ipv4.udp_wmem_min = 33554432
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Tuesday, November 15, 2022 2:30 PM
>>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>> <david@lang.hm>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> what does the pstats output look like when it's dropping messages?
>>>> (give a couple cycles please)
>>>>
>>>> did you try to eliminate the action queue for /var/log/secure?
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>>
>>>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>>>> <david@lang.hm>
>>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Building on this -
>>>>>
>>>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>>>> In:imdup spikes to ~10%
>>>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>>>> usage.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Redbourne,Michael via rsyslog
>>>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>>> <david@lang.hm>
>>>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>>>> Udp:
>>>>> 5820820 packets received
>>>>> 1504 packets to unknown port received.
>>>>> 798900 packet receive errors
>>>>> 3338814 packets sent
>>>>> 798900 receive buffer errors
>>>>> 0 send buffer errors
>>>>>
>>>>> I have doubled the values in net.ipv4.udp_mem.
>>>>>
>>>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>>>
>>>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>>>> "<subnet>" stop
>>>>>
>>>>> Example Checkpoint Log:
>>>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>>>> dvc=<dvc_ip_addr>
>>>>>
>>>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>>>
>>>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Rainer Gerhards via rsyslog
>>>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>>>> <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>>>
>>>>> Rainer
>>>>>
>>>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>>>
>>>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>>>> input (usually I run into bottlenecks in processing the messages
>>>>>> long before I have problems accepting them)
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>>>
>>>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>>>
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>>> w
>>>>>>> w
>>>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
>>>>>>> l
>>>>>>> &
>>>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
>>>>>>> 6
>>>>>>> 2
>>>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
>>>>>>> %
>>>>>>> 7
>>>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
>>>>>>> J
>>>>>>> Q
>>>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>>>> t
>>>>>>> a
>>>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>>>
>>>>>>> Rainer
>>>>>>>
>>>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>>>
>>>>>>>> Some additional comments on the config
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> These action queue configs probably don't do what you intend them
>>>>>>>> to do
>>>>>>>>
>>>>>>>> the first thing is that they only affect the next action, which
>>>>>>>> is
>>>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>>>> write these logs out. That will create a HUGE amount of
>>>>>>>> contention for the queue lock and under load you should see it
>>>>>>>> maxing out quite quickly
>>>>>>>>
>>>>>>>> what is it that you are attempting to do here?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Performance Tuning #
>>>>>>>> $ActionQueueWorkerThreads 2000
>>>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
>>>>>>>> 600000
>>>>>>>>
>>>>>>>> #### RULES ####
>>>>>>>> # Log all kernel messages to the console.
>>>>>>>> # Logging much else clutters up the screen.
>>>>>>>> #kern.* /dev/console
>>>>>>>>
>>>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>>>> authentication messages!
>>>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>>>
>>>>>>>> # The authpriv file has restricted access.
>>>>>>>> authpriv.* /var/log/secure
>>>>>>>>
>>>>>>>>
>>>>>>>> since the queue only applied to the next action with this config,
>>>>>>>> everything below this is operating from the main queue again as
>>>>>>>> if there was no action queue configuration
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Log all the mail messages in one place.
>>>>>>>> mail.* -/var/log/maillog
>>>>>>>>
>>>>>>>> # Log cron stuff
>>>>>>>> cron.* /var/log/cron
>>>>>>>>
>>>>>>>> # Everybody gets emergency messages
>>>>>>>> *.emerg :omusrmsg:*
>>>>>>>>
>>>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>>>> uucp,news.crit /var/log/spooler
>>>>>>>>
>>>>>>>> # Save boot messages also to boot.log
>>>>>>>> # local7.* /var/syslog/boot.log
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ereregex is a fairly expensive filter to apply, it's much better
>>>>>>>> to figure out a non-regex approach to filtering these. Can you
>>>>>>>> post some examples of what you are trying to filter? mmnormalize
>>>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>>>
>>>>>>>>
>>>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>>>> # [Firewall Log Filtering] #
>>>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
>>>>>>>>
>>>>>>>>
>>>>>>>> check the messages to see where CEF: and ASA- are in the message,
>>>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>>>
>>>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>>>> then @127.0.0.1:25224 & stop
>>>>>>>>
>>>>>>>>
>>>>>>>> combining multiple filters into one action, or having the filters
>>>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>>>
>>>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>>>
>>>>>>>> local0.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local1.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local2.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local3.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local4.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local5.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local6.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local7.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> auth.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> authpriv.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> daemon.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> syslog.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>>>> user.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>> F
>>>>>>>> l
>>>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
>>>>>>>> 7
>>>>>>>> C
>>>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
>>>>>>>> 6
>>>>>>>> e
>>>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
>>>>>>>> 1
>>>>>>>> 5
>>>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
>>>>>>>> J
>>>>>>>> B
>>>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
>>>>>>>> y
>>>>>>>> n
>>>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
>>>>>>>> w
>>>>>>>> w
>>>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
>>>>>>>> e
>>>>>>>> l
>>>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
>>>>>>>> %
>>>>>>>> 7
>>>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
>>>>>>>> U
>>>>>>>> n
>>>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
>>>>>>>> k
>>>>>>>> 1
>>>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
>>>>>>>> I
>>>>>>>> o
>>>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>>>> What's up with rsyslog? Follow
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>> F
>>>>>>>> t
>>>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
>>>>>>>> l
>>>>>>>> e
>>>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
>>>>>>>> b
>>>>>>>> b
>>>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
>>>>>>>> b
>>>>>>>> 3
>>>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
>>>>>>>> %
>>>>>>>> 3
>>>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
>>>>>>>> O
>>>>>>>> Y
>>>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
>>>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>>> s
>>>>> t
>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>>> a
>>>>> e
>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>>> C
>>>>> 9
>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>>> w
>>>>> n
>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>>> C
>>>>> J
>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>>> %
>>>>> 2
>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>>> e
>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>>> C
>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>>> C
>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>> t
>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>>> b
>>>>> o
>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>>> 3
>>>>> 8
>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>>> F
>>>>> p
>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>>> M
>>>>> n
>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>>> w
>>>>> N
>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>>> i
>>>>> t
>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>>> o
>>>>> f
>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>>> e
>>>>> 2
>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>>> M
>>>>> C
>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>>> C
>>>>> %
>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>>> s
>>>>> e
>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>> ________________________________________
>>>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>>>
>>>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>>>> ________________________________________
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>>> s
>>>>> t
>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
>>>>> a
>>>>> e
>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
>>>>> C
>>>>> 9
>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
>>>>> w
>>>>> n
>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>>> C
>>>>> J
>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
>>>>> %
>>>>> 2
>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
>>>>> e
>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>>> C
>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>>>> C
>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
>>>>> t
>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
>>>>> b
>>>>> o
>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
>>>>> 3
>>>>> 8
>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
>>>>> F
>>>>> p
>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>>>>> M
>>>>> n
>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
>>>>> w
>>>>> N
>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
>>>>> i
>>>>> t
>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
>>>>> o
>>>>> f
>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
>>>>> e
>>>>> 2
>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>>>>> M
>>>>> C
>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
>>>>> C
>>>>> %
>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
>>>>> s
>>>>> e
>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
I hadn't even noticed that... OK, time to go speak with someone else... Port 25224 (UDP) is the OMS Agent's Syslog. Port 25226 (TCP) is for CEF events. With the events being dropped at a different application (OMS Agent for Linux), this isn't something you can likely assist with. I'll submit a support case with Microsoft. The setup on that is baseline...

Back when I posted the original /etc/rsyslog.d/security-config-omsagent.conf file, there were several daemons pointed to 25224, in addition to other log sources (Infoblox).

0100007F -> 1.0.0.127, which I'm assuming it probably meant 127.0.0.1.



-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, November 15, 2022 6:06 PM
To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

ahh, I thought that was a rsyslog thread that could be maxing out a core.

my logging strategy is that everything should get sent to the central syslog server, and only there should it get thrown away (and before you throw it away, consider counting it, the number of times that an uninteresting thing happens can be an interesting thing to track)

per the impstats output you included, you are hitting up to ~240k messages/30 seconds, or around 8k messages/sec. That should not be challenging to rsyslog to receive (I've done over 300k messages/sec), we just need to watch the cost of processing them.

you don't show any times when the queue actually filled up, so messages should not be dropped because of that.

with the new configs, are you sure you are still dropping syslog messages? the proc stats you show below show drops on port hex 6288 (25224 decimal), not on hex 202 (514 decimal)

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> Date: Tue, 15 Nov 2022 21:55:48 +0000
> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and might help clear up what the actual issue is.
>
> If you're referencing this:
>
>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>
> It's owned by a completely different process (and package) from rsyslog. It's the Microsoft OMS Agent for Linux Package, available here: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FOMS-Agent-for-Linux&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7C8706a226fa1240396fee08dac7558349%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041467339516478%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=7xWiZ3%2FskQ3E8kghTzYd%2FAwZ%2BVddR3HJPK%2FMKgDWmxg%3D&amp;reserved=0. Unfortunately, it's not something I can get rid of... yet. I'm waiting on Microsoft to support the newer Azure ARC agent for new builds of CentOS/RHEL 9.
>
> Cheers,
> Mike
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 5:49 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users
> <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently.
>
> Rsyslog is very efficient at processing messages, but regex and
> contains are about the most expensive tests that you can do
>
> If you really do need multiple threads to process the log messages, you need to set the queue parameters on the main queue with the global() stanza.
>
> I'm not convinced that we've caught the full problem during our sampling yet.
>
> As I said, it's probably a good idea to put a queue on the TCP forwarding that you have configured to isolate any issues there from the general processing.
>
>
> @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?
>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 21:39:45 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
>>
>> If I'm still havcing issues after that, I'll respond back.
>>
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 4:19 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>> Events
>>
>> using the new action() syntax, you can name the actions so they aren't just numbered.
>>
>> starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
>>
>> that could be the tcp forwarding of messages, which can stall if the
>> device you are forwarding to can't keep up (that is a good place to
>> have an action queue, but please use the action() syntax so that it's
>> clear what the queue is doing)
>>
>> David Lang
>>
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 20:01:52 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15
>>> 00:22:18
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
>>> origin=core.action processed=1628 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
>>> origin=core.action processed=66 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
>>> origin=core.action processed=37449 failed=37448 suspended=1
>>> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
>>> origin=core.action processed=4582 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
>>> origin=core.action processed=5039 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
>>> origin=core.action processed=862 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
>>> origin=core.action processed=397 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
>>> origin=core.action processed=50 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
>>> origin=core.action processed=766 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
>>> origin=core.action processed=11058 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
>>> origin=core.action processed=1099 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
>>> origin=core.action processed=12 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
>>> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
>>> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
>>> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov
>>> 15
>>> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161
>>> full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
>>> msgs.received=24227
>>>
>>>
>>> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15
>>> 08:41:43
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
>>> origin=core.action processed=1728 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
>>> origin=core.action processed=47 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
>>> origin=core.action processed=35778 failed=35777 suspended=1
>>> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
>>> origin=core.action processed=9249 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
>>> origin=core.action processed=2296 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
>>> origin=core.action processed=1009 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
>>> origin=core.action processed=558 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
>>> origin=core.action processed=40 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
>>> origin=core.action processed=720 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
>>> origin=core.action processed=16593 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
>>> origin=core.action processed=359 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
>>> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
>>> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
>>> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov
>>> 15
>>> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998
>>> full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
>>> msgs.received=31257
>>>
>>>
>>> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15
>>> 08:42:13
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
>>> origin=core.action processed=2181 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
>>> origin=core.action processed=57 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
>>> origin=core.action processed=20793 failed=137 suspended=0
>>> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
>>> origin=core.action processed=7599 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
>>> origin=core.action processed=5540 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
>>> origin=core.action processed=1120 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
>>> origin=core.action processed=157807 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
>>> origin=core.action processed=50 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
>>> origin=core.action processed=3 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
>>> origin=core.action processed=293 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
>>> origin=core.action processed=20243 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
>>> origin=core.action processed=866 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
>>> origin=core.action processed=10 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
>>> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
>>> origin=impstats utime=5417168 stime=3346362 maxrss=16800
>>> minflt=11196
>>> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
>>> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
>>> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
>>> called.recvmsg=0 msgs.received=194836
>>>
>>>
>>> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15
>>> 10:15:04
>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
>>> origin=core.action processed=2009 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
>>> origin=core.action processed=151 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
>>> origin=core.action processed=52035 failed=52034 suspended=1
>>> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
>>> origin=core.action processed=19442 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
>>> origin=core.action processed=4480 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
>>> origin=core.action processed=1125 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
>>> origin=core.action processed=415 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
>>> origin=core.action processed=144 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
>>> origin=core.action processed=2 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
>>> origin=core.action processed=13 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
>>> origin=core.action processed=745 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
>>> origin=core.action processed=23626 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
>>> origin=core.action processed=682 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
>>> origin=core.action processed=45 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
>>> origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
>>> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
>>> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
>>> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov
>>> 15
>>> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235
>>> full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
>>> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
>>> msgs.received=51446 -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, November 15, 2022 3:43 PM
>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>>>
>>> David Lang
>>>
>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>
>>>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>> To: David Lang <david@lang.hm>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> I'm still not understanding what you mean by pstats - it's not a
>>>> package or command available to me. It's apart of Unix from what I
>>>> can tell. I've placed below the unparsed information form
>>>> /proc/net/netstat and /proc/net/udp
>>>>
>>>> /proc/net/netstat
>>>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed
>>>> EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps
>>>> LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive
>>>> PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost
>>>> ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks
>>>> TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPSACKReorder
>>>> TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo
>>>> TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures
>>>> TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts
>>>> TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>>>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>>>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>>>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>>>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>>>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>>>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound
>>>> TCPMD5Unexpected TCPMD5Failure TCPSackShifted T
>>> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
>>> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
>>> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
>>> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
>>> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
>>> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
>>> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
>>> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
>>> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
>>> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
>>> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
>>> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
>>> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
>>> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
>>> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
>>> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
>>> TcpDuplicateDataRehash TCPDSACKRe
>> c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess
>> TCPMigrateReqFailure
>>>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402
>>>> 5231
>>>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0
>>>> 0
>>>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0
>>>> 0
>>>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24
>>>> 0
>>>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>>>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>>>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>>>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors
>>>> InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>>>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0
>>>> 0
>>>>
>>>> /proc/net/udp
>>>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>>>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>>>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>>>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>>>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>>>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>>>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>>>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>>>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>>>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>>>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>>>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>>>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>>>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>>>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>>>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>>>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>>>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>>>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>>>
>>>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>>>
>>>> Dropwatch (Cycle 1):
>>>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>>>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>>>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>
>>>> Dropwatch (Cycle 2):
>>>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>>>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>>>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>>
>>>> Top -H
>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>>>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>>>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>>>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>>>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>>>
>>>> /etc/sysctl.conf
>>>> net.core.rmem_default = 33554432
>>>> net.core.rmem_max = 268435456
>>>> net.core.wmem_default = 33554432
>>>> net.core.wmem_max = 268435456
>>>> net.ipv4.tcp_mem = 190611 254150 381222
>>>> net.ipv4.tcp_rmem = 4096 131072 6291456
>>>> net.ipv4.tcp_wmem = 4096 16384 4194304
>>>> net.ipv4.udp_mem = 762450 1524900 3049800
>>>> net.ipv4.udp_rmem_min = 33554432
>>>> net.ipv4.udp_wmem_min = 33554432
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Tuesday, November 15, 2022 2:30 PM
>>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>> <david@lang.hm>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> what does the pstats output look like when it's dropping messages?
>>>> (give a couple cycles please)
>>>>
>>>> did you try to eliminate the action queue for /var/log/secure?
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>>
>>>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>>>> <david@lang.hm>
>>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Building on this -
>>>>>
>>>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>>>> In:imdup spikes to ~10%
>>>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>>>> usage.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Redbourne,Michael via rsyslog
>>>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>>> <david@lang.hm>
>>>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>>>> Udp:
>>>>> 5820820 packets received
>>>>> 1504 packets to unknown port received.
>>>>> 798900 packet receive errors
>>>>> 3338814 packets sent
>>>>> 798900 receive buffer errors
>>>>> 0 send buffer errors
>>>>>
>>>>> I have doubled the values in net.ipv4.udp_mem.
>>>>>
>>>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>>>
>>>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>>>> "<subnet>" stop
>>>>>
>>>>> Example Checkpoint Log:
>>>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>>>> dvc=<dvc_ip_addr>
>>>>>
>>>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>>>
>>>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Rainer Gerhards via rsyslog
>>>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>>>> <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>>>
>>>>> Rainer
>>>>>
>>>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>>>
>>>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>>>> input (usually I run into bottlenecks in processing the messages
>>>>>> long before I have problems accepting them)
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>>>
>>>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>>>
>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>>>>> 2F
>>>>>>> w
>>>>>>> w
>>>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.h
>>>>>>> tm
>>>>>>> l
>>>>>>> &
>>>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc
>>>>>>> 61
>>>>>>> 6
>>>>>>> 2
>>>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>>>>> C0
>>>>>>> %
>>>>>>> 7
>>>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
>>>>>>> LC
>>>>>>> J
>>>>>>> Q
>>>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>>>> da
>>>>>>> t
>>>>>>> a
>>>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>>>
>>>>>>> Rainer
>>>>>>>
>>>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>>>
>>>>>>>> Some additional comments on the config
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> These action queue configs probably don't do what you intend
>>>>>>>> them to do
>>>>>>>>
>>>>>>>> the first thing is that they only affect the next action, which
>>>>>>>> is
>>>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>>>> write these logs out. That will create a HUGE amount of
>>>>>>>> contention for the queue lock and under load you should see it
>>>>>>>> maxing out quite quickly
>>>>>>>>
>>>>>>>> what is it that you are attempting to do here?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Performance Tuning #
>>>>>>>> $ActionQueueWorkerThreads 2000
>>>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>>>> 1000000 $ActionQueueDiscardMark 800000
>>>>>>>> $ActionQueueHighWaterMark
>>>>>>>> 600000
>>>>>>>>
>>>>>>>> #### RULES ####
>>>>>>>> # Log all kernel messages to the console.
>>>>>>>> # Logging much else clutters up the screen.
>>>>>>>> #kern.* /dev/console
>>>>>>>>
>>>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>>>> authentication messages!
>>>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>>>
>>>>>>>> # The authpriv file has restricted access.
>>>>>>>> authpriv.* /var/log/secure
>>>>>>>>
>>>>>>>>
>>>>>>>> since the queue only applied to the next action with this
>>>>>>>> config, everything below this is operating from the main queue
>>>>>>>> again as if there was no action queue configuration
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Log all the mail messages in one place.
>>>>>>>> mail.* -/var/log/maillog
>>>>>>>>
>>>>>>>> # Log cron stuff
>>>>>>>> cron.* /var/log/cron
>>>>>>>>
>>>>>>>> # Everybody gets emergency messages
>>>>>>>> *.emerg :omusrmsg:*
>>>>>>>>
>>>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>>>> uucp,news.crit /var/log/spooler
>>>>>>>>
>>>>>>>> # Save boot messages also to boot.log
>>>>>>>> # local7.* /var/syslog/boot.log
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ereregex is a fairly expensive filter to apply, it's much
>>>>>>>> better to figure out a non-regex approach to filtering these.
>>>>>>>> Can you post some examples of what you are trying to filter?
>>>>>>>> mmnormalize to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>>>
>>>>>>>>
>>>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>>>> # [Firewall Log Filtering] #
>>>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)"
>>>>>>>> stop
>>>>>>>>
>>>>>>>>
>>>>>>>> check the messages to see where CEF: and ASA- are in the
>>>>>>>> message, can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>>>
>>>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>>>> then @127.0.0.1:25224 & stop
>>>>>>>>
>>>>>>>>
>>>>>>>> combining multiple filters into one action, or having the
>>>>>>>> filters call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>>>
>>>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>>>
>>>>>>>> local0.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local1.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local2.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local3.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local4.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local5.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local6.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> local7.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> auth.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> authpriv.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> daemon.info @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> syslog.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>>>> user.* @127.0.0.1:25224
>>>>>>>> & stop
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F
>>>>>>>> %2
>>>>>>>> F
>>>>>>>> l
>>>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C0
>>>>>>>> 1%
>>>>>>>> 7
>>>>>>>> C
>>>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08d
>>>>>>>> ac
>>>>>>>> 6
>>>>>>>> e
>>>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297
>>>>>>>> 03
>>>>>>>> 1
>>>>>>>> 5
>>>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
>>>>>>>> LC
>>>>>>>> J
>>>>>>>> B
>>>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2F
>>>>>>>> vQ
>>>>>>>> y
>>>>>>>> n
>>>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%
>>>>>>>> 2F
>>>>>>>> w
>>>>>>>> w
>>>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmic
>>>>>>>> ha
>>>>>>>> e
>>>>>>>> l
>>>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e978
>>>>>>>> 4e
>>>>>>>> %
>>>>>>>> 7
>>>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%
>>>>>>>> 7C
>>>>>>>> U
>>>>>>>> n
>>>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
>>>>>>>> 6I
>>>>>>>> k
>>>>>>>> 1
>>>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gY
>>>>>>>> B6
>>>>>>>> I
>>>>>>>> o
>>>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>>>> What's up with rsyslog? Follow
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F
>>>>>>>> %2
>>>>>>>> F
>>>>>>>> t
>>>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40b
>>>>>>>> ul
>>>>>>>> l
>>>>>>>> e
>>>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea4
>>>>>>>> 11
>>>>>>>> b
>>>>>>>> b
>>>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbG
>>>>>>>> Zs
>>>>>>>> b
>>>>>>>> 3
>>>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>>>>>>> n0
>>>>>>>> %
>>>>>>>> 3
>>>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZ
>>>>>>>> V3
>>>>>>>> O
>>>>>>>> Y
>>>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing
>>>>>>>> list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> li
>>>>> s
>>>>> t
>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmi
>>>>> ch
>>>>> a
>>>>> e
>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35
>>>>> %7
>>>>> C
>>>>> 9
>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnk
>>>>> no
>>>>> w
>>>>> n
>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
>>>>> iL
>>>>> C
>>>>> J
>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlh
>>>>> fZ
>>>>> %
>>>>> 2
>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf
>>>>> 44
>>>>> e
>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1
>>>>> %7
>>>>> C
>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
>>>>> iL
>>>>> C
>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>> da
>>>>> t
>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=
>>>>> 0
>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.r
>>>>> ed
>>>>> b
>>>>> o
>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63
>>>>> d1
>>>>> 3
>>>>> 8
>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7C
>>>>> TW
>>>>> F
>>>>> p
>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>>>> I6
>>>>> M
>>>>> n
>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2
>>>>> B3
>>>>> w
>>>>> N
>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> tw
>>>>> i
>>>>> t
>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletp
>>>>> ro
>>>>> o
>>>>> f
>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458
>>>>> b7
>>>>> e
>>>>> 2
>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>>>>> oi
>>>>> M
>>>>> C
>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>>> %7
>>>>> C
>>>>> %
>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;
>>>>> re
>>>>> s
>>>>> e
>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>> ________________________________________
>>>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>>>
>>>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>>>> ________________________________________
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> li
>>>>> s
>>>>> t
>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmi
>>>>> ch
>>>>> a
>>>>> e
>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35
>>>>> %7
>>>>> C
>>>>> 9
>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnk
>>>>> no
>>>>> w
>>>>> n
>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
>>>>> iL
>>>>> C
>>>>> J
>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlh
>>>>> fZ
>>>>> %
>>>>> 2
>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf
>>>>> 44
>>>>> e
>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1
>>>>> %7
>>>>> C
>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
>>>>> iL
>>>>> C
>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>> da
>>>>> t
>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=
>>>>> 0
>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.r
>>>>> ed
>>>>> b
>>>>> o
>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63
>>>>> d1
>>>>> 3
>>>>> 8
>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7C
>>>>> TW
>>>>> F
>>>>> p
>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>>>> I6
>>>>> M
>>>>> n
>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2
>>>>> B3
>>>>> w
>>>>> N
>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> tw
>>>>> i
>>>>> t
>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletp
>>>>> ro
>>>>> o
>>>>> f
>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458
>>>>> b7
>>>>> e
>>>>> 2
>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>>>>> oi
>>>>> M
>>>>> C
>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>>> %7
>>>>> C
>>>>> %
>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;
>>>>> re
>>>>> s
>>>>> e
>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
ok, if you are losing packets over loopback to the oms agent, that's a problem
with it not with rsyslog.

David Lang

On Tue, 15 Nov 2022, Redbourne,Michael wrote:

> I hadn't even noticed that... OK, time to go speak with someone else... Port 25224 (UDP) is the OMS Agent's Syslog. Port 25226 (TCP) is for CEF events. With the events being dropped at a different application (OMS Agent for Linux), this isn't something you can likely assist with. I'll submit a support case with Microsoft. The setup on that is baseline...
>
> Back when I posted the original /etc/rsyslog.d/security-config-omsagent.conf file, there were several daemons pointed to 25224, in addition to other log sources (Infoblox).
>
> 0100007F -> 1.0.0.127, which I'm assuming it probably meant 127.0.0.1.
>
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, November 15, 2022 6:06 PM
> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>
> ahh, I thought that was a rsyslog thread that could be maxing out a core.
>
> my logging strategy is that everything should get sent to the central syslog server, and only there should it get thrown away (and before you throw it away, consider counting it, the number of times that an uninteresting thing happens can be an interesting thing to track)
>
> per the impstats output you included, you are hitting up to ~240k messages/30 seconds, or around 8k messages/sec. That should not be challenging to rsyslog to receive (I've done over 300k messages/sec), we just need to watch the cost of processing them.
>
> you don't show any times when the queue actually filled up, so messages should not be dropped because of that.
>
> with the new configs, are you sure you are still dropping syslog messages? the proc stats you show below show drops on port hex 6288 (25224 decimal), not on hex 202 (514 decimal)
>
> David Lang
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
>> Date: Tue, 15 Nov 2022 21:55:48 +0000
>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and might help clear up what the actual issue is.
>>
>> If you're referencing this:
>>
>>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>
>> It's owned by a completely different process (and package) from rsyslog. It's the Microsoft OMS Agent for Linux Package, available here: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FOMS-Agent-for-Linux&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7C8706a226fa1240396fee08dac7558349%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041467339516478%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=7xWiZ3%2FskQ3E8kghTzYd%2FAwZ%2BVddR3HJPK%2FMKgDWmxg%3D&amp;reserved=0. Unfortunately, it's not something I can get rid of... yet. I'm waiting on Microsoft to support the newer Azure ARC agent for new builds of CentOS/RHEL 9.
>>
>> Cheers,
>> Mike
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, November 15, 2022 5:49 PM
>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
>>
>> As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently.
>>
>> Rsyslog is very efficient at processing messages, but regex and
>> contains are about the most expensive tests that you can do
>>
>> If you really do need multiple threads to process the log messages, you need to set the queue parameters on the main queue with the global() stanza.
>>
>> I'm not convinced that we've caught the full problem during our sampling yet.
>>
>> As I said, it's probably a good idea to put a queue on the TCP forwarding that you have configured to isolate any issues there from the general processing.
>>
>>
>> @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?
>>
>> David Lang
>>
>>
>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>
>>> Date: Tue, 15 Nov 2022 21:39:45 +0000
>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
>>>
>>> If I'm still havcing issues after that, I'll respond back.
>>>
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, November 15, 2022 4:19 PM
>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users
>>> <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>> Events
>>>
>>> using the new action() syntax, you can name the actions so they aren't just numbered.
>>>
>>> starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
>>>
>>> that could be the tcp forwarding of messages, which can stall if the
>>> device you are forwarding to can't keep up (that is a good place to
>>> have an action queue, but please use the action() syntax so that it's
>>> clear what the queue is doing)
>>>
>>> David Lang
>>>
>>>
>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>
>>>> Date: Tue, 15 Nov 2022 20:01:52 +0000
>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>> To: David Lang <david@lang.hm>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15
>>>> 00:22:18
>>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>>> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
>>>> origin=core.action processed=1628 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
>>>> origin=core.action processed=66 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
>>>> origin=core.action processed=37449 failed=37448 suspended=1
>>>> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
>>>> origin=core.action processed=4582 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
>>>> origin=core.action processed=5039 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
>>>> origin=core.action processed=862 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
>>>> origin=core.action processed=397 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
>>>> origin=core.action processed=50 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
>>>> origin=core.action processed=2 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
>>>> origin=core.action processed=13 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
>>>> origin=core.action processed=766 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
>>>> origin=core.action processed=11058 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
>>>> origin=core.action processed=1099 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
>>>> origin=core.action processed=12 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>>> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
>>>> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
>>>> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
>>>> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
>>>> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov
>>>> 15
>>>> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161
>>>> full=0
>>>> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
>>>> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
>>>> msgs.received=24227
>>>>
>>>>
>>>> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15
>>>> 08:41:43
>>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>>> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
>>>> origin=core.action processed=1728 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
>>>> origin=core.action processed=47 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
>>>> origin=core.action processed=35778 failed=35777 suspended=1
>>>> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
>>>> origin=core.action processed=9249 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
>>>> origin=core.action processed=2296 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
>>>> origin=core.action processed=1009 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
>>>> origin=core.action processed=558 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
>>>> origin=core.action processed=40 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
>>>> origin=core.action processed=2 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
>>>> origin=core.action processed=13 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
>>>> origin=core.action processed=720 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
>>>> origin=core.action processed=16593 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
>>>> origin=core.action processed=359 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
>>>> origin=core.action processed=2 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>>> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
>>>> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
>>>> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
>>>> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
>>>> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov
>>>> 15
>>>> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998
>>>> full=0
>>>> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
>>>> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
>>>> msgs.received=31257
>>>>
>>>>
>>>> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15
>>>> 08:42:13
>>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>>> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
>>>> origin=core.action processed=2181 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
>>>> origin=core.action processed=57 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
>>>> origin=core.action processed=20793 failed=137 suspended=0
>>>> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
>>>> origin=core.action processed=7599 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
>>>> origin=core.action processed=5540 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
>>>> origin=core.action processed=1120 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
>>>> origin=core.action processed=157807 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
>>>> origin=core.action processed=50 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
>>>> origin=core.action processed=3 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
>>>> origin=core.action processed=293 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
>>>> origin=core.action processed=20243 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
>>>> origin=core.action processed=866 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
>>>> origin=core.action processed=10 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>>> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
>>>> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
>>>> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
>>>> origin=impstats utime=5417168 stime=3346362 maxrss=16800
>>>> minflt=11196
>>>> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
>>>> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
>>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
>>>> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
>>>> called.recvmsg=0 msgs.received=194836
>>>>
>>>>
>>>> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15
>>>> 10:15:04
>>>> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
>>>> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
>>>> origin=core.action processed=2009 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
>>>> origin=core.action processed=151 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
>>>> origin=core.action processed=52035 failed=52034 suspended=1
>>>> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
>>>> origin=core.action processed=19442 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
>>>> origin=core.action processed=4480 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
>>>> origin=core.action processed=1125 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
>>>> origin=core.action processed=415 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
>>>> origin=core.action processed=144 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
>>>> origin=core.action processed=2 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
>>>> origin=core.action processed=13 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
>>>> origin=core.action processed=745 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
>>>> origin=core.action processed=23626 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
>>>> origin=core.action processed=682 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
>>>> origin=core.action processed=45 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
>>>> origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>>> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
>>>> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
>>>> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
>>>> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
>>>> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov
>>>> 15
>>>> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235
>>>> full=0
>>>> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
>>>> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
>>>> msgs.received=51446 -----Original Message-----
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Tuesday, November 15, 2022 3:43 PM
>>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>> Cc: David Lang <david@lang.hm>; rsyslog-users
>>>> <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>> Events
>>>>
>>>> you have the impstats module loaded in your config and writing stats out, please post the output of this.
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>>
>>>>> Date: Tue, 15 Nov 2022 19:38:27 +0000
>>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> I'm still not understanding what you mean by pstats - it's not a
>>>>> package or command available to me. It's apart of Unix from what I
>>>>> can tell. I've placed below the unparsed information form
>>>>> /proc/net/netstat and /proc/net/udp
>>>>>
>>>>> /proc/net/netstat
>>>>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed
>>>>> EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps
>>>>> LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive
>>>>> PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost
>>>>> ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks
>>>>> TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPSACKReorder
>>>>> TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo
>>>>> TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures
>>>>> TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts
>>>>> TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
>>>>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
>>>>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
>>>>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
>>>>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
>>>>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
>>>>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound
>>>>> TCPMD5Unexpected TCPMD5Failure TCPSackShifted T
>>>> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
>>>> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
>>>> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
>>>> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
>>>> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
>>>> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
>>>> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
>>>> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
>>>> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
>>>> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
>>>> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
>>>> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
>>>> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
>>>> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
>>>> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
>>>> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
>>>> TcpDuplicateDataRehash TCPDSACKRe
>>> c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess
>>> TCPMigrateReqFailure
>>>>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402
>>>>> 5231
>>>>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0
>>>>> 0
>>>>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0
>>>>> 0
>>>>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24
>>>>> 0
>>>>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
>>>>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
>>>>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
>>>>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors
>>>>> InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
>>>>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0
>>>>> 0
>>>>>
>>>>> /proc/net/udp
>>>>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
>>>>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
>>>>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
>>>>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
>>>>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
>>>>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
>>>>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
>>>>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
>>>>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
>>>>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
>>>>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
>>>>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
>>>>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
>>>>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
>>>>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
>>>>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
>>>>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
>>>>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
>>>>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
>>>>>
>>>>> The ActionQueue for /var/log/secure has been commented out from the configuration.
>>>>>
>>>>> Dropwatch (Cycle 1):
>>>>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
>>>>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
>>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
>>>>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
>>>>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
>>>>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
>>>>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
>>>>>
>>>>> Dropwatch (Cycle 2):
>>>>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
>>>>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
>>>>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
>>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
>>>>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
>>>>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
>>>>>
>>>>> Top -H
>>>>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
>>>>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
>>>>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
>>>>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
>>>>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
>>>>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
>>>>>
>>>>> /etc/sysctl.conf
>>>>> net.core.rmem_default = 33554432
>>>>> net.core.rmem_max = 268435456
>>>>> net.core.wmem_default = 33554432
>>>>> net.core.wmem_max = 268435456
>>>>> net.ipv4.tcp_mem = 190611 254150 381222
>>>>> net.ipv4.tcp_rmem = 4096 131072 6291456
>>>>> net.ipv4.tcp_wmem = 4096 16384 4194304
>>>>> net.ipv4.udp_mem = 762450 1524900 3049800
>>>>> net.ipv4.udp_rmem_min = 33554432
>>>>> net.ipv4.udp_wmem_min = 33554432
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: David Lang <david@lang.hm>
>>>>> Sent: Tuesday, November 15, 2022 2:30 PM
>>>>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>>> <david@lang.hm>
>>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>> Events
>>>>>
>>>>> what does the pstats output look like when it's dropping messages?
>>>>> (give a couple cycles please)
>>>>>
>>>>> did you try to eliminate the action queue for /var/log/secure?
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>>>>>
>>>>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
>>>>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
>>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
>>>>>> <david@lang.hm>
>>>>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>>> Events
>>>>>>
>>>>>> Building on this -
>>>>>>
>>>>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
>>>>>> In:imdup spikes to ~10%
>>>>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
>>>>>> usage.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>> Redbourne,Michael via rsyslog
>>>>>> Sent: Tuesday, November 15, 2022 8:42 AM
>>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
>>>>>> <david@lang.hm>
>>>>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
>>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>>> Events
>>>>>>
>>>>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
>>>>>> Udp:
>>>>>> 5820820 packets received
>>>>>> 1504 packets to unknown port received.
>>>>>> 798900 packet receive errors
>>>>>> 3338814 packets sent
>>>>>> 798900 receive buffer errors
>>>>>> 0 send buffer errors
>>>>>>
>>>>>> I have doubled the values in net.ipv4.udp_mem.
>>>>>>
>>>>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
>>>>>>
>>>>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
>>>>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
>>>>>> "<subnet>" stop
>>>>>>
>>>>>> Example Checkpoint Log:
>>>>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
>>>>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
>>>>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
>>>>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
>>>>>> priority=8 deviceSeverity=Very-High rt=1599552617058
>>>>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
>>>>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
>>>>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
>>>>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
>>>>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
>>>>>> dvc=<dvc_ip_addr>
>>>>>>
>>>>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
>>>>>>
>>>>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>> Rainer Gerhards via rsyslog
>>>>>> Sent: Tuesday, November 15, 2022 5:11 AM
>>>>>> To: David Lang <david@lang.hm>
>>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
>>>>>> <rsyslog@lists.adiscon.com>
>>>>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
>>>>>> Events
>>>>>>
>>>>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
>>>>>>>
>>>>>>> I haven't needed to do that to handle 300k messages/sec on UDP
>>>>>>> input (usually I run into bottlenecks in processing the messages
>>>>>>> long before I have problems accepting them)
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>>>>>>>
>>>>>>>> let me add: look into setting imudp to realtime priority. Doc:
>>>>>>>>
>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>>>>>> 2F
>>>>>>>> w
>>>>>>>> w
>>>>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.h
>>>>>>>> tm
>>>>>>>> l
>>>>>>>> &
>>>>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc
>>>>>>>> 61
>>>>>>>> 6
>>>>>>>> 2
>>>>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
>>>>>>>> C0
>>>>>>>> %
>>>>>>>> 7
>>>>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
>>>>>>>> LC
>>>>>>>> J
>>>>>>>> Q
>>>>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>>>>> da
>>>>>>>> t
>>>>>>>> a
>>>>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
>>>>>>>>
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
>>>>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
>>>>>>>>>
>>>>>>>>> Some additional comments on the config
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> These action queue configs probably don't do what you intend
>>>>>>>>> them to do
>>>>>>>>>
>>>>>>>>> the first thing is that they only affect the next action, which
>>>>>>>>> is
>>>>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
>>>>>>>>> write these logs out. That will create a HUGE amount of
>>>>>>>>> contention for the queue lock and under load you should see it
>>>>>>>>> maxing out quite quickly
>>>>>>>>>
>>>>>>>>> what is it that you are attempting to do here?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Performance Tuning #
>>>>>>>>> $ActionQueueWorkerThreads 2000
>>>>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
>>>>>>>>> 1000000 $ActionQueueDiscardMark 800000
>>>>>>>>> $ActionQueueHighWaterMark
>>>>>>>>> 600000
>>>>>>>>>
>>>>>>>>> #### RULES ####
>>>>>>>>> # Log all kernel messages to the console.
>>>>>>>>> # Logging much else clutters up the screen.
>>>>>>>>> #kern.* /dev/console
>>>>>>>>>
>>>>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
>>>>>>>>> authentication messages!
>>>>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
>>>>>>>>>
>>>>>>>>> # The authpriv file has restricted access.
>>>>>>>>> authpriv.* /var/log/secure
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> since the queue only applied to the next action with this
>>>>>>>>> config, everything below this is operating from the main queue
>>>>>>>>> again as if there was no action queue configuration
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Log all the mail messages in one place.
>>>>>>>>> mail.* -/var/log/maillog
>>>>>>>>>
>>>>>>>>> # Log cron stuff
>>>>>>>>> cron.* /var/log/cron
>>>>>>>>>
>>>>>>>>> # Everybody gets emergency messages
>>>>>>>>> *.emerg :omusrmsg:*
>>>>>>>>>
>>>>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>>>>> uucp,news.crit /var/log/spooler
>>>>>>>>>
>>>>>>>>> # Save boot messages also to boot.log
>>>>>>>>> # local7.* /var/syslog/boot.log
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ereregex is a fairly expensive filter to apply, it's much
>>>>>>>>> better to figure out a non-regex approach to filtering these.
>>>>>>>>> Can you post some examples of what you are trying to filter?
>>>>>>>>> mmnormalize to parse the logs and then make decisions on the parsed results id probably much faster.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
>>>>>>>>> # [Firewall Log Filtering] #
>>>>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
>>>>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
>>>>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
>>>>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
>>>>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
>>>>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
>>>>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)"
>>>>>>>>> stop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> check the messages to see where CEF: and ASA- are in the
>>>>>>>>> message, can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
>>>>>>>>> instead of 'contains'?, again mmnormalize may be much faster
>>>>>>>>>
>>>>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
>>>>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
>>>>>>>>> then @127.0.0.1:25224 & stop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> combining multiple filters into one action, or having the
>>>>>>>>> filters call a ruleset can be far more efficient than all of them writing things out independently.
>>>>>>>>>
>>>>>>>>> the if..then filter structure lets you easily combine filters
>>>>>>>>>
>>>>>>>>> local0.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local1.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local2.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local3.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local4.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local5.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local6.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> local7.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> auth.* @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> authpriv.* @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> daemon.info @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> syslog.* @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
>>>>>>>>> user.* @127.0.0.1:25224
>>>>>>>>> & stop
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F
>>>>>>>>> %2
>>>>>>>>> F
>>>>>>>>> l
>>>>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C0
>>>>>>>>> 1%
>>>>>>>>> 7
>>>>>>>>> C
>>>>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08d
>>>>>>>>> ac
>>>>>>>>> 6
>>>>>>>>> e
>>>>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297
>>>>>>>>> 03
>>>>>>>>> 1
>>>>>>>>> 5
>>>>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
>>>>>>>>> LC
>>>>>>>>> J
>>>>>>>>> B
>>>>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2F
>>>>>>>>> vQ
>>>>>>>>> y
>>>>>>>>> n
>>>>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
>>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%
>>>>>>>>> 2F
>>>>>>>>> w
>>>>>>>>> w
>>>>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmic
>>>>>>>>> ha
>>>>>>>>> e
>>>>>>>>> l
>>>>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e978
>>>>>>>>> 4e
>>>>>>>>> %
>>>>>>>>> 7
>>>>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%
>>>>>>>>> 7C
>>>>>>>>> U
>>>>>>>>> n
>>>>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
>>>>>>>>> 6I
>>>>>>>>> k
>>>>>>>>> 1
>>>>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gY
>>>>>>>>> B6
>>>>>>>>> I
>>>>>>>>> o
>>>>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F
>>>>>>>>> %2
>>>>>>>>> F
>>>>>>>>> t
>>>>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40b
>>>>>>>>> ul
>>>>>>>>> l
>>>>>>>>> e
>>>>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea4
>>>>>>>>> 11
>>>>>>>>> b
>>>>>>>>> b
>>>>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbG
>>>>>>>>> Zs
>>>>>>>>> b
>>>>>>>>> 3
>>>>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
>>>>>>>>> n0
>>>>>>>>> %
>>>>>>>>> 3
>>>>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZ
>>>>>>>>> V3
>>>>>>>>> O
>>>>>>>>> Y
>>>>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing
>>>>>>>>> list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> li
>>>>>> s
>>>>>> t
>>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmi
>>>>>> ch
>>>>>> a
>>>>>> e
>>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35
>>>>>> %7
>>>>>> C
>>>>>> 9
>>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnk
>>>>>> no
>>>>>> w
>>>>>> n
>>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
>>>>>> iL
>>>>>> C
>>>>>> J
>>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlh
>>>>>> fZ
>>>>>> %
>>>>>> 2
>>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf
>>>>>> 44
>>>>>> e
>>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1
>>>>>> %7
>>>>>> C
>>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
>>>>>> iL
>>>>>> C
>>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>>> da
>>>>>> t
>>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=
>>>>>> 0
>>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.r
>>>>>> ed
>>>>>> b
>>>>>> o
>>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63
>>>>>> d1
>>>>>> 3
>>>>>> 8
>>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7C
>>>>>> TW
>>>>>> F
>>>>>> p
>>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>>>>> I6
>>>>>> M
>>>>>> n
>>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2
>>>>>> B3
>>>>>> w
>>>>>> N
>>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> tw
>>>>>> i
>>>>>> t
>>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletp
>>>>>> ro
>>>>>> o
>>>>>> f
>>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458
>>>>>> b7
>>>>>> e
>>>>>> 2
>>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>>>>>> oi
>>>>>> M
>>>>>> C
>>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>>>> %7
>>>>>> C
>>>>>> %
>>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;
>>>>>> re
>>>>>> s
>>>>>> e
>>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>> ________________________________________
>>>>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>>>>
>>>>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
>>>>>> ________________________________________
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> li
>>>>>> s
>>>>>> t
>>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmi
>>>>>> ch
>>>>>> a
>>>>>> e
>>>>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35
>>>>>> %7
>>>>>> C
>>>>>> 9
>>>>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnk
>>>>>> no
>>>>>> w
>>>>>> n
>>>>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
>>>>>> iL
>>>>>> C
>>>>>> J
>>>>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlh
>>>>>> fZ
>>>>>> %
>>>>>> 2
>>>>>> BRRguRnpir9RsV8%3D&amp;reserved=0
>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
>>>>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf
>>>>>> 44
>>>>>> e
>>>>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1
>>>>>> %7
>>>>>> C
>>>>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
>>>>>> iL
>>>>>> C
>>>>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;s
>>>>>> da
>>>>>> t
>>>>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=
>>>>>> 0
>>>>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.r
>>>>>> ed
>>>>>> b
>>>>>> o
>>>>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63
>>>>>> d1
>>>>>> 3
>>>>>> 8
>>>>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7C
>>>>>> TW
>>>>>> F
>>>>>> p
>>>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>>>>> I6
>>>>>> M
>>>>>> n
>>>>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2
>>>>>> B3
>>>>>> w
>>>>>> N
>>>>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> tw
>>>>>> i
>>>>>> t
>>>>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletp
>>>>>> ro
>>>>>> o
>>>>>> f
>>>>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458
>>>>>> b7
>>>>>> e
>>>>>> 2
>>>>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>>>>>> oi
>>>>>> M
>>>>>> C
>>>>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>>>> %7
>>>>>> C
>>>>>> %
>>>>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;
>>>>>> re
>>>>>> s
>>>>>> e
>>>>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>
>>>>>
>>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
> @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?

Not rsyslog, looks like fluentd:

https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_syslog.rb

Rainer

>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
> > Date: Tue, 15 Nov 2022 21:39:45 +0000
> > From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> > To: David Lang <david@lang.hm>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >
> > I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
> >
> > If I'm still havcing issues after that, I'll respond back.
> >
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Tuesday, November 15, 2022 4:19 PM
> > To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> > Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >
> > using the new action() syntax, you can name the actions so they aren't just numbered.
> >
> > starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
> >
> > that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)
> >
> > David Lang
> >
> >
> > On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >
> >> Date: Tue, 15 Nov 2022 20:01:52 +0000
> >> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >> To: David Lang <david@lang.hm>
> >> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >>
> >> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
> >> origin=core.action processed=1628 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
> >> origin=core.action processed=66 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
> >> origin=core.action processed=37449 failed=37448 suspended=1
> >> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
> >> origin=core.action processed=4582 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
> >> origin=core.action processed=5039 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
> >> origin=core.action processed=862 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
> >> origin=core.action processed=397 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
> >> origin=core.action processed=50 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
> >> origin=core.action processed=766 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
> >> origin=core.action processed=11058 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
> >> origin=core.action processed=1099 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
> >> origin=core.action processed=12 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
> >> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
> >> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
> >> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
> >> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
> >> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
> >> msgs.received=24227
> >>
> >>
> >> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
> >> origin=core.action processed=1728 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
> >> origin=core.action processed=47 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
> >> origin=core.action processed=35778 failed=35777 suspended=1
> >> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
> >> origin=core.action processed=9249 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
> >> origin=core.action processed=2296 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
> >> origin=core.action processed=1009 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
> >> origin=core.action processed=558 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
> >> origin=core.action processed=40 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
> >> origin=core.action processed=720 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
> >> origin=core.action processed=16593 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
> >> origin=core.action processed=359 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
> >> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
> >> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
> >> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
> >> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
> >> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
> >> msgs.received=31257
> >>
> >>
> >> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
> >> origin=core.action processed=2181 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
> >> origin=core.action processed=57 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
> >> origin=core.action processed=20793 failed=137 suspended=0
> >> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
> >> origin=core.action processed=7599 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
> >> origin=core.action processed=5540 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
> >> origin=core.action processed=1120 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
> >> origin=core.action processed=157807 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
> >> origin=core.action processed=50 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
> >> origin=core.action processed=3 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
> >> origin=core.action processed=293 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
> >> origin=core.action processed=20243 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
> >> origin=core.action processed=866 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
> >> origin=core.action processed=10 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
> >> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
> >> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
> >> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
> >> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
> >> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
> >> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
> >> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
> >> called.recvmsg=0 msgs.received=194836
> >>
> >>
> >> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
> >> origin=core.action processed=2009 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
> >> origin=core.action processed=151 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
> >> origin=core.action processed=52035 failed=52034 suspended=1
> >> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
> >> origin=core.action processed=19442 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
> >> origin=core.action processed=4480 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
> >> origin=core.action processed=1125 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
> >> origin=core.action processed=415 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
> >> origin=core.action processed=144 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
> >> origin=core.action processed=745 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
> >> origin=core.action processed=23626 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
> >> origin=core.action processed=682 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
> >> origin=core.action processed=45 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
> >> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
> >> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
> >> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
> >> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
> >> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
> >> msgs.received=51446 -----Original Message-----
> >> From: David Lang <david@lang.hm>
> >> Sent: Tuesday, November 15, 2022 3:43 PM
> >> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >> Cc: David Lang <david@lang.hm>; rsyslog-users
> >> <rsyslog@lists.adiscon.com>
> >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >>
> >> you have the impstats module loaded in your config and writing stats out, please post the output of this.
> >>
> >> David Lang
> >>
> >> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >>
> >>> Date: Tue, 15 Nov 2022 19:38:27 +0000
> >>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >>> To: David Lang <david@lang.hm>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>> Events
> >>>
> >>> I'm still not understanding what you mean by pstats - it's not a
> >>> package or command available to me. It's apart of Unix from what I
> >>> can tell. I've placed below the unparsed information form
> >>> /proc/net/netstat and /proc/net/udp
> >>>
> >>> /proc/net/netstat
> >>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
> >>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
> >>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
> >>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
> >>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
> >>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
> >>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
> >>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
> >>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
> >>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
> >>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
> >>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
> >>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
> >>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
> >>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
> >>> TCPMD5Failure TCPSackShifted T
> >> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
> >> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
> >> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
> >> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
> >> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
> >> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
> >> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
> >> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
> >> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
> >> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
> >> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
> >> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
> >> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
> >> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
> >> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
> >> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
> >> TcpDuplicateDataRehash TCPDSACKRe
> > c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
> >>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
> >>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
> >>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
> >>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
> >>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
> >>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
> >>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
> >>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
> >>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
> >>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
> >>>
> >>> /proc/net/udp
> >>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
> >>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
> >>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
> >>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
> >>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
> >>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
> >>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
> >>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
> >>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
> >>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
> >>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
> >>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
> >>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
> >>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
> >>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
> >>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
> >>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
> >>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
> >>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
> >>>
> >>> The ActionQueue for /var/log/secure has been commented out from the configuration.
> >>>
> >>> Dropwatch (Cycle 1):
> >>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
> >>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> >>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
> >>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>>
> >>> Dropwatch (Cycle 2):
> >>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
> >>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
> >>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> >>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> >>>
> >>> Top -H
> >>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
> >>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
> >>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
> >>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
> >>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
> >>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
> >>>
> >>> /etc/sysctl.conf
> >>> net.core.rmem_default = 33554432
> >>> net.core.rmem_max = 268435456
> >>> net.core.wmem_default = 33554432
> >>> net.core.wmem_max = 268435456
> >>> net.ipv4.tcp_mem = 190611 254150 381222
> >>> net.ipv4.tcp_rmem = 4096 131072 6291456
> >>> net.ipv4.tcp_wmem = 4096 16384 4194304
> >>> net.ipv4.udp_mem = 762450 1524900 3049800
> >>> net.ipv4.udp_rmem_min = 33554432
> >>> net.ipv4.udp_wmem_min = 33554432
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: David Lang <david@lang.hm>
> >>> Sent: Tuesday, November 15, 2022 2:30 PM
> >>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> >>> <david@lang.hm>
> >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>> Events
> >>>
> >>> what does the pstats output look like when it's dropping messages?
> >>> (give a couple cycles please)
> >>>
> >>> did you try to eliminate the action queue for /var/log/secure?
> >>>
> >>> David Lang
> >>>
> >>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >>>
> >>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
> >>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
> >>>> <david@lang.hm>
> >>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Building on this -
> >>>>
> >>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
> >>>> In:imdup spikes to ~10%
> >>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
> >>>> usage.
> >>>>
> >>>> -----Original Message-----
> >>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> >>>> Redbourne,Michael via rsyslog
> >>>> Sent: Tuesday, November 15, 2022 8:42 AM
> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> >>>> <david@lang.hm>
> >>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
> >>>> Udp:
> >>>> 5820820 packets received
> >>>> 1504 packets to unknown port received.
> >>>> 798900 packet receive errors
> >>>> 3338814 packets sent
> >>>> 798900 receive buffer errors
> >>>> 0 send buffer errors
> >>>>
> >>>> I have doubled the values in net.ipv4.udp_mem.
> >>>>
> >>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
> >>>>
> >>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
> >>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
> >>>> "<subnet>" stop
> >>>>
> >>>> Example Checkpoint Log:
> >>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
> >>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
> >>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
> >>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
> >>>> priority=8 deviceSeverity=Very-High rt=1599552617058
> >>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
> >>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
> >>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
> >>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
> >>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
> >>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
> >>>> dvc=<dvc_ip_addr>
> >>>>
> >>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
> >>>>
> >>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
> >>>>
> >>>> -----Original Message-----
> >>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> >>>> Rainer Gerhards via rsyslog
> >>>> Sent: Tuesday, November 15, 2022 5:11 AM
> >>>> To: David Lang <david@lang.hm>
> >>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
> >>>> <rsyslog@lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
> >>>>
> >>>> Rainer
> >>>>
> >>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribió:
> >>>>>
> >>>>> I haven't needed to do that to handle 300k messages/sec on UDP
> >>>>> input (usually I run into bottlenecks in processing the messages
> >>>>> long before I have problems accepting them)
> >>>>>
> >>>>> David Lang
> >>>>>
> >>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
> >>>>>
> >>>>>> let me add: look into setting imudp to realtime priority. Doc:
> >>>>>>
> >>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> >>>>>> w
> >>>>>> w
> >>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
> >>>>>> l
> >>>>>> &
> >>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
> >>>>>> 6
> >>>>>> 2
> >>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
> >>>>>> %
> >>>>>> 7
> >>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
> >>>>>> J
> >>>>>> Q
> >>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>>>> t
> >>>>>> a
> >>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
> >>>>>>
> >>>>>> Rainer
> >>>>>>
> >>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> >>>>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>
> >>>>>>> Some additional comments on the config
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> These action queue configs probably don't do what you intend them
> >>>>>>> to do
> >>>>>>>
> >>>>>>> the first thing is that they only affect the next action, which
> >>>>>>> is
> >>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
> >>>>>>> write these logs out. That will create a HUGE amount of
> >>>>>>> contention for the queue lock and under load you should see it
> >>>>>>> maxing out quite quickly
> >>>>>>>
> >>>>>>> what is it that you are attempting to do here?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> # Performance Tuning #
> >>>>>>> $ActionQueueWorkerThreads 2000
> >>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
> >>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
> >>>>>>> 600000
> >>>>>>>
> >>>>>>> #### RULES ####
> >>>>>>> # Log all kernel messages to the console.
> >>>>>>> # Logging much else clutters up the screen.
> >>>>>>> #kern.* /dev/console
> >>>>>>>
> >>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
> >>>>>>> authentication messages!
> >>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>>>>>>
> >>>>>>> # The authpriv file has restricted access.
> >>>>>>> authpriv.* /var/log/secure
> >>>>>>>
> >>>>>>>
> >>>>>>> since the queue only applied to the next action with this config,
> >>>>>>> everything below this is operating from the main queue again as
> >>>>>>> if there was no action queue configuration
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> # Log all the mail messages in one place.
> >>>>>>> mail.* -/var/log/maillog
> >>>>>>>
> >>>>>>> # Log cron stuff
> >>>>>>> cron.* /var/log/cron
> >>>>>>>
> >>>>>>> # Everybody gets emergency messages
> >>>>>>> *.emerg :omusrmsg:*
> >>>>>>>
> >>>>>>> # Save news errors of level crit and higher in a special file.
> >>>>>>> uucp,news.crit /var/log/spooler
> >>>>>>>
> >>>>>>> # Save boot messages also to boot.log
> >>>>>>> # local7.* /var/syslog/boot.log
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ereregex is a fairly expensive filter to apply, it's much better
> >>>>>>> to figure out a non-regex approach to filtering these. Can you
> >>>>>>> post some examples of what you are trying to filter? mmnormalize
> >>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
> >>>>>>>
> >>>>>>>
> >>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
> >>>>>>> # [Firewall Log Filtering] #
> >>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
> >>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
> >>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
> >>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
> >>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
> >>>>>>>
> >>>>>>>
> >>>>>>> check the messages to see where CEF: and ASA- are in the message,
> >>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> >>>>>>> instead of 'contains'?, again mmnormalize may be much faster
> >>>>>>>
> >>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
> >>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
> >>>>>>> then @127.0.0.1:25224 & stop
> >>>>>>>
> >>>>>>>
> >>>>>>> combining multiple filters into one action, or having the filters
> >>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
> >>>>>>>
> >>>>>>> the if..then filter structure lets you easily combine filters
> >>>>>>>
> >>>>>>> local0.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local1.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local2.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local3.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local4.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local5.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local6.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local7.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> auth.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> authpriv.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> daemon.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> syslog.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
> >>>>>>> user.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> >>>>>>> F
> >>>>>>> l
> >>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
> >>>>>>> 7
> >>>>>>> C
> >>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
> >>>>>>> 6
> >>>>>>> e
> >>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
> >>>>>>> 1
> >>>>>>> 5
> >>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
> >>>>>>> J
> >>>>>>> B
> >>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
> >>>>>>> y
> >>>>>>> n
> >>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> >>>>>>> w
> >>>>>>> w
> >>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
> >>>>>>> e
> >>>>>>> l
> >>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
> >>>>>>> %
> >>>>>>> 7
> >>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
> >>>>>>> U
> >>>>>>> n
> >>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> >>>>>>> k
> >>>>>>> 1
> >>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
> >>>>>>> I
> >>>>>>> o
> >>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
> >>>>>>> What's up with rsyslog? Follow
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> >>>>>>> F
> >>>>>>> t
> >>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
> >>>>>>> l
> >>>>>>> e
> >>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
> >>>>>>> b
> >>>>>>> b
> >>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
> >>>>>>> b
> >>>>>>> 3
> >>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> >>>>>>> %
> >>>>>>> 3
> >>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
> >>>>>>> O
> >>>>>>> Y
> >>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
> >>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> >>>> s
> >>>> t
> >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
> >>>> a
> >>>> e
> >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
> >>>> C
> >>>> 9
> >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
> >>>> w
> >>>> n
> >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>>> C
> >>>> J
> >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
> >>>> %
> >>>> 2
> >>>> BRRguRnpir9RsV8%3D&amp;reserved=0
> >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
> >>>> e
> >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
> >>>> C
> >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> >>>> C
> >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>> t
> >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
> >>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
> >>>> b
> >>>> o
> >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
> >>>> 3
> >>>> 8
> >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
> >>>> F
> >>>> p
> >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> >>>> M
> >>>> n
> >>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
> >>>> w
> >>>> N
> >>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
> >>>> i
> >>>> t
> >>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
> >>>> o
> >>>> f
> >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
> >>>> e
> >>>> 2
> >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >>>> M
> >>>> C
> >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> >>>> C
> >>>> %
> >>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
> >>>> s
> >>>> e
> >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>> ________________________________________
> >>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
> >>>>
> >>>> Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> >>>> ________________________________________
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> >>>> s
> >>>> t
> >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
> >>>> a
> >>>> e
> >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
> >>>> C
> >>>> 9
> >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
> >>>> w
> >>>> n
> >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>>> C
> >>>> J
> >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
> >>>> %
> >>>> 2
> >>>> BRRguRnpir9RsV8%3D&amp;reserved=0
> >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
> >>>> e
> >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
> >>>> C
> >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> >>>> C
> >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>> t
> >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
> >>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
> >>>> b
> >>>> o
> >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
> >>>> 3
> >>>> 8
> >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
> >>>> F
> >>>> p
> >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> >>>> M
> >>>> n
> >>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
> >>>> w
> >>>> N
> >>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
> >>>> i
> >>>> t
> >>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
> >>>> o
> >>>> f
> >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
> >>>> e
> >>>> 2
> >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >>>> M
> >>>> C
> >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> >>>> C
> >>>> %
> >>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
> >>>> s
> >>>> e
> >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>
> >>>
> >>
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog Performance Tuning - Dropped UDP Events [ In reply to ]
A little late to the party here, but I had a similar issue a few months ago with firewall logs...

Ended up being an issue/bug on the firewall (Palo Alto) that would cause TCP syslog to be unreliable. Switching to UDP corrected the issue, but I had to prove it wasn't the syslog receiver side first to the firewall team.

I used loggen (https://www.systutorials.com/docs/linux/man/1-loggen/) to "replay" some firewall logs into the syslog receivers at a rate 2~3x what the firewalls delivered on average to prove it.

Might be helpful from a testing perspective ~ $.02
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com>
Sent: Wednesday, November 16, 2022 3:14 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

> @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu?

Not rsyslog, looks like fluentd:

https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_syslog.rb

Rainer

>
> David Lang
>
>
> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
>
> > Date: Tue, 15 Nov 2022 21:39:45 +0000
> > From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> > To: David Lang <david@lang.hm>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >
> > I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though.
> >
> > If I'm still havcing issues after that, I'll respond back.
> >
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Tuesday, November 15, 2022 4:19 PM
> > To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> > Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >
> > using the new action() syntax, you can name the actions so they aren't just numbered.
> >
> > starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which.
> >
> > that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing)
> >
> > David Lang
> >
> >
> > On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >
> >> Date: Tue, 15 Nov 2022 20:01:52 +0000
> >> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >> To: David Lang <david@lang.hm>
> >> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >>
> >> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0:
> >> origin=core.action processed=1628 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1:
> >> origin=core.action processed=66 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2:
> >> origin=core.action processed=37449 failed=37448 suspended=1
> >> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3:
> >> origin=core.action processed=4582 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4:
> >> origin=core.action processed=5039 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8:
> >> origin=core.action processed=862 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10:
> >> origin=core.action processed=397 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11:
> >> origin=core.action processed=50 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14:
> >> origin=core.action processed=766 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15:
> >> origin=core.action processed=11058 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17:
> >> origin=core.action processed=1099 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18:
> >> origin=core.action processed=12 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514):
> >> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514):
> >> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage:
> >> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469
> >> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15
> >> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0
> >> msgs.received=24227
> >>
> >>
> >> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0:
> >> origin=core.action processed=1728 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1:
> >> origin=core.action processed=47 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2:
> >> origin=core.action processed=35778 failed=35777 suspended=1
> >> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3:
> >> origin=core.action processed=9249 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4:
> >> origin=core.action processed=2296 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8:
> >> origin=core.action processed=1009 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10:
> >> origin=core.action processed=558 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11:
> >> origin=core.action processed=40 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14:
> >> origin=core.action processed=720 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15:
> >> origin=core.action processed=16593 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17:
> >> origin=core.action processed=359 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514):
> >> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514):
> >> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage:
> >> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884
> >> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15
> >> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0
> >> msgs.received=31257
> >>
> >>
> >> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0:
> >> origin=core.action processed=2181 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1:
> >> origin=core.action processed=57 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2:
> >> origin=core.action processed=20793 failed=137 suspended=0
> >> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3:
> >> origin=core.action processed=7599 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4:
> >> origin=core.action processed=5540 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8:
> >> origin=core.action processed=1120 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10:
> >> origin=core.action processed=157807 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11:
> >> origin=core.action processed=50 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13:
> >> origin=core.action processed=3 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14:
> >> origin=core.action processed=293 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15:
> >> origin=core.action processed=20243 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17:
> >> origin=core.action processed=866 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18:
> >> origin=core.action processed=10 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514):
> >> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514):
> >> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage:
> >> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196
> >> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov
> >> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698
> >> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15
> >> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595
> >> called.recvmsg=0 msgs.received=194836
> >>
> >>
> >> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04
> >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0
> >> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0:
> >> origin=core.action processed=2009 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1:
> >> origin=core.action processed=151 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2:
> >> origin=core.action processed=52035 failed=52034 suspended=1
> >> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3:
> >> origin=core.action processed=19442 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4:
> >> origin=core.action processed=4480 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8:
> >> origin=core.action processed=1125 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10:
> >> origin=core.action processed=415 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11:
> >> origin=core.action processed=144 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12:
> >> origin=core.action processed=2 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13:
> >> origin=core.action processed=13 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14:
> >> origin=core.action processed=745 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15:
> >> origin=core.action processed=23626 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17:
> >> origin=core.action processed=682 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18:
> >> origin=core.action processed=45 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20:
> >> origin=core.action processed=0 failed=0 suspended=0
> >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514):
> >> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514):
> >> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514):
> >> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage:
> >> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979
> >> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15
> >> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0
> >> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04
> >> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0
> >> msgs.received=51446 -----Original Message-----
> >> From: David Lang <david@lang.hm>
> >> Sent: Tuesday, November 15, 2022 3:43 PM
> >> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >> Cc: David Lang <david@lang.hm>; rsyslog-users
> >> <rsyslog@lists.adiscon.com>
> >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >>
> >> you have the impstats module loaded in your config and writing stats out, please post the output of this.
> >>
> >> David Lang
> >>
> >> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >>
> >>> Date: Tue, 15 Nov 2022 19:38:27 +0000
> >>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >>> To: David Lang <david@lang.hm>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>> Events
> >>>
> >>> I'm still not understanding what you mean by pstats - it's not a
> >>> package or command available to me. It's apart of Unix from what I
> >>> can tell. I've placed below the unparsed information form
> >>> /proc/net/netstat and /proc/net/udp
> >>>
> >>> /proc/net/netstat
> >>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts
> >>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps
> >>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs
> >>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits
> >>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging
> >>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo
> >>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures
> >>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans
> >>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail
> >>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce
> >>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv
> >>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout
> >>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
> >>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld
> >>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected
> >>> TCPMD5Failure TCPSackShifted T
> >> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop
> >> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter
> >> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
> >> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge
> >> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive
> >> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail
> >> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole
> >> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking
> >> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv
> >> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect
> >> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd
> >> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq
> >> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge
> >> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered
> >> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop
> >> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash
> >> TcpDuplicateDataRehash TCPDSACKRe
> > c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure
> >>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231
> >>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0
> >>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0
> >>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0
> >>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0
> >>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
> >>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets
> >>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts
> >>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
> >>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0
> >>>
> >>> /proc/net/udp
> >>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
> >>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0
> >>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0
> >>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0
> >>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0
> >>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0
> >>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728
> >>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0
> >>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0
> >>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0
> >>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0
> >>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0
> >>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0
> >>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0
> >>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0
> >>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0
> >>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0
> >>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0
> >>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0
> >>>
> >>> The ActionQueue for /var/log/secure has been commented out from the configuration.
> >>>
> >>> Dropwatch (Cycle 1):
> >>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e)
> >>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> >>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d)
> >>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f)
> >>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d)
> >>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab)
> >>>
> >>> Dropwatch (Cycle 2):
> >>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5)
> >>> 14 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d)
> >>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 6 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> >>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 10 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 5 drops at skb_release_data+12b (0xffffffff937f8cab)
> >>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528)
> >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d)
> >>>
> >>> Top -H
> >>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*
> >>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140
> >>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg
> >>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1*
> >>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp
> >>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp
> >>>
> >>> /etc/sysctl.conf
> >>> net.core.rmem_default = 33554432
> >>> net.core.rmem_max = 268435456
> >>> net.core.wmem_default = 33554432
> >>> net.core.wmem_max = 268435456
> >>> net.ipv4.tcp_mem = 190611 254150 381222
> >>> net.ipv4.tcp_rmem = 4096 131072 6291456
> >>> net.ipv4.tcp_wmem = 4096 16384 4194304
> >>> net.ipv4.udp_mem = 762450 1524900 3049800
> >>> net.ipv4.udp_rmem_min = 33554432
> >>> net.ipv4.udp_wmem_min = 33554432
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: David Lang <david@lang.hm>
> >>> Sent: Tuesday, November 15, 2022 2:30 PM
> >>> To: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> >>> <david@lang.hm>
> >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>> Events
> >>>
> >>> what does the pstats output look like when it's dropping messages?
> >>> (give a couple cycles please)
> >>>
> >>> did you try to eliminate the action queue for /var/log/secure?
> >>>
> >>> David Lang
> >>>
> >>> On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> >>>
> >>>> Date: Tue, 15 Nov 2022 13:01:02 +0000
> >>>> From: "Redbourne,Michael" <michael.redbourne@bulletproofsi.com>
> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
> >>>> <david@lang.hm>
> >>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Building on this -
> >>>>
> >>>> When the drop count spikes top is showing a spike in CPU usage among the previously listed threads:
> >>>> In:imdup spikes to ~10%
> >>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25%
> >>>> usage.
> >>>>
> >>>> -----Original Message-----
> >>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> >>>> Redbourne,Michael via rsyslog
> >>>> Sent: Tuesday, November 15, 2022 8:42 AM
> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
> >>>> <david@lang.hm>
> >>>> Cc: Redbourne,Michael <michael.redbourne@bulletproofsi.com>
> >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs:
> >>>> Udp:
> >>>> 5820820 packets received
> >>>> 1504 packets to unknown port received.
> >>>> 798900 packet receive errors
> >>>> 3338814 packets sent
> >>>> 798900 receive buffer errors
> >>>> 0 send buffer errors
> >>>>
> >>>> I have doubled the values in net.ipv4.udp_mem.
> >>>>
> >>>> The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings).
> >>>>
> >>>> The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo):
> >>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains
> >>>> "<subnet>" stop
> >>>>
> >>>> Example Checkpoint Log:
> >>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
> >>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392
> >>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648
> >>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0
> >>>> priority=8 deviceSeverity=Very-High rt=1599552617058
> >>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
> >>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
> >>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
> >>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
> >>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
> >>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
> >>>> dvc=<dvc_ip_addr>
> >>>>
> >>>> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets.
> >>>>
> >>>> Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip.
> >>>>
> >>>> -----Original Message-----
> >>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> >>>> Rainer Gerhards via rsyslog
> >>>> Sent: Tuesday, November 15, 2022 5:11 AM
> >>>> To: David Lang <david@lang.hm>
> >>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users
> >>>> <rsyslog@lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP
> >>>> Events
> >>>>
> >>>> Just wanted to make sure awareness of that option. Agree that it is not often needed.
> >>>>
> >>>> Rainer
> >>>>
> >>>> El mar, 15 nov 2022 a las 10:02, David Lang (<david@lang.hm>) escribi?:
> >>>>>
> >>>>> I haven't needed to do that to handle 300k messages/sec on UDP
> >>>>> input (usually I run into bottlenecks in processing the messages
> >>>>> long before I have problems accepting them)
> >>>>>
> >>>>> David Lang
> >>>>>
> >>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
> >>>>>
> >>>>>> let me add: look into setting imudp to realtime priority. Doc:
> >>>>>>
> >>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> >>>>>> w
> >>>>>> w
> >>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm
> >>>>>> l
> >>>>>> &
> >>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61
> >>>>>> 6
> >>>>>> 2
> >>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0
> >>>>>> %
> >>>>>> 7
> >>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
> >>>>>> J
> >>>>>> Q
> >>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>>>> t
> >>>>>> a
> >>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&amp;reserved=0
> >>>>>>
> >>>>>> Rainer
> >>>>>>
> >>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> >>>>>> (<rsyslog@lists.adiscon.com>) escribi?:
> >>>>>>>
> >>>>>>> Some additional comments on the config
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> These action queue configs probably don't do what you intend them
> >>>>>>> to do
> >>>>>>>
> >>>>>>> the first thing is that they only affect the next action, which
> >>>>>>> is
> >>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to
> >>>>>>> write these logs out. That will create a HUGE amount of
> >>>>>>> contention for the queue lock and under load you should see it
> >>>>>>> maxing out quite quickly
> >>>>>>>
> >>>>>>> what is it that you are attempting to do here?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> # Performance Tuning #
> >>>>>>> $ActionQueueWorkerThreads 2000
> >>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
> >>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
> >>>>>>> 600000
> >>>>>>>
> >>>>>>> #### RULES ####
> >>>>>>> # Log all kernel messages to the console.
> >>>>>>> # Logging much else clutters up the screen.
> >>>>>>> #kern.* /dev/console
> >>>>>>>
> >>>>>>> # Log anything (except mail authpriv, cron) # Dont log private
> >>>>>>> authentication messages!
> >>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>>>>>>
> >>>>>>> # The authpriv file has restricted access.
> >>>>>>> authpriv.* /var/log/secure
> >>>>>>>
> >>>>>>>
> >>>>>>> since the queue only applied to the next action with this config,
> >>>>>>> everything below this is operating from the main queue again as
> >>>>>>> if there was no action queue configuration
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> # Log all the mail messages in one place.
> >>>>>>> mail.* -/var/log/maillog
> >>>>>>>
> >>>>>>> # Log cron stuff
> >>>>>>> cron.* /var/log/cron
> >>>>>>>
> >>>>>>> # Everybody gets emergency messages
> >>>>>>> *.emerg :omusrmsg:*
> >>>>>>>
> >>>>>>> # Save news errors of level crit and higher in a special file.
> >>>>>>> uucp,news.crit /var/log/spooler
> >>>>>>>
> >>>>>>> # Save boot messages also to boot.log
> >>>>>>> # local7.* /var/syslog/boot.log
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ereregex is a fairly expensive filter to apply, it's much better
> >>>>>>> to figure out a non-regex approach to filtering these. Can you
> >>>>>>> post some examples of what you are trying to filter? mmnormalize
> >>>>>>> to parse the logs and then make decisions on the parsed results id probably much faster.
> >>>>>>>
> >>>>>>>
> >>>>>>> /etc/rsyslog.d/security-config-omsagent.conf
> >>>>>>> # [Firewall Log Filtering] #
> >>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
> >>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
> >>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
> >>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop
> >>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP
> >>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
> >>>>>>>
> >>>>>>>
> >>>>>>> check the messages to see where CEF: and ASA- are in the message,
> >>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith'
> >>>>>>> instead of 'contains'?, again mmnormalize may be much faster
> >>>>>>>
> >>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
> >>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
> >>>>>>> then @127.0.0.1:25224 & stop
> >>>>>>>
> >>>>>>>
> >>>>>>> combining multiple filters into one action, or having the filters
> >>>>>>> call a ruleset can be far more efficient than all of them writing things out independently.
> >>>>>>>
> >>>>>>> the if..then filter structure lets you easily combine filters
> >>>>>>>
> >>>>>>> local0.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local1.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local2.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local3.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local4.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local5.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local6.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> local7.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> auth.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> authpriv.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> daemon.info @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> syslog.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
> >>>>>>> user.* @127.0.0.1:25224
> >>>>>>> & stop
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> >>>>>>> F
> >>>>>>> l
> >>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%
> >>>>>>> 7
> >>>>>>> C
> >>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac
> >>>>>>> 6
> >>>>>>> e
> >>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703
> >>>>>>> 1
> >>>>>>> 5
> >>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC
> >>>>>>> J
> >>>>>>> B
> >>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BlY86%2FvQ
> >>>>>>> y
> >>>>>>> n
> >>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&amp;reserved=0
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> >>>>>>> w
> >>>>>>> w
> >>>>>>> w.rsyslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmicha
> >>>>>>> e
> >>>>>>> l
> >>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e
> >>>>>>> %
> >>>>>>> 7
> >>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C
> >>>>>>> U
> >>>>>>> n
> >>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> >>>>>>> k
> >>>>>>> 1
> >>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=W96%2BKd2Th68p6gYB6
> >>>>>>> I
> >>>>>>> o
> >>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&amp;reserved=0
> >>>>>>> What's up with rsyslog? Follow
> >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> >>>>>>> F
> >>>>>>> t
> >>>>>>> witter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bul
> >>>>>>> l
> >>>>>>> e
> >>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411
> >>>>>>> b
> >>>>>>> b
> >>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs
> >>>>>>> b
> >>>>>>> 3
> >>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> >>>>>>> %
> >>>>>>> 3
> >>>>>>> D%7C3000%7C%7C%7C&amp;sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3
> >>>>>>> O
> >>>>>>> Y
> >>>>>>> kHGY%3D&amp;reserved=0 NOTE WELL: This is a PUBLIC mailing list,
> >>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> >>>> s
> >>>> t
> >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
> >>>> a
> >>>> e
> >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
> >>>> C
> >>>> 9
> >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
> >>>> w
> >>>> n
> >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>>> C
> >>>> J
> >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
> >>>> %
> >>>> 2
> >>>> BRRguRnpir9RsV8%3D&amp;reserved=0
> >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
> >>>> e
> >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
> >>>> C
> >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> >>>> C
> >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>> t
> >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
> >>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
> >>>> b
> >>>> o
> >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
> >>>> 3
> >>>> 8
> >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
> >>>> F
> >>>> p
> >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> >>>> M
> >>>> n
> >>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
> >>>> w
> >>>> N
> >>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
> >>>> i
> >>>> t
> >>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
> >>>> o
> >>>> f
> >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
> >>>> e
> >>>> 2
> >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >>>> M
> >>>> C
> >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> >>>> C
> >>>> %
> >>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
> >>>> s
> >>>> e
> >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>> ________________________________________
> >>>> This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
> >>>>
> >>>> Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
> >>>> ________________________________________
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> >>>> s
> >>>> t
> >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=05%7C01%7Cmich
> >>>> a
> >>>> e
> >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7
> >>>> C
> >>>> 9
> >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno
> >>>> w
> >>>> n
> >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>>> C
> >>>> J
> >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ
> >>>> %
> >>>> 2
> >>>> BRRguRnpir9RsV8%3D&amp;reserved=0
> >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >>>> r%2F&amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44
> >>>> e
> >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7
> >>>> C
> >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> >>>> C
> >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sda
> >>>> t
> >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&amp;reserved=0
> >>>> syslog.com%2Fprofessional-services%2F&amp;data=05%7C01%7Cmichael.red
> >>>> b
> >>>> o
> >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1
> >>>> 3
> >>>> 8
> >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW
> >>>> F
> >>>> p
> >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> >>>> M
> >>>> n
> >>>> 0%3D%7C3000%7C%7C%7C&amp;sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3
> >>>> w
> >>>> N
> >>>> KcbIFqhg%3D&amp;reserved=0 What's up with rsyslog? Follow
> >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw
> >>>> i
> >>>> t
> >>>> ter.com%2Frgerhards&amp;data=05%7C01%7Cmichael.redbourne%40bulletpro
> >>>> o
> >>>> f
> >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7
> >>>> e
> >>>> 2
> >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >>>> M
> >>>> C
> >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> >>>> C
> >>>> %
> >>>> 7C&amp;sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&amp;re
> >>>> s
> >>>> e
> >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>
> >>>
> >>
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.