Mailing List Archive

setting up reliable forwarding of syslog Messages with Rsyslog
I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:

https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html

I confirm that remote logging is working initially by doing

# logger "hello, world"

on the client, and verifying that this message shows up in the server (in this case in /var/log/syslog)

I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.

I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?

Some configuration files:

--------------------------------------------------------------------------------------------
client rsyslog.conf file:

# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# setup reliable local buffering
#
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
*.* @@<redacted>:514

------------------------------------------------------------------
server rsyslog.conf file

# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

------------------------------------------------------------------
version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)

# rsyslogd -version (same version for both client and server)

rsyslogd 8.32.0, compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
Firstly, after you confirm that your queueing works properly, I'd advise
you to switch to RELP so you have "more reliability".

But regarding your setup - as you defined

$WorkDirectory /var/spool/rsyslog

Your queue should be placed there.

Question is whether you do indeed have such directory in your system.
Because if you don't, the rsyslog daemon won't be able to save the queue
contents.

But in case of just a few messages you shouldn't be saving the contents
do disk at all. (it would be saved when you have unsent messages and
shut down the rsyslog daemon).

Also, notice that
https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeatedmsgreduction.html
"This parameter models old sysklogd legacy. *Note that many people,
including the rsyslog authors, consider this to be a misfeature.* See
/Discussion/ below to learn why."

But in general, the setup should work... with one caveat. Your "never"
might in fact not be "never". You didn't tweak the settings that control
action resuming so they are at default 30 second initial interval which
is getting raised after every 10 tries up to a default 1800 seconds. So
if the server was off for long enough, the client might simply have
paused sending for a really significant time.

See the description of parameters at
https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.

You might set (just for test! you probably don't want to set it in prod
for that often)

$ActionResumeInterval 1

And then run your client instance in debug mode to see interactively
what it's trying to do.

rsyslogd -f rsyslog.conf -i NONE -n -d




On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>
> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html
>
> I confirm that remote logging is working initially by doing
>
> # logger "hello, world"
>
> on the client, and verifying that this message shows up in the server (in this case in /var/log/syslog)
>
> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>
> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>
> Some configuration files:
>
> --------------------------------------------------------------------------------------------
> client rsyslog.conf file:
>
> # /etc/rsyslog.conf Configuration file for rsyslog.
> #
> # For more information see
> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
> #
> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>
>
> #################
> #### MODULES ####
> #################
>
> module(load="imuxsock") # provides support for local system logging
> #module(load="immark") # provides --MARK-- message capability
>
> # provides UDP syslog reception
> #module(load="imudp")
> #input(type="imudp" port="514")
>
> # provides TCP syslog reception
> #module(load="imtcp")
> #input(type="imtcp" port="514")
>
> # provides kernel logging support and enable non-kernel klog messages
> module(load="imklog" permitnonkernelfacility="on")
>
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
>
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Filter duplicated messages
> $RepeatedMsgReduction on
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> #
> # Where to place spool and state files
> #
> $WorkDirectory /var/spool/rsyslog
>
> #
> # setup reliable local buffering
> #
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>
> #
> # Include all config files in /etc/rsyslog.d/
> #
> $IncludeConfig /etc/rsyslog.d/*.conf
> *.* @@<redacted>:514
>
> ------------------------------------------------------------------
> server rsyslog.conf file
>
> # /etc/rsyslog.conf Configuration file for rsyslog.
> #
> # For more information see
> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
> #
> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>
>
> #################
> #### MODULES ####
> #################
>
> module(load="imuxsock") # provides support for local system logging
> #module(load="immark") # provides --MARK-- message capability
>
> # provides UDP syslog reception
> #module(load="imudp")
> #input(type="imudp" port="514")
>
> # provides TCP syslog reception
> module(load="imtcp")
> input(type="imtcp" port="514")
>
> # provides kernel logging support and enable non-kernel klog messages
> module(load="imklog" permitnonkernelfacility="on")
>
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
>
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Filter duplicated messages
> $RepeatedMsgReduction on
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> #
> # Where to place spool and state files
> #
> $WorkDirectory /var/spool/rsyslog
>
> #
> # Include all config files in /etc/rsyslog.d/
> #
> $IncludeConfig /etc/rsyslog.d/*.conf
>
> ------------------------------------------------------------------
> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>
> # rsyslogd -version (same version for both client and server)
>
> rsyslogd 8.32.0, compiled with:
> PLATFORM: x86_64-pc-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> systemd support: Yes
> Number of Bits in RainerScript integers: 64
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Followhttps://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
So, following your advice, I've confirmed the following

1. I switched to RELP. as per the following:

add the following to server rsyslog.conf

module(load="imrelp")
input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")

add the following to server 50-default.conf:

local7.* -/var/log/local7.log

add the following to client 50-default.conf

local7.* -/var/log/local7.log
local7.* :omrelp:<server>:2514

2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.

3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up

Here's exactly how I tested:

1. log a message from the client, verify that it shows up on the server
# logger -p local7.info -s 'hello world'

shows up in /var/log/local7.log on the server
shows up in /var/log/local7.log on the client

2. disable rsyslog on the server
# systemctl stop syslog.socket rsyslog.service

3. log a message on the client
# logger -p local7.info -s 'hello world 2'

shows up in /var/log/local7.log on the client

4. enable rsyslog on the server
# systemctl start syslog.socket rsyslog.service

5. log a message on the client
# logger -p local7.info -s 'hello world 3'

shows up in /var/log/local7.log on the server
shows up in /var/log/local7.log on the client

"hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.

I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.

There must be something I'm doing wrong, but what?

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: Friday, February 18, 2022 4:18 AM
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".

But regarding your setup - as you defined

$WorkDirectory /var/spool/rsyslog

Your queue should be placed there.

Question is whether you do indeed have such directory in your system.
Because if you don't, the rsyslog daemon won't be able to save the queue contents.

But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).

Also, notice that
https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeatedmsgreduction.html
"This parameter models old sysklogd legacy. *Note that many people, including the rsyslog authors, consider this to be a misfeature.* See /Discussion/ below to learn why."

But in general, the setup should work... with one caveat. Your "never"
might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.

See the description of parameters at
https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.

You might set (just for test! you probably don't want to set it in prod for that often)

$ActionResumeInterval 1

And then run your client instance in debug mode to see interactively what it's trying to do.

rsyslogd -f rsyslog.conf -i NONE -n -d




On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>
> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html
>
> I confirm that remote logging is working initially by doing
>
> # logger "hello, world"
>
> on the client, and verifying that this message shows up in the server
> (in this case in /var/log/syslog)
>
> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>
> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>
> Some configuration files:
>
> ----------------------------------------------------------------------
> ----------------------
> client rsyslog.conf file:
>
> # /etc/rsyslog.conf Configuration file for rsyslog.
> #
> # For more information see
> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
> #
> # Default logging rules can be found in
> /etc/rsyslog.d/50-default.conf
>
>
> #################
> #### MODULES ####
> #################
>
> module(load="imuxsock") # provides support for local system logging
> #module(load="immark") # provides --MARK-- message capability
>
> # provides UDP syslog reception
> #module(load="imudp")
> #input(type="imudp" port="514")
>
> # provides TCP syslog reception
> #module(load="imtcp")
> #input(type="imtcp" port="514")
>
> # provides kernel logging support and enable non-kernel klog messages
> module(load="imklog" permitnonkernelfacility="on")
>
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
>
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Filter duplicated messages
> $RepeatedMsgReduction on
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> #
> # Where to place spool and state files # $WorkDirectory
> /var/spool/rsyslog
>
> #
> # setup reliable local buffering
> #
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
> down
>
> #
> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
> /etc/rsyslog.d/*.conf
> *.* @@<redacted>:514
>
> ------------------------------------------------------------------
> server rsyslog.conf file
>
> # /etc/rsyslog.conf Configuration file for rsyslog.
> #
> # For more information see
> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
> #
> # Default logging rules can be found in
> /etc/rsyslog.d/50-default.conf
>
>
> #################
> #### MODULES ####
> #################
>
> module(load="imuxsock") # provides support for local system logging
> #module(load="immark") # provides --MARK-- message capability
>
> # provides UDP syslog reception
> #module(load="imudp")
> #input(type="imudp" port="514")
>
> # provides TCP syslog reception
> module(load="imtcp")
> input(type="imtcp" port="514")
>
> # provides kernel logging support and enable non-kernel klog messages
> module(load="imklog" permitnonkernelfacility="on")
>
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
>
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Filter duplicated messages
> $RepeatedMsgReduction on
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> #
> # Where to place spool and state files # $WorkDirectory
> /var/spool/rsyslog
>
> #
> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
> /etc/rsyslog.d/*.conf
>
> ------------------------------------------------------------------
> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>
> # rsyslogd -version (same version for both client and server)
>
> rsyslogd 8.32.0, compiled with:
> PLATFORM: x86_64-pc-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> systemd support: Yes
> Number of Bits in RainerScript integers: 64
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Followhttps://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
If you run a client as

rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry

You should see some text blob at the start but then, when the server is
running, the client should not emit any more messages.

But when you stop the server, the client should start emiting messages like

5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0, ResumeInRow 1
rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
[v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ]
5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007
5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd check for max retries,
iResumeRetryCount -1, iRetries 0

And if you look for the string '<somenumber> messages' in debug log, if
you close the client some time after stopping the server and pushing
some more messages to the client, you should get something like

rsyslog internal message (6,-2041): action-0-builtin:omfwd queue: queue
holds 2 messages after shutdown of workers. queue.saveonshutdown is set,
so data will now be spooled to disk [v8.2102.0-4.fc35 try
https://www.rsyslog.com/e/2041 ]

I'm not fully sure, however, since you use the legacy config format
what's the interaction between both actions within the same queue. In
order to be sure to have proper queueing _on the forwarding action_ I'd
do a separate queue for this omfwd (or omrelp or whatever you're gonna
use in the end) action alone.

On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
> So, following your advice, I've confirmed the following
>
> 1. I switched to RELP. as per the following:
>
> add the following to server rsyslog.conf
>
> module(load="imrelp")
> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>
> add the following to server 50-default.conf:
>
> local7.* -/var/log/local7.log
>
> add the following to client 50-default.conf
>
> local7.* -/var/log/local7.log
> local7.* :omrelp:<server>:2514
>
> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>
> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>
> Here's exactly how I tested:
>
> 1. log a message from the client, verify that it shows up on the server
> # logger -p local7.info -s 'hello world'
>
> shows up in /var/log/local7.log on the server
> shows up in /var/log/local7.log on the client
>
> 2. disable rsyslog on the server
> # systemctl stop syslog.socket rsyslog.service
>
> 3. log a message on the client
> # logger -p local7.info -s 'hello world 2'
>
> shows up in /var/log/local7.log on the client
>
> 4. enable rsyslog on the server
> # systemctl start syslog.socket rsyslog.service
>
> 5. log a message on the client
> # logger -p local7.info -s 'hello world 3'
>
> shows up in /var/log/local7.log on the server
> shows up in /var/log/local7.log on the client
>
> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>
> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>
> There must be something I'm doing wrong, but what?
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: Friday, February 18, 2022 4:18 AM
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>
> But regarding your setup - as you defined
>
> $WorkDirectory /var/spool/rsyslog
>
> Your queue should be placed there.
>
> Question is whether you do indeed have such directory in your system.
> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>
> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>
> Also, notice that
> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeatedmsgreduction.html
> "This parameter models old sysklogd legacy. *Note that many people, including the rsyslog authors, consider this to be a misfeature.* See /Discussion/ below to learn why."
>
> But in general, the setup should work... with one caveat. Your "never"
> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>
> See the description of parameters at
> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>
> You might set (just for test! you probably don't want to set it in prod for that often)
>
> $ActionResumeInterval 1
>
> And then run your client instance in debug mode to see interactively what it's trying to do.
>
> rsyslogd -f rsyslog.conf -i NONE -n -d
>
>
>
>
> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>
>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html
>>
>> I confirm that remote logging is working initially by doing
>>
>> # logger "hello, world"
>>
>> on the client, and verifying that this message shows up in the server
>> (in this case in /var/log/syslog)
>>
>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>
>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>
>> Some configuration files:
>>
>> ----------------------------------------------------------------------
>> ----------------------
>> client rsyslog.conf file:
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in
>> /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> module(load="imuxsock") # provides support for local system logging
>> #module(load="immark") # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> #module(load="imudp")
>> #input(type="imudp" port="514")
>>
>> # provides TCP syslog reception
>> #module(load="imtcp")
>> #input(type="imtcp" port="514")
>>
>> # provides kernel logging support and enable non-kernel klog messages
>> module(load="imklog" permitnonkernelfacility="on")
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files # $WorkDirectory
>> /var/spool/rsyslog
>>
>> #
>> # setup reliable local buffering
>> #
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>> down
>>
>> #
>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>> /etc/rsyslog.d/*.conf
>> *.* @@<redacted>:514
>>
>> ------------------------------------------------------------------
>> server rsyslog.conf file
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in
>> /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> module(load="imuxsock") # provides support for local system logging
>> #module(load="immark") # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> #module(load="imudp")
>> #input(type="imudp" port="514")
>>
>> # provides TCP syslog reception
>> module(load="imtcp")
>> input(type="imtcp" port="514")
>>
>> # provides kernel logging support and enable non-kernel klog messages
>> module(load="imklog" permitnonkernelfacility="on")
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files # $WorkDirectory
>> /var/spool/rsyslog
>>
>> #
>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>> /etc/rsyslog.d/*.conf
>>
>> ------------------------------------------------------------------
>> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>>
>> # rsyslogd -version (same version for both client and server)
>>
>> rsyslogd 8.32.0, compiled with:
>> PLATFORM: x86_64-pc-linux-gnu
>> PLATFORM (lsb_release -d):
>> FEATURE_REGEXP: Yes
>> GSSAPI Kerberos 5 support: Yes
>> FEATURE_DEBUG (debug build, slow code): No
>> 32bit Atomic operations supported: Yes
>> 64bit Atomic operations supported: Yes
>> memory allocator: system default
>> Runtime Instrumentation (slow code): No
>> uuid support: Yes
>> systemd support: Yes
>> Number of Bits in RainerScript integers: 64
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:

> I'm not fully sure, however, since you use the legacy config format what's
> the interaction between both actions within the same queue. In order to be
> sure to have proper queueing _on the forwarding action_ I'd do a separate
> queue for this omfwd (or omrelp or whatever you're gonna use in the end)
> action alone.

you can't have one queue for multiple actions, you can have a queue on a ruleset
that contains multiple actions, but a queue on one action is only on that one
action.

This is one of the reasons that the legacy format is discouraged for this sort
of thing. In the new format where the queue is part of the action() statement,
it is very clear that the queue is only on that action, but in the legacy
format, even though the behavior is the same (the queue is only on one action),
it reads as if the queue could/should apply to multiple actions.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
My bad. I meant that both actions are within the same main implied ruleset as far as I understand the legacy format.
Anyway, just to be on the safe side, I'd do a separate "output" ruleset with its own queue and within that ruleset I'd do the output action.
That does work. I have several dozens hosts working like that.


On 18 February 2022 22:51:54 CET, David Lang <david@lang.hm> wrote:
>On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>
>> I'm not fully sure, however, since you use the legacy config format what's
>> the interaction between both actions within the same queue. In order to be
>> sure to have proper queueing _on the forwarding action_ I'd do a separate
>> queue for this omfwd (or omrelp or whatever you're gonna use in the end)
>> action alone.
>
>you can't have one queue for multiple actions, you can have a queue on a ruleset
>that contains multiple actions, but a queue on one action is only on that one
>action.
>
>This is one of the reasons that the legacy format is discouraged for this sort
>of thing. In the new format where the queue is part of the action() statement,
>it is very clear that the queue is only on that action, but in the legacy
>format, even though the behavior is the same (the queue is only on one action),
>it reads as if the queue could/should apply to multiple actions.
>
>David Lang

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
right, but the following doesn't setup buffering on the implied ruleset, it sets
it up on the first action in the first file found /etc/rsyslog.d/*.conf


$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


yes, setting up an action on the ruleset works, but I don't think that's
supported in the legacy format at all

David Lang

On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:

> My bad. I meant that both actions are within the same main implied ruleset as far as I understand the legacy format.
> Anyway, just to be on the safe side, I'd do a separate "output" ruleset with its own queue and within that ruleset I'd do the output action.
> That does work. I have several dozens hosts working like that.
>
>
> On 18 February 2022 22:51:54 CET, David Lang <david@lang.hm> wrote:
>> On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>>
>>> I'm not fully sure, however, since you use the legacy config format what's
>>> the interaction between both actions within the same queue. In order to be
>>> sure to have proper queueing _on the forwarding action_ I'd do a separate
>>> queue for this omfwd (or omrelp or whatever you're gonna use in the end)
>>> action alone.
>>
>> you can't have one queue for multiple actions, you can have a queue on a ruleset
>> that contains multiple actions, but a queue on one action is only on that one
>> action.
>>
>> This is one of the reasons that the legacy format is discouraged for this sort
>> of thing. In the new format where the queue is part of the action() statement,
>> it is very clear that the queue is only on that action, but in the legacy
>> format, even though the behavior is the same (the queue is only on one action),
>> it reads as if the queue could/should apply to multiple actions.
>>
>> David Lang
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
OK, let me get this straight because the legacy format is confusing at
times :-)

If I do

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* /tmp/whatever.log
*.* @@some.host:port

The queue parameters will get applied only to the first action? (in this
case - to file write)?

In order to apply them to the next action I'd have to repeat the
$ActionWhatever statements?

MK

On 19.02.2022 00:21, David Lang via rsyslog wrote:
> right, but the following doesn't setup buffering on the implied
> ruleset, it sets it up on the first action in the first file found
> /etc/rsyslog.d/*.conf
>
>
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>
> #
> # Include all config files in /etc/rsyslog.d/
> #
> $IncludeConfig /etc/rsyslog.d/*.conf
>
>
> yes, setting up an action on the ruleset works, but I don't think
> that's supported in the legacy format at all
>
> David Lang
>
> On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>
>> My bad. I meant that both actions are within the same main implied
>> ruleset as far as I understand the legacy format.
>> Anyway, just to be on the safe side, I'd do a separate "output"
>> ruleset with its own queue and within that ruleset I'd do the output
>> action.
>> That does work. I have several dozens hosts working like that.
>>
>>
>> On 18 February 2022 22:51:54 CET, David Lang <david@lang.hm> wrote:
>>> On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>>>
>>>> I'm not fully sure, however, since you use the legacy config format
>>>> what's
>>>> the interaction between both actions within the same queue. In
>>>> order to be
>>>> sure to have proper queueing _on the forwarding action_ I'd do a
>>>> separate
>>>> queue for this omfwd (or omrelp or whatever you're gonna use in the
>>>> end)
>>>> action alone.
>>>
>>> you can't have one queue for multiple actions, you can have a queue
>>> on a ruleset
>>> that contains multiple actions, but a queue on one action is only on
>>> that one
>>> action.
>>>
>>> This is one of the reasons that the legacy format is discouraged for
>>> this sort
>>> of thing. In the new format where the queue is part of the action()
>>> statement,
>>> it is very clear that the queue is only on that action, but in the
>>> legacy
>>> format, even though the behavior is the same (the queue is only on
>>> one action),
>>> it reads as if the queue could/should apply to multiple actions.
>>>
>>> David Lang
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
On Sat, 19 Feb 2022, Mariusz Kruk via rsyslog wrote:

> OK, let me get this straight because the legacy format is confusing at times
> :-)
>
> If I do
>
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
> *.* /tmp/whatever.log
> *.* @@some.host:port
>
> The queue parameters will get applied only to the first action? (in this case
> - to file write)?
>
> In order to apply them to the next action I'd have to repeat the
> $ActionWhatever statements?

mostly, not always.

queue items only apply to the next action, I dont' remember if retryCount
applies only to the next action or to all actions until it's set to something
else.

there is a reason why most of the $foo settings are discouraged. If you can do
something in one line with the old format, it can make sense to do it. but if
you have to do several $foo lines and then your action, you probably should
switch to the action() format to make it explicitly clear what you are doing.

David Lang

> MK
>
> On 19.02.2022 00:21, David Lang via rsyslog wrote:
>> right, but the following doesn't setup buffering on the implied ruleset, it
>> sets it up on the first action in the first file found
>> /etc/rsyslog.d/*.conf
>>
>>
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>>
>> #
>> # Include all config files in /etc/rsyslog.d/
>> #
>> $IncludeConfig /etc/rsyslog.d/*.conf
>>
>>
>> yes, setting up an action on the ruleset works, but I don't think that's
>> supported in the legacy format at all
>>
>> David Lang
>>
>> On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>>
>>> My bad. I meant that both actions are within the same main implied ruleset
>>> as far as I understand the legacy format.
>>> Anyway, just to be on the safe side, I'd do a separate "output" ruleset
>>> with its own queue and within that ruleset I'd do the output action.
>>> That does work. I have several dozens hosts working like that.
>>>
>>>
>>> On 18 February 2022 22:51:54 CET, David Lang <david@lang.hm> wrote:
>>>> On Fri, 18 Feb 2022, Mariusz Kruk via rsyslog wrote:
>>>>
>>>>> I'm not fully sure, however, since you use the legacy config format
>>>>> what's
>>>>> the interaction between both actions within the same queue. In order to
>>>>> be
>>>>> sure to have proper queueing _on the forwarding action_ I'd do a
>>>>> separate
>>>>> queue for this omfwd (or omrelp or whatever you're gonna use in the end)
>>>>> action alone.
>>>>
>>>> you can't have one queue for multiple actions, you can have a queue on a
>>>> ruleset
>>>> that contains multiple actions, but a queue on one action is only on that
>>>> one
>>>> action.
>>>>
>>>> This is one of the reasons that the legacy format is discouraged for this
>>>> sort
>>>> of thing. In the new format where the queue is part of the action()
>>>> statement,
>>>> it is very clear that the queue is only on that action, but in the legacy
>>>> format, even though the behavior is the same (the queue is only on one
>>>> action),
>>>> it reads as if the queue could/should apply to multiple actions.
>>>>
>>>> David Lang
>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
On 19.02.2022 12:40, David Lang via rsyslog wrote:
> On Sat, 19 Feb 2022, Mariusz Kruk via rsyslog wrote:
>
>> OK, let me get this straight because the legacy format is confusing
>> at times :-)
>>
>> If I do
>>
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>> *.* /tmp/whatever.log
>> *.* @@some.host:port
>>
>> The queue parameters will get applied only to the first action? (in
>> this case - to file write)?
>>
>> In order to apply them to the next action I'd have to repeat the
>> $ActionWhatever statements?
>
> mostly, not always.
>
> queue items only apply to the next action, I dont' remember if
> retryCount applies only to the next action or to all actions until
> it's set to something else.
>
> there is a reason why most of the $foo settings are discouraged. If
> you can do something in one line with the old format, it can make
> sense to do it. but if you have to do several $foo lines and then your
> action, you probably should switch to the action() format to make it
> explicitly clear what you are doing.
>
OK. I did some testing on my own and it seems to confirm that.

I did a very simple setup

module(load="imudp")
input(type="imudp" port="10514")
$WorkDirectory /tmp/rsyslog1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionResumeInterval 1
#*.* -/tmp/log.log
*.* @@127.0.0.1:10514

If I run rsyslogd in debug mode and I don't have anything running on TCP
port to receive forwarded data, I see that messages do indeed get queued
and retried every second.

But if I uncomment local file logging, it's that action that gets
retried in case of failure (I had some pre-existing /tmp/log.log file
with wrong ownership so the action would fail until I removed it) but
the forward is tried only once and forgotten about (at least for some
time; I didn't wait very long for retry but it definitely wasn't retried
as often as $ActionResumeInterval).

So I suppose the clou of the OP's problem is that his config (which
relies on a very old guide - way back from 2008 if I see correctly) sets
the retry parameters for the wrong action.

Maybe it's worth mentioning in that document that it's a bit outdated
and possibly misleading if applied "incorrectly"?

MK


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
On 19.02.2022 14:11, Mariusz Kruk via rsyslog wrote:
> I did a very simple setup
>
> module(load="imudp")
> input(type="imudp" port="10514")
> $WorkDirectory /tmp/rsyslog1
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
> $ActionResumeInterval 1
> #*.* -/tmp/log.log
> *.* @@127.0.0.1:10514
>
> If I run rsyslogd in debug mode and I don't have anything running on
> TCP port to receive forwarded data, I see that messages do indeed get
> queued and retried every second.
>
> But if I uncomment local file logging, it's that action that gets
> retried in case of failure (I had some pre-existing /tmp/log.log file
> with wrong ownership so the action would fail until I removed it) but
> the forward is tried only once and forgotten about (at least for some
> time; I didn't wait very long for retry but it definitely wasn't
> retried as often as $ActionResumeInterval).
>
> So I suppose the clou of the OP's problem is that his config (which
> relies on a very old guide - way back from 2008 if I see correctly)
> sets the retry parameters for the wrong action.
>
> Maybe it's worth mentioning in that document that it's a bit outdated
> and possibly misleading if applied "incorrectly"?
>
OK. I did some further testing and indeed, duplicating all those $Action
parameters again after the first action results in the forward action
being indeed repeated properly every second.

So the OP's problem was really not that it didn't work but that it
worked only on the first action it encountered in the config.

But it's indeed best to use the new format

action(type="omfwd" action.resumeRetryCount="-1" target="my.server" )

It's much clearer this way.


MK

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
What I found was that I had to do this in 50-default.conf:

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionQueueMaxDiskSpace 1g
$ActionResumeInterval 1
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

local7.* :omrelp:will:2514

I believe that's what you meant here, yes?
>I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.

When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.

I wish I understood the configuration better. I have to admit, I find the documentation really confusing.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: Friday, February 18, 2022 3:22 PM
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

If you run a client as

rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry

You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.

But when you stop the server, the client should start emiting messages like

5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0, ResumeInRow 1
rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
[v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd check for max retries, iResumeRetryCount -1, iRetries 0

And if you look for the string '<somenumber> messages' in debug log, if you close the client some time after stopping the server and pushing some more messages to the client, you should get something like

rsyslog internal message (6,-2041): action-0-builtin:omfwd queue: queue holds 2 messages after shutdown of workers. queue.saveonshutdown is set, so data will now be spooled to disk [v8.2102.0-4.fc35 try
https://www.rsyslog.com/e/2041 ]

I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.

On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
> So, following your advice, I've confirmed the following
>
> 1. I switched to RELP. as per the following:
>
> add the following to server rsyslog.conf
>
> module(load="imrelp")
> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>
> add the following to server 50-default.conf:
>
> local7.* -/var/log/local7.log
>
> add the following to client 50-default.conf
>
> local7.* -/var/log/local7.log
> local7.* :omrelp:<server>:2514
>
> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>
> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>
> Here's exactly how I tested:
>
> 1. log a message from the client, verify that it shows up on the server
> # logger -p local7.info -s 'hello world'
>
> shows up in /var/log/local7.log on the server
> shows up in /var/log/local7.log on the client
>
> 2. disable rsyslog on the server
> # systemctl stop syslog.socket rsyslog.service
>
> 3. log a message on the client
> # logger -p local7.info -s 'hello world 2'
>
> shows up in /var/log/local7.log on the client
>
> 4. enable rsyslog on the server
> # systemctl start syslog.socket rsyslog.service
>
> 5. log a message on the client
> # logger -p local7.info -s 'hello world 3'
>
> shows up in /var/log/local7.log on the server
> shows up in /var/log/local7.log on the client
>
> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>
> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>
> There must be something I'm doing wrong, but what?
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
> Kruk via rsyslog
> Sent: Friday, February 18, 2022 4:18 AM
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
> Messages with Rsyslog
>
> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>
> But regarding your setup - as you defined
>
> $WorkDirectory /var/spool/rsyslog
>
> Your queue should be placed there.
>
> Question is whether you do indeed have such directory in your system.
> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>
> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>
> Also, notice that
> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeat
> edmsgreduction.html "This parameter models old sysklogd legacy. *Note
> that many people, including the rsyslog authors, consider this to be a misfeature.* See /Discussion/ below to learn why."
>
> But in general, the setup should work... with one caveat. Your "never"
> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>
> See the description of parameters at
> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>
> You might set (just for test! you probably don't want to set it in
> prod for that often)
>
> $ActionResumeInterval 1
>
> And then run your client instance in debug mode to see interactively what it's trying to do.
>
> rsyslogd -f rsyslog.conf -i NONE -n -d
>
>
>
>
> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>
>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html
>>
>> I confirm that remote logging is working initially by doing
>>
>> # logger "hello, world"
>>
>> on the client, and verifying that this message shows up in the server
>> (in this case in /var/log/syslog)
>>
>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>
>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>
>> Some configuration files:
>>
>> ---------------------------------------------------------------------
>> -
>> ----------------------
>> client rsyslog.conf file:
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in
>> /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> module(load="imuxsock") # provides support for local system logging
>> #module(load="immark") # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> #module(load="imudp")
>> #input(type="imudp" port="514")
>>
>> # provides TCP syslog reception
>> #module(load="imtcp")
>> #input(type="imtcp" port="514")
>>
>> # provides kernel logging support and enable non-kernel klog messages
>> module(load="imklog" permitnonkernelfacility="on")
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files # $WorkDirectory
>> /var/spool/rsyslog
>>
>> #
>> # setup reliable local buffering
>> #
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>> down
>>
>> #
>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>> /etc/rsyslog.d/*.conf
>> *.* @@<redacted>:514
>>
>> ------------------------------------------------------------------
>> server rsyslog.conf file
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in
>> /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> module(load="imuxsock") # provides support for local system logging
>> #module(load="immark") # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> #module(load="imudp")
>> #input(type="imudp" port="514")
>>
>> # provides TCP syslog reception
>> module(load="imtcp")
>> input(type="imtcp" port="514")
>>
>> # provides kernel logging support and enable non-kernel klog messages
>> module(load="imklog" permitnonkernelfacility="on")
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files # $WorkDirectory
>> /var/spool/rsyslog
>>
>> #
>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>> /etc/rsyslog.d/*.conf
>>
>> ------------------------------------------------------------------
>> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>>
>> # rsyslogd -version (same version for both client and server)
>>
>> rsyslogd 8.32.0, compiled with:
>> PLATFORM: x86_64-pc-linux-gnu
>> PLATFORM (lsb_release -d):
>> FEATURE_REGEXP: Yes
>> GSSAPI Kerberos 5 support: Yes
>> FEATURE_DEBUG (debug build, slow code): No
>> 32bit Atomic operations supported: Yes
>> 64bit Atomic operations supported: Yes
>> memory allocator: system default
>> Runtime Instrumentation (slow code): No
>> uuid support: Yes
>> systemd support: Yes
>> Number of Bits in RainerScript integers: 64
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
Not exactly, because with "creating a spearate ruleset" I meant a
completely different RainerScript-based configuration but this one
should also work as I wrote "somewhere around".

Anyway, as David wrote somewhere in this thread - legacy config format
is OK for simple setups where it's more readable than Rainer Script but
if your config requires multiple directives modifying functionality of
the action, it's probably easier to write it as (in your case)

if ($syslogfacility == "local7") then
    action(type="omfwd" Target="wll" Port="2514"
action.resumeRetryCount="0" [... more action.parameters and
queue.parameters ...] )

It's more obvious then what the parameters are for and you don't have
them scattered around (possibly intertwining with other parameters
modifying the resulting config).

MK

PS: I'm not sure if this condition will work this way; there was some
bug lately about textual representation but I don't recall if it was
facility or severity or both.

On 22.02.2022 14:31, MACGREGOR Will wrote:
> What I found was that I had to do this in 50-default.conf:
>
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionQueueMaxDiskSpace 1g
> $ActionResumeInterval 1
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>
> local7.* :omrelp:will:2514
>
> I believe that's what you meant here, yes?
>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>
> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: Friday, February 18, 2022 3:22 PM
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> If you run a client as
>
> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>
> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>
> But when you stop the server, the client should start emiting messages like
>
> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0, ResumeInRow 1
> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd check for max retries, iResumeRetryCount -1, iRetries 0
>
> And if you look for the string '<somenumber> messages' in debug log, if you close the client some time after stopping the server and pushing some more messages to the client, you should get something like
>
> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue: queue holds 2 messages after shutdown of workers. queue.saveonshutdown is set, so data will now be spooled to disk [v8.2102.0-4.fc35 try
> https://www.rsyslog.com/e/2041 ]
>
> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>
> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>> So, following your advice, I've confirmed the following
>>
>> 1. I switched to RELP. as per the following:
>>
>> add the following to server rsyslog.conf
>>
>> module(load="imrelp")
>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>
>> add the following to server 50-default.conf:
>>
>> local7.* -/var/log/local7.log
>>
>> add the following to client 50-default.conf
>>
>> local7.* -/var/log/local7.log
>> local7.* :omrelp:<server>:2514
>>
>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>
>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>
>> Here's exactly how I tested:
>>
>> 1. log a message from the client, verify that it shows up on the server
>> # logger -p local7.info -s 'hello world'
>>
>> shows up in /var/log/local7.log on the server
>> shows up in /var/log/local7.log on the client
>>
>> 2. disable rsyslog on the server
>> # systemctl stop syslog.socket rsyslog.service
>>
>> 3. log a message on the client
>> # logger -p local7.info -s 'hello world 2'
>>
>> shows up in /var/log/local7.log on the client
>>
>> 4. enable rsyslog on the server
>> # systemctl start syslog.socket rsyslog.service
>>
>> 5. log a message on the client
>> # logger -p local7.info -s 'hello world 3'
>>
>> shows up in /var/log/local7.log on the server
>> shows up in /var/log/local7.log on the client
>>
>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>
>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>
>> There must be something I'm doing wrong, but what?
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
>> Kruk via rsyslog
>> Sent: Friday, February 18, 2022 4:18 AM
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>
>> But regarding your setup - as you defined
>>
>> $WorkDirectory /var/spool/rsyslog
>>
>> Your queue should be placed there.
>>
>> Question is whether you do indeed have such directory in your system.
>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>
>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>
>> Also, notice that
>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeat
>> edmsgreduction.html "This parameter models old sysklogd legacy. *Note
>> that many people, including the rsyslog authors, consider this to be a misfeature.* See /Discussion/ below to learn why."
>>
>> But in general, the setup should work... with one caveat. Your "never"
>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>
>> See the description of parameters at
>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>
>> You might set (just for test! you probably don't want to set it in
>> prod for that often)
>>
>> $ActionResumeInterval 1
>>
>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>
>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>
>>
>>
>>
>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>
>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html
>>>
>>> I confirm that remote logging is working initially by doing
>>>
>>> # logger "hello, world"
>>>
>>> on the client, and verifying that this message shows up in the server
>>> (in this case in /var/log/syslog)
>>>
>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>
>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>
>>> Some configuration files:
>>>
>>> ---------------------------------------------------------------------
>>> -
>>> ----------------------
>>> client rsyslog.conf file:
>>>
>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>> #
>>> # For more information see
>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>> #
>>> # Default logging rules can be found in
>>> /etc/rsyslog.d/50-default.conf
>>>
>>>
>>> #################
>>> #### MODULES ####
>>> #################
>>>
>>> module(load="imuxsock") # provides support for local system logging
>>> #module(load="immark") # provides --MARK-- message capability
>>>
>>> # provides UDP syslog reception
>>> #module(load="imudp")
>>> #input(type="imudp" port="514")
>>>
>>> # provides TCP syslog reception
>>> #module(load="imtcp")
>>> #input(type="imtcp" port="514")
>>>
>>> # provides kernel logging support and enable non-kernel klog messages
>>> module(load="imklog" permitnonkernelfacility="on")
>>>
>>> ###########################
>>> #### GLOBAL DIRECTIVES ####
>>> ###########################
>>>
>>> #
>>> # Use traditional timestamp format.
>>> # To enable high precision timestamps, comment out the following line.
>>> #
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Filter duplicated messages
>>> $RepeatedMsgReduction on
>>>
>>> #
>>> # Set the default permissions for all log files.
>>> #
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> #
>>> # Where to place spool and state files # $WorkDirectory
>>> /var/spool/rsyslog
>>>
>>> #
>>> # setup reliable local buffering
>>> #
>>> $ActionQueueType LinkedList # use asynchronous processing
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>> down
>>>
>>> #
>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>> /etc/rsyslog.d/*.conf
>>> *.* @@<redacted>:514
>>>
>>> ------------------------------------------------------------------
>>> server rsyslog.conf file
>>>
>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>> #
>>> # For more information see
>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>> #
>>> # Default logging rules can be found in
>>> /etc/rsyslog.d/50-default.conf
>>>
>>>
>>> #################
>>> #### MODULES ####
>>> #################
>>>
>>> module(load="imuxsock") # provides support for local system logging
>>> #module(load="immark") # provides --MARK-- message capability
>>>
>>> # provides UDP syslog reception
>>> #module(load="imudp")
>>> #input(type="imudp" port="514")
>>>
>>> # provides TCP syslog reception
>>> module(load="imtcp")
>>> input(type="imtcp" port="514")
>>>
>>> # provides kernel logging support and enable non-kernel klog messages
>>> module(load="imklog" permitnonkernelfacility="on")
>>>
>>> ###########################
>>> #### GLOBAL DIRECTIVES ####
>>> ###########################
>>>
>>> #
>>> # Use traditional timestamp format.
>>> # To enable high precision timestamps, comment out the following line.
>>> #
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Filter duplicated messages
>>> $RepeatedMsgReduction on
>>>
>>> #
>>> # Set the default permissions for all log files.
>>> #
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> #
>>> # Where to place spool and state files # $WorkDirectory
>>> /var/spool/rsyslog
>>>
>>> #
>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>> /etc/rsyslog.d/*.conf
>>>
>>> ------------------------------------------------------------------
>>> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>>>
>>> # rsyslogd -version (same version for both client and server)
>>>
>>> rsyslogd 8.32.0, compiled with:
>>> PLATFORM: x86_64-pc-linux-gnu
>>> PLATFORM (lsb_release -d):
>>> FEATURE_REGEXP: Yes
>>> GSSAPI Kerberos 5 support: Yes
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported: Yes
>>> 64bit Atomic operations supported: Yes
>>> memory allocator: system default
>>> Runtime Instrumentation (slow code): No
>>> uuid support: Yes
>>> systemd support: Yes
>>> Number of Bits in RainerScript integers: 64
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
So there's still something I'm not understanding about DA queues.

In my configuration, I have
$ActionQueueSize 1000
$WorkDirectory /var/spool/rsyslog
$ActionQueueFileName srvrfwd # set file name, also enables disk mode


If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.

If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.

I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: Tuesday, February 22, 2022 8:44 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".

Anyway, as David wrote somewhere in this thread - legacy config format is OK for simple setups where it's more readable than Rainer Script but if your config requires multiple directives modifying functionality of the action, it's probably easier to write it as (in your case)

if ($syslogfacility == "local7") then
    action(type="omfwd" Target="wll" Port="2514"
action.resumeRetryCount="0" [... more action.parameters and queue.parameters ...] )

It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).

MK

PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.

On 22.02.2022 14:31, MACGREGOR Will wrote:
> What I found was that I had to do this in 50-default.conf:
>
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
> down
>
> local7.* :omrelp:will:2514
>
> I believe that's what you meant here, yes?
>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>
> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
> Kruk via rsyslog
> Sent: Friday, February 18, 2022 3:22 PM
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
> Messages with Rsyslog
>
> If you run a client as
>
> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>
> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>
> But when you stop the server, the client should start emiting messages
> like
>
> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
> ResumeInRow 1
> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
> actionDoRetry: action-0-builtin:omfwd check for max retries,
> iResumeRetryCount -1, iRetries 0
>
> And if you look for the string '<somenumber> messages' in debug log,
> if you close the client some time after stopping the server and
> pushing some more messages to the client, you should get something
> like
>
> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
> queue holds 2 messages after shutdown of workers. queue.saveonshutdown
> is set, so data will now be spooled to disk [v8.2102.0-4.fc35 try
> https://www.rsyslog.com/e/2041 ]
>
> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>
> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>> So, following your advice, I've confirmed the following
>>
>> 1. I switched to RELP. as per the following:
>>
>> add the following to server rsyslog.conf
>>
>> module(load="imrelp")
>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>
>> add the following to server 50-default.conf:
>>
>> local7.* -/var/log/local7.log
>>
>> add the following to client 50-default.conf
>>
>> local7.* -/var/log/local7.log
>> local7.* :omrelp:<server>:2514
>>
>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>
>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>
>> Here's exactly how I tested:
>>
>> 1. log a message from the client, verify that it shows up on the server
>> # logger -p local7.info -s 'hello world'
>>
>> shows up in /var/log/local7.log on the server
>> shows up in /var/log/local7.log on the client
>>
>> 2. disable rsyslog on the server
>> # systemctl stop syslog.socket rsyslog.service
>>
>> 3. log a message on the client
>> # logger -p local7.info -s 'hello world 2'
>>
>> shows up in /var/log/local7.log on the client
>>
>> 4. enable rsyslog on the server
>> # systemctl start syslog.socket rsyslog.service
>>
>> 5. log a message on the client
>> # logger -p local7.info -s 'hello world 3'
>>
>> shows up in /var/log/local7.log on the server
>> shows up in /var/log/local7.log on the client
>>
>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>
>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>
>> There must be something I'm doing wrong, but what?
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Mariusz Kruk via rsyslog
>> Sent: Friday, February 18, 2022 4:18 AM
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>
>> But regarding your setup - as you defined
>>
>> $WorkDirectory /var/spool/rsyslog
>>
>> Your queue should be placed there.
>>
>> Question is whether you do indeed have such directory in your system.
>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>
>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>
>> Also, notice that
>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repea
>> t edmsgreduction.html "This parameter models old sysklogd legacy.
>> *Note that many people, including the rsyslog authors, consider this
>> to be a misfeature.* See /Discussion/ below to learn why."
>>
>> But in general, the setup should work... with one caveat. Your "never"
>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>
>> See the description of parameters at
>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>
>> You might set (just for test! you probably don't want to set it in
>> prod for that often)
>>
>> $ActionResumeInterval 1
>>
>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>
>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>
>>
>>
>>
>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>
>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.htm
>>> l
>>>
>>> I confirm that remote logging is working initially by doing
>>>
>>> # logger "hello, world"
>>>
>>> on the client, and verifying that this message shows up in the
>>> server (in this case in /var/log/syslog)
>>>
>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>
>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>
>>> Some configuration files:
>>>
>>> --------------------------------------------------------------------
>>> -
>>> -
>>> ----------------------
>>> client rsyslog.conf file:
>>>
>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>> #
>>> # For more information see
>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>> #
>>> # Default logging rules can be found in
>>> /etc/rsyslog.d/50-default.conf
>>>
>>>
>>> #################
>>> #### MODULES ####
>>> #################
>>>
>>> module(load="imuxsock") # provides support for local system logging
>>> #module(load="immark") # provides --MARK-- message capability
>>>
>>> # provides UDP syslog reception
>>> #module(load="imudp")
>>> #input(type="imudp" port="514")
>>>
>>> # provides TCP syslog reception
>>> #module(load="imtcp")
>>> #input(type="imtcp" port="514")
>>>
>>> # provides kernel logging support and enable non-kernel klog
>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>
>>> ###########################
>>> #### GLOBAL DIRECTIVES ####
>>> ###########################
>>>
>>> #
>>> # Use traditional timestamp format.
>>> # To enable high precision timestamps, comment out the following line.
>>> #
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Filter duplicated messages
>>> $RepeatedMsgReduction on
>>>
>>> #
>>> # Set the default permissions for all log files.
>>> #
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> #
>>> # Where to place spool and state files # $WorkDirectory
>>> /var/spool/rsyslog
>>>
>>> #
>>> # setup reliable local buffering
>>> #
>>> $ActionQueueType LinkedList # use asynchronous processing
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>> down
>>>
>>> #
>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>> /etc/rsyslog.d/*.conf
>>> *.* @@<redacted>:514
>>>
>>> ------------------------------------------------------------------
>>> server rsyslog.conf file
>>>
>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>> #
>>> # For more information see
>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>> #
>>> # Default logging rules can be found in
>>> /etc/rsyslog.d/50-default.conf
>>>
>>>
>>> #################
>>> #### MODULES ####
>>> #################
>>>
>>> module(load="imuxsock") # provides support for local system logging
>>> #module(load="immark") # provides --MARK-- message capability
>>>
>>> # provides UDP syslog reception
>>> #module(load="imudp")
>>> #input(type="imudp" port="514")
>>>
>>> # provides TCP syslog reception
>>> module(load="imtcp")
>>> input(type="imtcp" port="514")
>>>
>>> # provides kernel logging support and enable non-kernel klog
>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>
>>> ###########################
>>> #### GLOBAL DIRECTIVES ####
>>> ###########################
>>>
>>> #
>>> # Use traditional timestamp format.
>>> # To enable high precision timestamps, comment out the following line.
>>> #
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Filter duplicated messages
>>> $RepeatedMsgReduction on
>>>
>>> #
>>> # Set the default permissions for all log files.
>>> #
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> #
>>> # Where to place spool and state files # $WorkDirectory
>>> /var/spool/rsyslog
>>>
>>> #
>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>> /etc/rsyslog.d/*.conf
>>>
>>> ------------------------------------------------------------------
>>> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>>>
>>> # rsyslogd -version (same version for both client and server)
>>>
>>> rsyslogd 8.32.0, compiled with:
>>> PLATFORM: x86_64-pc-linux-gnu
>>> PLATFORM (lsb_release -d):
>>> FEATURE_REGEXP: Yes
>>> GSSAPI Kerberos 5 support: Yes
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported: Yes
>>> 64bit Atomic operations supported: Yes
>>> memory allocator: system default
>>> Runtime Instrumentation (slow code): No
>>> uuid support: Yes
>>> systemd support: Yes
>>> Number of Bits in RainerScript integers: 64
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
enable impstats and post the results so that we can see what's happening with
the queues

with a DA queue you have both a memory queue and a disk queue. did you restart
the sending system while the server was down?

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:

> Date: Tue, 22 Feb 2022 16:44:58 +0000
> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> So there's still something I'm not understanding about DA queues.
>
> In my configuration, I have
> $ActionQueueSize 1000
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>
>
> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>
> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>
> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: Tuesday, February 22, 2022 8:44 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>
> Anyway, as David wrote somewhere in this thread - legacy config format is OK for simple setups where it's more readable than Rainer Script but if your config requires multiple directives modifying functionality of the action, it's probably easier to write it as (in your case)
>
> if ($syslogfacility == "local7") then
>     action(type="omfwd" Target="wll" Port="2514"
> action.resumeRetryCount="0" [... more action.parameters and queue.parameters ...] )
>
> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>
> MK
>
> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>
> On 22.02.2022 14:31, MACGREGOR Will wrote:
>> What I found was that I had to do this in 50-default.conf:
>>
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>> down
>>
>> local7.* :omrelp:will:2514
>>
>> I believe that's what you meant here, yes?
>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>
>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
>> Kruk via rsyslog
>> Sent: Friday, February 18, 2022 3:22 PM
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> If you run a client as
>>
>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>
>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>
>> But when you stop the server, the client should start emiting messages
>> like
>>
>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>> ResumeInRow 1
>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>> iResumeRetryCount -1, iRetries 0
>>
>> And if you look for the string '<somenumber> messages' in debug log,
>> if you close the client some time after stopping the server and
>> pushing some more messages to the client, you should get something
>> like
>>
>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>> queue holds 2 messages after shutdown of workers. queue.saveonshutdown
>> is set, so data will now be spooled to disk [v8.2102.0-4.fc35 try
>> https://www.rsyslog.com/e/2041 ]
>>
>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>
>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>> So, following your advice, I've confirmed the following
>>>
>>> 1. I switched to RELP. as per the following:
>>>
>>> add the following to server rsyslog.conf
>>>
>>> module(load="imrelp")
>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>
>>> add the following to server 50-default.conf:
>>>
>>> local7.* -/var/log/local7.log
>>>
>>> add the following to client 50-default.conf
>>>
>>> local7.* -/var/log/local7.log
>>> local7.* :omrelp:<server>:2514
>>>
>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>
>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>
>>> Here's exactly how I tested:
>>>
>>> 1. log a message from the client, verify that it shows up on the server
>>> # logger -p local7.info -s 'hello world'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> 2. disable rsyslog on the server
>>> # systemctl stop syslog.socket rsyslog.service
>>>
>>> 3. log a message on the client
>>> # logger -p local7.info -s 'hello world 2'
>>>
>>> shows up in /var/log/local7.log on the client
>>>
>>> 4. enable rsyslog on the server
>>> # systemctl start syslog.socket rsyslog.service
>>>
>>> 5. log a message on the client
>>> # logger -p local7.info -s 'hello world 3'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>
>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>
>>> There must be something I'm doing wrong, but what?
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Friday, February 18, 2022 4:18 AM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>
>>> But regarding your setup - as you defined
>>>
>>> $WorkDirectory /var/spool/rsyslog
>>>
>>> Your queue should be placed there.
>>>
>>> Question is whether you do indeed have such directory in your system.
>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>
>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>
>>> Also, notice that
>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repea
>>> t edmsgreduction.html "This parameter models old sysklogd legacy.
>>> *Note that many people, including the rsyslog authors, consider this
>>> to be a misfeature.* See /Discussion/ below to learn why."
>>>
>>> But in general, the setup should work... with one caveat. Your "never"
>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>
>>> See the description of parameters at
>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>
>>> You might set (just for test! you probably don't want to set it in
>>> prod for that often)
>>>
>>> $ActionResumeInterval 1
>>>
>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>
>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>
>>>
>>>
>>>
>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>
>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.htm
>>>> l
>>>>
>>>> I confirm that remote logging is working initially by doing
>>>>
>>>> # logger "hello, world"
>>>>
>>>> on the client, and verifying that this message shows up in the
>>>> server (in this case in /var/log/syslog)
>>>>
>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>
>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>
>>>> Some configuration files:
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> -
>>>> ----------------------
>>>> client rsyslog.conf file:
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> #module(load="imtcp")
>>>> #input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # setup reliable local buffering
>>>> #
>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>>> down
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>> *.* @@<redacted>:514
>>>>
>>>> ------------------------------------------------------------------
>>>> server rsyslog.conf file
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> module(load="imtcp")
>>>> input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>>
>>>> ------------------------------------------------------------------
>>>> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW)
>>>>
>>>> # rsyslogd -version (same version for both client and server)
>>>>
>>>> rsyslogd 8.32.0, compiled with:
>>>> PLATFORM: x86_64-pc-linux-gnu
>>>> PLATFORM (lsb_release -d):
>>>> FEATURE_REGEXP: Yes
>>>> GSSAPI Kerberos 5 support: Yes
>>>> FEATURE_DEBUG (debug build, slow code): No
>>>> 32bit Atomic operations supported: Yes
>>>> 64bit Atomic operations supported: Yes
>>>> memory allocator: system default
>>>> Runtime Instrumentation (slow code): No
>>>> uuid support: Yes
>>>> systemd support: Yes
>>>> Number of Bits in RainerScript integers: 64
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
I've attached the output of the impstat module for the following scenario:

1. impstat update rate is 30 seconds

2. restarted rsyslog on client, with server rsyslog is disabled

3. attempt to queue 2000 messages (just a simple 'C' program that calls syslog repeatedly)

I can see where the DA queue only gets 1120 messages, in these two entries here:

Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901

---------------impstat output----------------

Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats
Feb 22 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0 oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0 discarded.nf=0 maxqsize=41
Feb 22 12:24:51 AA3945 rsyslogd-pstats: global: origin=dynstats
Feb 22 12:24:51 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2025 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:24:51 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844 minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313 openfiles=14
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
Feb 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41
Feb 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats
Feb 22 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Feb 22 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0 oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0 discarded.nf=0 maxqsize=41

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, February 22, 2022 11:47 AM
To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

enable impstats and post the results so that we can see what's happening with the queues

with a DA queue you have both a memory queue and a disk queue. did you restart the sending system while the server was down?

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:

> Date: Tue, 22 Feb 2022 16:44:58 +0000
> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> So there's still something I'm not understanding about DA queues.
>
> In my configuration, I have
> $ActionQueueSize 1000
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>
>
> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>
> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>
> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
> Kruk via rsyslog
> Sent: Tuesday, February 22, 2022 8:44 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
> Messages with Rsyslog
>
> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>
> Anyway, as David wrote somewhere in this thread - legacy config format
> is OK for simple setups where it's more readable than Rainer Script
> but if your config requires multiple directives modifying
> functionality of the action, it's probably easier to write it as (in
> your case)
>
> if ($syslogfacility == "local7") then
>     action(type="omfwd" Target="wll" Port="2514"
> action.resumeRetryCount="0" [... more action.parameters and
> queue.parameters ...] )
>
> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>
> MK
>
> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>
> On 22.02.2022 14:31, MACGREGOR Will wrote:
>> What I found was that I had to do this in 50-default.conf:
>>
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>> down
>>
>> local7.* :omrelp:will:2514
>>
>> I believe that's what you meant here, yes?
>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>
>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Mariusz Kruk via rsyslog
>> Sent: Friday, February 18, 2022 3:22 PM
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> If you run a client as
>>
>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>
>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>
>> But when you stop the server, the client should start emiting
>> messages like
>>
>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>> ResumeInRow 1
>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>> iResumeRetryCount -1, iRetries 0
>>
>> And if you look for the string '<somenumber> messages' in debug log,
>> if you close the client some time after stopping the server and
>> pushing some more messages to the client, you should get something
>> like
>>
>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>> queue holds 2 messages after shutdown of workers.
>> queue.saveonshutdown is set, so data will now be spooled to disk
>> [v8.2102.0-4.fc35 try
>> https://www.rsyslog.com/e/2041 ]
>>
>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>
>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>> So, following your advice, I've confirmed the following
>>>
>>> 1. I switched to RELP. as per the following:
>>>
>>> add the following to server rsyslog.conf
>>>
>>> module(load="imrelp")
>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>
>>> add the following to server 50-default.conf:
>>>
>>> local7.* -/var/log/local7.log
>>>
>>> add the following to client 50-default.conf
>>>
>>> local7.* -/var/log/local7.log
>>> local7.* :omrelp:<server>:2514
>>>
>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>
>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>
>>> Here's exactly how I tested:
>>>
>>> 1. log a message from the client, verify that it shows up on the server
>>> # logger -p local7.info -s 'hello world'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> 2. disable rsyslog on the server
>>> # systemctl stop syslog.socket rsyslog.service
>>>
>>> 3. log a message on the client
>>> # logger -p local7.info -s 'hello world 2'
>>>
>>> shows up in /var/log/local7.log on the client
>>>
>>> 4. enable rsyslog on the server
>>> # systemctl start syslog.socket rsyslog.service
>>>
>>> 5. log a message on the client
>>> # logger -p local7.info -s 'hello world 3'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>
>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>
>>> There must be something I'm doing wrong, but what?
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Friday, February 18, 2022 4:18 AM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>
>>> But regarding your setup - as you defined
>>>
>>> $WorkDirectory /var/spool/rsyslog
>>>
>>> Your queue should be placed there.
>>>
>>> Question is whether you do indeed have such directory in your system.
>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>
>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>
>>> Also, notice that
>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repe
>>> a t edmsgreduction.html "This parameter models old sysklogd legacy.
>>> *Note that many people, including the rsyslog authors, consider this
>>> to be a misfeature.* See /Discussion/ below to learn why."
>>>
>>> But in general, the setup should work... with one caveat. Your "never"
>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>
>>> See the description of parameters at
>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>
>>> You might set (just for test! you probably don't want to set it in
>>> prod for that often)
>>>
>>> $ActionResumeInterval 1
>>>
>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>
>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>
>>>
>>>
>>>
>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>
>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.ht
>>>> m
>>>> l
>>>>
>>>> I confirm that remote logging is working initially by doing
>>>>
>>>> # logger "hello, world"
>>>>
>>>> on the client, and verifying that this message shows up in the
>>>> server (in this case in /var/log/syslog)
>>>>
>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>
>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>
>>>> Some configuration files:
>>>>
>>>> -------------------------------------------------------------------
>>>> -
>>>> -
>>>> -
>>>> ----------------------
>>>> client rsyslog.conf file:
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> #module(load="imtcp")
>>>> #input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # setup reliable local buffering
>>>> #
>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>> rsyslog shuts down
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>> *.* @@<redacted>:514
>>>>
>>>> ------------------------------------------------------------------
>>>> server rsyslog.conf file
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> module(load="imtcp")
>>>> input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>>
>>>> ------------------------------------------------------------------
>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>> FWIW)
>>>>
>>>> # rsyslogd -version (same version for both client and server)
>>>>
>>>> rsyslogd 8.32.0, compiled with:
>>>> PLATFORM: x86_64-pc-linux-gnu
>>>> PLATFORM (lsb_release -d):
>>>> FEATURE_REGEXP: Yes
>>>> GSSAPI Kerberos 5 support: Yes
>>>> FEATURE_DEBUG (debug build, slow code): No
>>>> 32bit Atomic operations supported: Yes
>>>> 64bit Atomic operations supported: Yes
>>>> memory allocator: system default
>>>> Runtime Instrumentation (slow code): No
>>>> uuid support: Yes
>>>> systemd support: Yes
>>>> Number of Bits in RainerScript integers: 64
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
Apologies for the horrible formatting in my last message. Attaching text file this time.



-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of MACGREGOR Will via rsyslog
Sent: Tuesday, February 22, 2022 12:38 PM
To: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

I've attached the output of the impstat module for the following scenario:

1. impstat update rate is 30 seconds

2. restarted rsyslog on client, with server rsyslog is disabled

3. attempt to queue 2000 messages (just a simple 'C' program that calls syslog repeatedly)

I can see where the DA queue only gets 1120 messages, in these two entries here:

Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901

---------------impstat output----------------

Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0 oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2025 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844 minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313 openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0 oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0 discarded.nf=0 maxqsize=41

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, February 22, 2022 11:47 AM
To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

enable impstats and post the results so that we can see what's happening with the queues

with a DA queue you have both a memory queue and a disk queue. did you restart the sending system while the server was down?

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:

> Date: Tue, 22 Feb 2022 16:44:58 +0000
> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> So there's still something I'm not understanding about DA queues.
>
> In my configuration, I have
> $ActionQueueSize 1000
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>
>
> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>
> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>
> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
> Kruk via rsyslog
> Sent: Tuesday, February 22, 2022 8:44 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
> Messages with Rsyslog
>
> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>
> Anyway, as David wrote somewhere in this thread - legacy config format
> is OK for simple setups where it's more readable than Rainer Script
> but if your config requires multiple directives modifying
> functionality of the action, it's probably easier to write it as (in
> your case)
>
> if ($syslogfacility == "local7") then
>     action(type="omfwd" Target="wll" Port="2514"
> action.resumeRetryCount="0" [... more action.parameters and
> queue.parameters ...] )
>
> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>
> MK
>
> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>
> On 22.02.2022 14:31, MACGREGOR Will wrote:
>> What I found was that I had to do this in 50-default.conf:
>>
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>> down
>>
>> local7.* :omrelp:will:2514
>>
>> I believe that's what you meant here, yes?
>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>
>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Mariusz Kruk via rsyslog
>> Sent: Friday, February 18, 2022 3:22 PM
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> If you run a client as
>>
>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>
>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>
>> But when you stop the server, the client should start emiting
>> messages like
>>
>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>> ResumeInRow 1
>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>> iResumeRetryCount -1, iRetries 0
>>
>> And if you look for the string '<somenumber> messages' in debug log,
>> if you close the client some time after stopping the server and
>> pushing some more messages to the client, you should get something
>> like
>>
>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>> queue holds 2 messages after shutdown of workers.
>> queue.saveonshutdown is set, so data will now be spooled to disk
>> [v8.2102.0-4.fc35 try
>> https://www.rsyslog.com/e/2041 ]
>>
>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>
>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>> So, following your advice, I've confirmed the following
>>>
>>> 1. I switched to RELP. as per the following:
>>>
>>> add the following to server rsyslog.conf
>>>
>>> module(load="imrelp")
>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>
>>> add the following to server 50-default.conf:
>>>
>>> local7.* -/var/log/local7.log
>>>
>>> add the following to client 50-default.conf
>>>
>>> local7.* -/var/log/local7.log
>>> local7.* :omrelp:<server>:2514
>>>
>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>
>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>
>>> Here's exactly how I tested:
>>>
>>> 1. log a message from the client, verify that it shows up on the server
>>> # logger -p local7.info -s 'hello world'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> 2. disable rsyslog on the server
>>> # systemctl stop syslog.socket rsyslog.service
>>>
>>> 3. log a message on the client
>>> # logger -p local7.info -s 'hello world 2'
>>>
>>> shows up in /var/log/local7.log on the client
>>>
>>> 4. enable rsyslog on the server
>>> # systemctl start syslog.socket rsyslog.service
>>>
>>> 5. log a message on the client
>>> # logger -p local7.info -s 'hello world 3'
>>>
>>> shows up in /var/log/local7.log on the server
>>> shows up in /var/log/local7.log on the client
>>>
>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>
>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>
>>> There must be something I'm doing wrong, but what?
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Friday, February 18, 2022 4:18 AM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>
>>> But regarding your setup - as you defined
>>>
>>> $WorkDirectory /var/spool/rsyslog
>>>
>>> Your queue should be placed there.
>>>
>>> Question is whether you do indeed have such directory in your system.
>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>
>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>
>>> Also, notice that
>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repe
>>> a t edmsgreduction.html "This parameter models old sysklogd legacy.
>>> *Note that many people, including the rsyslog authors, consider this
>>> to be a misfeature.* See /Discussion/ below to learn why."
>>>
>>> But in general, the setup should work... with one caveat. Your "never"
>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>
>>> See the description of parameters at
>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>
>>> You might set (just for test! you probably don't want to set it in
>>> prod for that often)
>>>
>>> $ActionResumeInterval 1
>>>
>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>
>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>
>>>
>>>
>>>
>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>
>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.ht
>>>> m
>>>> l
>>>>
>>>> I confirm that remote logging is working initially by doing
>>>>
>>>> # logger "hello, world"
>>>>
>>>> on the client, and verifying that this message shows up in the
>>>> server (in this case in /var/log/syslog)
>>>>
>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>
>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>
>>>> Some configuration files:
>>>>
>>>> -------------------------------------------------------------------
>>>> -
>>>> -
>>>> -
>>>> ----------------------
>>>> client rsyslog.conf file:
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> #module(load="imtcp")
>>>> #input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # setup reliable local buffering
>>>> #
>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>> rsyslog shuts down
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>> *.* @@<redacted>:514
>>>>
>>>> ------------------------------------------------------------------
>>>> server rsyslog.conf file
>>>>
>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>> #
>>>> # For more information see
>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>> #
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>> #################
>>>> #### MODULES ####
>>>> #################
>>>>
>>>> module(load="imuxsock") # provides support for local system logging
>>>> #module(load="immark") # provides --MARK-- message capability
>>>>
>>>> # provides UDP syslog reception
>>>> #module(load="imudp")
>>>> #input(type="imudp" port="514")
>>>>
>>>> # provides TCP syslog reception
>>>> module(load="imtcp")
>>>> input(type="imtcp" port="514")
>>>>
>>>> # provides kernel logging support and enable non-kernel klog
>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>
>>>> ###########################
>>>> #### GLOBAL DIRECTIVES ####
>>>> ###########################
>>>>
>>>> #
>>>> # Use traditional timestamp format.
>>>> # To enable high precision timestamps, comment out the following line.
>>>> #
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Filter duplicated messages
>>>> $RepeatedMsgReduction on
>>>>
>>>> #
>>>> # Set the default permissions for all log files.
>>>> #
>>>> $FileOwner syslog
>>>> $FileGroup adm
>>>> $FileCreateMode 0640
>>>> $DirCreateMode 0755
>>>> $Umask 0022
>>>> $PrivDropToUser syslog
>>>> $PrivDropToGroup syslog
>>>>
>>>> #
>>>> # Where to place spool and state files # $WorkDirectory
>>>> /var/spool/rsyslog
>>>>
>>>> #
>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>> /etc/rsyslog.d/*.conf
>>>>
>>>> ------------------------------------------------------------------
>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>> FWIW)
>>>>
>>>> # rsyslogd -version (same version for both client and server)
>>>>
>>>> rsyslogd 8.32.0, compiled with:
>>>> PLATFORM: x86_64-pc-linux-gnu
>>>> PLATFORM (lsb_release -d):
>>>> FEATURE_REGEXP: Yes
>>>> GSSAPI Kerberos 5 support: Yes
>>>> FEATURE_DEBUG (debug build, slow code): No
>>>> 32bit Atomic operations supported: Yes
>>>> 64bit Atomic operations supported: Yes
>>>> memory allocator: system default
>>>> Runtime Instrumentation (slow code): No
>>>> uuid support: Yes
>>>> systemd support: Yes
>>>> Number of Bits in RainerScript integers: 64
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
if you look there is an action 4 queue that also has 880 items in it, that's the
rest of them. That's the memory queue. those should also be delivered once the
link comes back up.

what does pstats show after you bring the server back up?

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will wrote:

> Date: Tue, 22 Feb 2022 17:37:40 +0000
> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
> To: David Lang <david@lang.hm>,
> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> I've attached the output of the impstat module for the following scenario:
>
> 1. impstat update rate is 30 seconds
>
> 2. restarted rsyslog on client, with server rsyslog is disabled
>
> 3. attempt to queue 2000 messages (just a simple 'C' program that calls syslog repeatedly)
>
> I can see where the DA queue only gets 1120 messages, in these two entries here:
>
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
>
> ---------------impstat output----------------
>
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0 oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0 discarded.nf=0 maxqsize=41
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: global: origin=dynstats
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2025 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844 minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313 openfiles=14
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
> Feb 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0 oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
> Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0 discarded.nf=0 maxqsize=41
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, February 22, 2022 11:47 AM
> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> enable impstats and post the results so that we can see what's happening with the queues
>
> with a DA queue you have both a memory queue and a disk queue. did you restart the sending system while the server was down?
>
> David Lang
>
> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>
>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
>> Rsyslog
>>
>> So there's still something I'm not understanding about DA queues.
>>
>> In my configuration, I have
>> $ActionQueueSize 1000
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>
>>
>> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>>
>> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>>
>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
>> Kruk via rsyslog
>> Sent: Tuesday, February 22, 2022 8:44 AM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>>
>> Anyway, as David wrote somewhere in this thread - legacy config format
>> is OK for simple setups where it's more readable than Rainer Script
>> but if your config requires multiple directives modifying
>> functionality of the action, it's probably easier to write it as (in
>> your case)
>>
>> if ($syslogfacility == "local7") then
>>     action(type="omfwd" Target="wll" Port="2514"
>> action.resumeRetryCount="0" [... more action.parameters and
>> queue.parameters ...] )
>>
>> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>>
>> MK
>>
>> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>>
>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>> What I found was that I had to do this in 50-default.conf:
>>>
>>> $ActionQueueType LinkedList # use asynchronous processing
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>> down
>>>
>>> local7.* :omrelp:will:2514
>>>
>>> I believe that's what you meant here, yes?
>>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>>
>>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Friday, February 18, 2022 3:22 PM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> If you run a client as
>>>
>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>
>>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>>
>>> But when you stop the server, the client should start emiting
>>> messages like
>>>
>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>> ResumeInRow 1
>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>> iResumeRetryCount -1, iRetries 0
>>>
>>> And if you look for the string '<somenumber> messages' in debug log,
>>> if you close the client some time after stopping the server and
>>> pushing some more messages to the client, you should get something
>>> like
>>>
>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>> queue holds 2 messages after shutdown of workers.
>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>> [v8.2102.0-4.fc35 try
>>> https://www.rsyslog.com/e/2041 ]
>>>
>>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>>
>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>> So, following your advice, I've confirmed the following
>>>>
>>>> 1. I switched to RELP. as per the following:
>>>>
>>>> add the following to server rsyslog.conf
>>>>
>>>> module(load="imrelp")
>>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>>
>>>> add the following to server 50-default.conf:
>>>>
>>>> local7.* -/var/log/local7.log
>>>>
>>>> add the following to client 50-default.conf
>>>>
>>>> local7.* -/var/log/local7.log
>>>> local7.* :omrelp:<server>:2514
>>>>
>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>>
>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>>
>>>> Here's exactly how I tested:
>>>>
>>>> 1. log a message from the client, verify that it shows up on the server
>>>> # logger -p local7.info -s 'hello world'
>>>>
>>>> shows up in /var/log/local7.log on the server
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> 2. disable rsyslog on the server
>>>> # systemctl stop syslog.socket rsyslog.service
>>>>
>>>> 3. log a message on the client
>>>> # logger -p local7.info -s 'hello world 2'
>>>>
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> 4. enable rsyslog on the server
>>>> # systemctl start syslog.socket rsyslog.service
>>>>
>>>> 5. log a message on the client
>>>> # logger -p local7.info -s 'hello world 3'
>>>>
>>>> shows up in /var/log/local7.log on the server
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>>
>>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>>
>>>> There must be something I'm doing wrong, but what?
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Mariusz Kruk via rsyslog
>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>> To: rsyslog@lists.adiscon.com
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>>
>>>> But regarding your setup - as you defined
>>>>
>>>> $WorkDirectory /var/spool/rsyslog
>>>>
>>>> Your queue should be placed there.
>>>>
>>>> Question is whether you do indeed have such directory in your system.
>>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>>
>>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>>
>>>> Also, notice that
>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repe
>>>> a t edmsgreduction.html "This parameter models old sysklogd legacy.
>>>> *Note that many people, including the rsyslog authors, consider this
>>>> to be a misfeature.* See /Discussion/ below to learn why."
>>>>
>>>> But in general, the setup should work... with one caveat. Your "never"
>>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>>
>>>> See the description of parameters at
>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>
>>>> You might set (just for test! you probably don't want to set it in
>>>> prod for that often)
>>>>
>>>> $ActionResumeInterval 1
>>>>
>>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>>
>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>
>>>>
>>>>
>>>>
>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>>
>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.ht
>>>>> m
>>>>> l
>>>>>
>>>>> I confirm that remote logging is working initially by doing
>>>>>
>>>>> # logger "hello, world"
>>>>>
>>>>> on the client, and verifying that this message shows up in the
>>>>> server (in this case in /var/log/syslog)
>>>>>
>>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>>
>>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>>
>>>>> Some configuration files:
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> -
>>>>> -
>>>>> -
>>>>> ----------------------
>>>>> client rsyslog.conf file:
>>>>>
>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>> #
>>>>> # For more information see
>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>> #
>>>>> # Default logging rules can be found in
>>>>> /etc/rsyslog.d/50-default.conf
>>>>>
>>>>>
>>>>> #################
>>>>> #### MODULES ####
>>>>> #################
>>>>>
>>>>> module(load="imuxsock") # provides support for local system logging
>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>
>>>>> # provides UDP syslog reception
>>>>> #module(load="imudp")
>>>>> #input(type="imudp" port="514")
>>>>>
>>>>> # provides TCP syslog reception
>>>>> #module(load="imtcp")
>>>>> #input(type="imtcp" port="514")
>>>>>
>>>>> # provides kernel logging support and enable non-kernel klog
>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>
>>>>> ###########################
>>>>> #### GLOBAL DIRECTIVES ####
>>>>> ###########################
>>>>>
>>>>> #
>>>>> # Use traditional timestamp format.
>>>>> # To enable high precision timestamps, comment out the following line.
>>>>> #
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Filter duplicated messages
>>>>> $RepeatedMsgReduction on
>>>>>
>>>>> #
>>>>> # Set the default permissions for all log files.
>>>>> #
>>>>> $FileOwner syslog
>>>>> $FileGroup adm
>>>>> $FileCreateMode 0640
>>>>> $DirCreateMode 0755
>>>>> $Umask 0022
>>>>> $PrivDropToUser syslog
>>>>> $PrivDropToGroup syslog
>>>>>
>>>>> #
>>>>> # Where to place spool and state files # $WorkDirectory
>>>>> /var/spool/rsyslog
>>>>>
>>>>> #
>>>>> # setup reliable local buffering
>>>>> #
>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>>> rsyslog shuts down
>>>>>
>>>>> #
>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>> /etc/rsyslog.d/*.conf
>>>>> *.* @@<redacted>:514
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> server rsyslog.conf file
>>>>>
>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>> #
>>>>> # For more information see
>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>> #
>>>>> # Default logging rules can be found in
>>>>> /etc/rsyslog.d/50-default.conf
>>>>>
>>>>>
>>>>> #################
>>>>> #### MODULES ####
>>>>> #################
>>>>>
>>>>> module(load="imuxsock") # provides support for local system logging
>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>
>>>>> # provides UDP syslog reception
>>>>> #module(load="imudp")
>>>>> #input(type="imudp" port="514")
>>>>>
>>>>> # provides TCP syslog reception
>>>>> module(load="imtcp")
>>>>> input(type="imtcp" port="514")
>>>>>
>>>>> # provides kernel logging support and enable non-kernel klog
>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>
>>>>> ###########################
>>>>> #### GLOBAL DIRECTIVES ####
>>>>> ###########################
>>>>>
>>>>> #
>>>>> # Use traditional timestamp format.
>>>>> # To enable high precision timestamps, comment out the following line.
>>>>> #
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Filter duplicated messages
>>>>> $RepeatedMsgReduction on
>>>>>
>>>>> #
>>>>> # Set the default permissions for all log files.
>>>>> #
>>>>> $FileOwner syslog
>>>>> $FileGroup adm
>>>>> $FileCreateMode 0640
>>>>> $DirCreateMode 0755
>>>>> $Umask 0022
>>>>> $PrivDropToUser syslog
>>>>> $PrivDropToGroup syslog
>>>>>
>>>>> #
>>>>> # Where to place spool and state files # $WorkDirectory
>>>>> /var/spool/rsyslog
>>>>>
>>>>> #
>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>> /etc/rsyslog.d/*.conf
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>>> FWIW)
>>>>>
>>>>> # rsyslogd -version (same version for both client and server)
>>>>>
>>>>> rsyslogd 8.32.0, compiled with:
>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>> PLATFORM (lsb_release -d):
>>>>> FEATURE_REGEXP: Yes
>>>>> GSSAPI Kerberos 5 support: Yes
>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>> 32bit Atomic operations supported: Yes
>>>>> 64bit Atomic operations supported: Yes
>>>>> memory allocator: system default
>>>>> Runtime Instrumentation (slow code): No
>>>>> uuid support: Yes
>>>>> systemd support: Yes
>>>>> Number of Bits in RainerScript integers: 64
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
So in this case, I _think_ this shows the queue was holding 1152 messages, the memory queue was holding 848, then after starting the server, the memory queue appears to get emptied - if that's what this line means:

Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0 maxqsize=925

But only the first 1152 messages ever come out on the server.

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, February 22, 2022 12:58 PM
To: MACGREGOR Will <will.macgregor@thalesgroup.com>
Cc: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

if you look there is an action 4 queue that also has 880 items in it, that's the rest of them. That's the memory queue. those should also be delivered once the link comes back up.

what does pstats show after you bring the server back up?

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will wrote:

> Date: Tue, 22 Feb 2022 17:37:40 +0000
> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
> To: David Lang <david@lang.hm>,
> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> I've attached the output of the impstat module for the following scenario:
>
> 1. impstat update rate is 30 seconds
>
> 2. restarted rsyslog on client, with server rsyslog is disabled
>
> 3. attempt to queue 2000 messages (just a simple 'C' program that
> calls syslog repeatedly)
>
> I can see where the DA queue only gets 1120 messages, in these two entries here:
>
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]:
> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0
> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats:
> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0
> discarded.full=0 discarded.nf=0 maxqsize=901
>
> ---------------impstat output----------------
>
> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0
> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21 AA3945
> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q:
> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0
> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0
> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 0: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 1: origin=core.action processed=16 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 2: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 3: origin=core.action processed=2000 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 4: origin=core.action processed=2000 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 5: origin=core.action processed=2025 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 6: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 7: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 8: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> action 9: origin=core.action processed=0 failed=0 suspended=0
> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844
> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313
> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4
> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0
> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945
> rsyslogd-pstats: action 4 queue: origin=core.queue size=880
> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb
> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15
> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0
> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21 AA3945
> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q:
> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0
> discarded.nf=0 maxqsize=41
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, February 22, 2022 11:47 AM
> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
> Messages with Rsyslog
>
> enable impstats and post the results so that we can see what's
> happening with the queues
>
> with a DA queue you have both a memory queue and a disk queue. did you restart the sending system while the server was down?
>
> David Lang
>
> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>
>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
>> Rsyslog
>>
>> So there's still something I'm not understanding about DA queues.
>>
>> In my configuration, I have
>> $ActionQueueSize 1000
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>
>>
>> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>>
>> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>>
>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Mariusz Kruk via rsyslog
>> Sent: Tuesday, February 22, 2022 8:44 AM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>>
>> Anyway, as David wrote somewhere in this thread - legacy config
>> format is OK for simple setups where it's more readable than Rainer
>> Script but if your config requires multiple directives modifying
>> functionality of the action, it's probably easier to write it as (in
>> your case)
>>
>> if ($syslogfacility == "local7") then
>>     action(type="omfwd" Target="wll" Port="2514"
>> action.resumeRetryCount="0" [... more action.parameters and
>> queue.parameters ...] )
>>
>> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>>
>> MK
>>
>> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>>
>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>> What I found was that I had to do this in 50-default.conf:
>>>
>>> $ActionQueueType LinkedList # use asynchronous processing
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>> down
>>>
>>> local7.* :omrelp:will:2514
>>>
>>> I believe that's what you meant here, yes?
>>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>>
>>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Friday, February 18, 2022 3:22 PM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> If you run a client as
>>>
>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>
>>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>>
>>> But when you stop the server, the client should start emiting
>>> messages like
>>>
>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>> ResumeInRow 1
>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>> iResumeRetryCount -1, iRetries 0
>>>
>>> And if you look for the string '<somenumber> messages' in debug log,
>>> if you close the client some time after stopping the server and
>>> pushing some more messages to the client, you should get something
>>> like
>>>
>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>> queue holds 2 messages after shutdown of workers.
>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>> [v8.2102.0-4.fc35 try
>>> https://www.rsyslog.com/e/2041 ]
>>>
>>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>>
>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>> So, following your advice, I've confirmed the following
>>>>
>>>> 1. I switched to RELP. as per the following:
>>>>
>>>> add the following to server rsyslog.conf
>>>>
>>>> module(load="imrelp")
>>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>>
>>>> add the following to server 50-default.conf:
>>>>
>>>> local7.* -/var/log/local7.log
>>>>
>>>> add the following to client 50-default.conf
>>>>
>>>> local7.* -/var/log/local7.log
>>>> local7.* :omrelp:<server>:2514
>>>>
>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>>
>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>>
>>>> Here's exactly how I tested:
>>>>
>>>> 1. log a message from the client, verify that it shows up on the server
>>>> # logger -p local7.info -s 'hello world'
>>>>
>>>> shows up in /var/log/local7.log on the server
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> 2. disable rsyslog on the server
>>>> # systemctl stop syslog.socket rsyslog.service
>>>>
>>>> 3. log a message on the client
>>>> # logger -p local7.info -s 'hello world 2'
>>>>
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> 4. enable rsyslog on the server
>>>> # systemctl start syslog.socket rsyslog.service
>>>>
>>>> 5. log a message on the client
>>>> # logger -p local7.info -s 'hello world 3'
>>>>
>>>> shows up in /var/log/local7.log on the server
>>>> shows up in /var/log/local7.log on the client
>>>>
>>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>>
>>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>>
>>>> There must be something I'm doing wrong, but what?
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Mariusz Kruk via rsyslog
>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>> To: rsyslog@lists.adiscon.com
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>>
>>>> But regarding your setup - as you defined
>>>>
>>>> $WorkDirectory /var/spool/rsyslog
>>>>
>>>> Your queue should be placed there.
>>>>
>>>> Question is whether you do indeed have such directory in your system.
>>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>>
>>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>>
>>>> Also, notice that
>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_rep
>>>> e a t edmsgreduction.html "This parameter models old sysklogd
>>>> legacy.
>>>> *Note that many people, including the rsyslog authors, consider
>>>> this to be a misfeature.* See /Discussion/ below to learn why."
>>>>
>>>> But in general, the setup should work... with one caveat. Your "never"
>>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>>
>>>> See the description of parameters at
>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>
>>>> You might set (just for test! you probably don't want to set it in
>>>> prod for that often)
>>>>
>>>> $ActionResumeInterval 1
>>>>
>>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>>
>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>
>>>>
>>>>
>>>>
>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>>
>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.h
>>>>> t
>>>>> m
>>>>> l
>>>>>
>>>>> I confirm that remote logging is working initially by doing
>>>>>
>>>>> # logger "hello, world"
>>>>>
>>>>> on the client, and verifying that this message shows up in the
>>>>> server (in this case in /var/log/syslog)
>>>>>
>>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>>
>>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>>
>>>>> Some configuration files:
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> -
>>>>> -
>>>>> -
>>>>> -
>>>>> ----------------------
>>>>> client rsyslog.conf file:
>>>>>
>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>> #
>>>>> # For more information see
>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>> #
>>>>> # Default logging rules can be found in
>>>>> /etc/rsyslog.d/50-default.conf
>>>>>
>>>>>
>>>>> #################
>>>>> #### MODULES ####
>>>>> #################
>>>>>
>>>>> module(load="imuxsock") # provides support for local system
>>>>> logging
>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>
>>>>> # provides UDP syslog reception
>>>>> #module(load="imudp")
>>>>> #input(type="imudp" port="514")
>>>>>
>>>>> # provides TCP syslog reception
>>>>> #module(load="imtcp")
>>>>> #input(type="imtcp" port="514")
>>>>>
>>>>> # provides kernel logging support and enable non-kernel klog
>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>
>>>>> ###########################
>>>>> #### GLOBAL DIRECTIVES ####
>>>>> ###########################
>>>>>
>>>>> #
>>>>> # Use traditional timestamp format.
>>>>> # To enable high precision timestamps, comment out the following line.
>>>>> #
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Filter duplicated messages
>>>>> $RepeatedMsgReduction on
>>>>>
>>>>> #
>>>>> # Set the default permissions for all log files.
>>>>> #
>>>>> $FileOwner syslog
>>>>> $FileGroup adm
>>>>> $FileCreateMode 0640
>>>>> $DirCreateMode 0755
>>>>> $Umask 0022
>>>>> $PrivDropToUser syslog
>>>>> $PrivDropToGroup syslog
>>>>>
>>>>> #
>>>>> # Where to place spool and state files # $WorkDirectory
>>>>> /var/spool/rsyslog
>>>>>
>>>>> #
>>>>> # setup reliable local buffering
>>>>> #
>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>>> rsyslog shuts down
>>>>>
>>>>> #
>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>> /etc/rsyslog.d/*.conf
>>>>> *.* @@<redacted>:514
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> server rsyslog.conf file
>>>>>
>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>> #
>>>>> # For more information see
>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>> #
>>>>> # Default logging rules can be found in
>>>>> /etc/rsyslog.d/50-default.conf
>>>>>
>>>>>
>>>>> #################
>>>>> #### MODULES ####
>>>>> #################
>>>>>
>>>>> module(load="imuxsock") # provides support for local system
>>>>> logging
>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>
>>>>> # provides UDP syslog reception
>>>>> #module(load="imudp")
>>>>> #input(type="imudp" port="514")
>>>>>
>>>>> # provides TCP syslog reception
>>>>> module(load="imtcp")
>>>>> input(type="imtcp" port="514")
>>>>>
>>>>> # provides kernel logging support and enable non-kernel klog
>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>
>>>>> ###########################
>>>>> #### GLOBAL DIRECTIVES ####
>>>>> ###########################
>>>>>
>>>>> #
>>>>> # Use traditional timestamp format.
>>>>> # To enable high precision timestamps, comment out the following line.
>>>>> #
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Filter duplicated messages
>>>>> $RepeatedMsgReduction on
>>>>>
>>>>> #
>>>>> # Set the default permissions for all log files.
>>>>> #
>>>>> $FileOwner syslog
>>>>> $FileGroup adm
>>>>> $FileCreateMode 0640
>>>>> $DirCreateMode 0755
>>>>> $Umask 0022
>>>>> $PrivDropToUser syslog
>>>>> $PrivDropToGroup syslog
>>>>>
>>>>> #
>>>>> # Where to place spool and state files # $WorkDirectory
>>>>> /var/spool/rsyslog
>>>>>
>>>>> #
>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>> /etc/rsyslog.d/*.conf
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>>> FWIW)
>>>>>
>>>>> # rsyslogd -version (same version for both client and server)
>>>>>
>>>>> rsyslogd 8.32.0, compiled with:
>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>> PLATFORM (lsb_release -d):
>>>>> FEATURE_REGEXP: Yes
>>>>> GSSAPI Kerberos 5 support: Yes
>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>> 32bit Atomic operations supported: Yes
>>>>> 64bit Atomic operations supported: Yes
>>>>> memory allocator: system default
>>>>> Runtime Instrumentation (slow code): No
>>>>> uuid support: Yes
>>>>> systemd support: Yes
>>>>> Number of Bits in RainerScript integers: 64
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
I'll have to look at the attachment later, but what does it show about the
number of items processed by action 4? and can you get a similar stats dump from
the system it's sending to?

if you switch to the new format, one advantage is that the action() statement
lets you give it a name, much easier to figure out what's what rather than just
'action 4'

David Lang

On Tue, 22 Feb 2022, MACGREGOR Will wrote:

> Date: Tue, 22 Feb 2022 18:38:05 +0000
> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
> To: David Lang <david@lang.hm>
> Cc: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> So in this case, I _think_ this shows the queue was holding 1152 messages, the memory queue was holding 848, then after starting the server, the memory queue appears to get emptied - if that's what this line means:
>
> Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0 maxqsize=925
>
> But only the first 1152 messages ever come out on the server.
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, February 22, 2022 12:58 PM
> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Cc: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> if you look there is an action 4 queue that also has 880 items in it, that's the rest of them. That's the memory queue. those should also be delivered once the link comes back up.
>
> what does pstats show after you bring the server back up?
>
> David Lang
>
> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>
>> Date: Tue, 22 Feb 2022 17:37:40 +0000
>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> To: David Lang <david@lang.hm>,
>> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
>> Rsyslog
>>
>> I've attached the output of the impstat module for the following scenario:
>>
>> 1. impstat update rate is 30 seconds
>>
>> 2. restarted rsyslog on client, with server rsyslog is disabled
>>
>> 3. attempt to queue 2000 messages (just a simple 'C' program that
>> calls syslog repeatedly)
>>
>> I can see where the DA queue only gets 1120 messages, in these two entries here:
>>
>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]:
>> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0
>> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats:
>> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=901
>>
>> ---------------impstat output----------------
>>
>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0
>> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21 AA3945
>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q:
>> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0
>> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0
>> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 0: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 1: origin=core.action processed=16 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 2: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 3: origin=core.action processed=2000 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 4: origin=core.action processed=2000 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 5: origin=core.action processed=2025 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 6: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 7: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 8: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> action 9: origin=core.action processed=0 failed=0 suspended=0
>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844
>> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313
>> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4
>> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0
>> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945
>> rsyslogd-pstats: action 4 queue: origin=core.queue size=880
>> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb
>> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15
>> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0
>> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21 AA3945
>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q:
>> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0
>> discarded.nf=0 maxqsize=41
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, February 22, 2022 11:47 AM
>> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> enable impstats and post the results so that we can see what's
>> happening with the queues
>>
>> with a DA queue you have both a memory queue and a disk queue. did you restart the sending system while the server was down?
>>
>> David Lang
>>
>> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>>
>>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with
>>> Rsyslog
>>>
>>> So there's still something I'm not understanding about DA queues.
>>>
>>> In my configuration, I have
>>> $ActionQueueSize 1000
>>> $WorkDirectory /var/spool/rsyslog
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>>
>>>
>>> If I disable the server, queue < 1000 messages, then re-enable the server, all messages are delivered.
>>>
>>> If I disable the server, queue 2000 messages, then re-enable the server, only 1120 messages get delivered.
>>>
>>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created, but it seems as if it does not contain anything beyond message 1120. It's like a lot of the messages didn't get flushed to the disk queue...
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: Tuesday, February 22, 2022 8:44 AM
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> Not exactly, because with "creating a spearate ruleset" I meant a completely different RainerScript-based configuration but this one should also work as I wrote "somewhere around".
>>>
>>> Anyway, as David wrote somewhere in this thread - legacy config
>>> format is OK for simple setups where it's more readable than Rainer
>>> Script but if your config requires multiple directives modifying
>>> functionality of the action, it's probably easier to write it as (in
>>> your case)
>>>
>>> if ($syslogfacility == "local7") then
>>>     action(type="omfwd" Target="wll" Port="2514"
>>> action.resumeRetryCount="0" [... more action.parameters and
>>> queue.parameters ...] )
>>>
>>> It's more obvious then what the parameters are for and you don't have them scattered around (possibly intertwining with other parameters modifying the resulting config).
>>>
>>> MK
>>>
>>> PS: I'm not sure if this condition will work this way; there was some bug lately about textual representation but I don't recall if it was facility or severity or both.
>>>
>>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>>> What I found was that I had to do this in 50-default.conf:
>>>>
>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>>> down
>>>>
>>>> local7.* :omrelp:will:2514
>>>>
>>>> I believe that's what you meant here, yes?
>>>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>>> When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help.
>>>>
>>>> I wish I understood the configuration better. I have to admit, I find the documentation really confusing.
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Mariusz Kruk via rsyslog
>>>> Sent: Friday, February 18, 2022 3:22 PM
>>>> To: rsyslog@lists.adiscon.com
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> If you run a client as
>>>>
>>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>>
>>>> You should see some text blob at the start but then, when the server is running, the client should not emit any more messages.
>>>>
>>>> But when you stop the server, the client should start emiting
>>>> messages like
>>>>
>>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>>> ResumeInRow 1
>>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>>> iResumeRetryCount -1, iRetries 0
>>>>
>>>> And if you look for the string '<somenumber> messages' in debug log,
>>>> if you close the client some time after stopping the server and
>>>> pushing some more messages to the client, you should get something
>>>> like
>>>>
>>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>>> queue holds 2 messages after shutdown of workers.
>>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>>> [v8.2102.0-4.fc35 try
>>>> https://www.rsyslog.com/e/2041 ]
>>>>
>>>> I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
>>>>
>>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>>> So, following your advice, I've confirmed the following
>>>>>
>>>>> 1. I switched to RELP. as per the following:
>>>>>
>>>>> add the following to server rsyslog.conf
>>>>>
>>>>> module(load="imrelp")
>>>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>>>
>>>>> add the following to server 50-default.conf:
>>>>>
>>>>> local7.* -/var/log/local7.log
>>>>>
>>>>> add the following to client 50-default.conf
>>>>>
>>>>> local7.* -/var/log/local7.log
>>>>> local7.* :omrelp:<server>:2514
>>>>>
>>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was only buffering one or two messages so the queue file would never be created.
>>>>>
>>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had to explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
>>>>>
>>>>> Here's exactly how I tested:
>>>>>
>>>>> 1. log a message from the client, verify that it shows up on the server
>>>>> # logger -p local7.info -s 'hello world'
>>>>>
>>>>> shows up in /var/log/local7.log on the server
>>>>> shows up in /var/log/local7.log on the client
>>>>>
>>>>> 2. disable rsyslog on the server
>>>>> # systemctl stop syslog.socket rsyslog.service
>>>>>
>>>>> 3. log a message on the client
>>>>> # logger -p local7.info -s 'hello world 2'
>>>>>
>>>>> shows up in /var/log/local7.log on the client
>>>>>
>>>>> 4. enable rsyslog on the server
>>>>> # systemctl start syslog.socket rsyslog.service
>>>>>
>>>>> 5. log a message on the client
>>>>> # logger -p local7.info -s 'hello world 3'
>>>>>
>>>>> shows up in /var/log/local7.log on the server
>>>>> shows up in /var/log/local7.log on the client
>>>>>
>>>>> "hello world 3" comes out on the server. "hello world 2" does not. Note that the server is only down for a few seconds in this scenario.
>>>>>
>>>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried running syslogd in debug mode, but frankly I don't understand the output very well and have no idea what I'm looking for. I don't see anything that would suggest the message is being queued on the client when the server is down as in step 3, but again, I'm not sure how that would show up in the debug trace.
>>>>>
>>>>> There must be something I'm doing wrong, but what?
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Mariusz Kruk via rsyslog
>>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>>> To: rsyslog@lists.adiscon.com
>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>> Messages with Rsyslog
>>>>>
>>>>> Firstly, after you confirm that your queueing works properly, I'd advise you to switch to RELP so you have "more reliability".
>>>>>
>>>>> But regarding your setup - as you defined
>>>>>
>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>
>>>>> Your queue should be placed there.
>>>>>
>>>>> Question is whether you do indeed have such directory in your system.
>>>>> Because if you don't, the rsyslog daemon won't be able to save the queue contents.
>>>>>
>>>>> But in case of just a few messages you shouldn't be saving the contents do disk at all. (it would be saved when you have unsent messages and shut down the rsyslog daemon).
>>>>>
>>>>> Also, notice that
>>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_rep
>>>>> e a t edmsgreduction.html "This parameter models old sysklogd
>>>>> legacy.
>>>>> *Note that many people, including the rsyslog authors, consider
>>>>> this to be a misfeature.* See /Discussion/ below to learn why."
>>>>>
>>>>> But in general, the setup should work... with one caveat. Your "never"
>>>>> might in fact not be "never". You didn't tweak the settings that control action resuming so they are at default 30 second initial interval which is getting raised after every 10 tries up to a default 1800 seconds. So if the server was off for long enough, the client might simply have paused sending for a really significant time.
>>>>>
>>>>> See the description of parameters at
>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>>
>>>>> You might set (just for test! you probably don't want to set it in
>>>>> prod for that often)
>>>>>
>>>>> $ActionResumeInterval 1
>>>>>
>>>>> And then run your client instance in debug mode to see interactively what it's trying to do.
>>>>>
>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog messages with rsyslog according to these instructions:
>>>>>>
>>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.h
>>>>>> t
>>>>>> m
>>>>>> l
>>>>>>
>>>>>> I confirm that remote logging is working initially by doing
>>>>>>
>>>>>> # logger "hello, world"
>>>>>>
>>>>>> on the client, and verifying that this message shows up in the
>>>>>> server (in this case in /var/log/syslog)
>>>>>>
>>>>>> I then shut down the rsyslog server, and log a few more messages on the client. As expected, these are not showing up on the server side any more. On the client, they seem to be going to its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>>>
>>>>>> I then re-enable the rsyslog server, but the entries that I wrote on the client never seem to make it back to the server. What am I doing wrong?
>>>>>>
>>>>>> Some configuration files:
>>>>>>
>>>>>> ------------------------------------------------------------------
>>>>>> -
>>>>>> -
>>>>>> -
>>>>>> -
>>>>>> ----------------------
>>>>>> client rsyslog.conf file:
>>>>>>
>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>> #
>>>>>> # For more information see
>>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>> #
>>>>>> # Default logging rules can be found in
>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>
>>>>>>
>>>>>> #################
>>>>>> #### MODULES ####
>>>>>> #################
>>>>>>
>>>>>> module(load="imuxsock") # provides support for local system
>>>>>> logging
>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>
>>>>>> # provides UDP syslog reception
>>>>>> #module(load="imudp")
>>>>>> #input(type="imudp" port="514")
>>>>>>
>>>>>> # provides TCP syslog reception
>>>>>> #module(load="imtcp")
>>>>>> #input(type="imtcp" port="514")
>>>>>>
>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>
>>>>>> ###########################
>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>> ###########################
>>>>>>
>>>>>> #
>>>>>> # Use traditional timestamp format.
>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>> #
>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>
>>>>>> # Filter duplicated messages
>>>>>> $RepeatedMsgReduction on
>>>>>>
>>>>>> #
>>>>>> # Set the default permissions for all log files.
>>>>>> #
>>>>>> $FileOwner syslog
>>>>>> $FileGroup adm
>>>>>> $FileCreateMode 0640
>>>>>> $DirCreateMode 0755
>>>>>> $Umask 0022
>>>>>> $PrivDropToUser syslog
>>>>>> $PrivDropToGroup syslog
>>>>>>
>>>>>> #
>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>> /var/spool/rsyslog
>>>>>>
>>>>>> #
>>>>>> # setup reliable local buffering
>>>>>> #
>>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>>>> rsyslog shuts down
>>>>>>
>>>>>> #
>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>> /etc/rsyslog.d/*.conf
>>>>>> *.* @@<redacted>:514
>>>>>>
>>>>>> ------------------------------------------------------------------
>>>>>> server rsyslog.conf file
>>>>>>
>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>> #
>>>>>> # For more information see
>>>>>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>> #
>>>>>> # Default logging rules can be found in
>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>
>>>>>>
>>>>>> #################
>>>>>> #### MODULES ####
>>>>>> #################
>>>>>>
>>>>>> module(load="imuxsock") # provides support for local system
>>>>>> logging
>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>
>>>>>> # provides UDP syslog reception
>>>>>> #module(load="imudp")
>>>>>> #input(type="imudp" port="514")
>>>>>>
>>>>>> # provides TCP syslog reception
>>>>>> module(load="imtcp")
>>>>>> input(type="imtcp" port="514")
>>>>>>
>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>
>>>>>> ###########################
>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>> ###########################
>>>>>>
>>>>>> #
>>>>>> # Use traditional timestamp format.
>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>> #
>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>
>>>>>> # Filter duplicated messages
>>>>>> $RepeatedMsgReduction on
>>>>>>
>>>>>> #
>>>>>> # Set the default permissions for all log files.
>>>>>> #
>>>>>> $FileOwner syslog
>>>>>> $FileGroup adm
>>>>>> $FileCreateMode 0640
>>>>>> $DirCreateMode 0755
>>>>>> $Umask 0022
>>>>>> $PrivDropToUser syslog
>>>>>> $PrivDropToGroup syslog
>>>>>>
>>>>>> #
>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>> /var/spool/rsyslog
>>>>>>
>>>>>> #
>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>> /etc/rsyslog.d/*.conf
>>>>>>
>>>>>> ------------------------------------------------------------------
>>>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>>>> FWIW)
>>>>>>
>>>>>> # rsyslogd -version (same version for both client and server)
>>>>>>
>>>>>> rsyslogd 8.32.0, compiled with:
>>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>>> PLATFORM (lsb_release -d):
>>>>>> FEATURE_REGEXP: Yes
>>>>>> GSSAPI Kerberos 5 support: Yes
>>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>>> 32bit Atomic operations supported: Yes
>>>>>> 64bit Atomic operations supported: Yes
>>>>>> memory allocator: system default
>>>>>> Runtime Instrumentation (slow code): No
>>>>>> uuid support: Yes
>>>>>> systemd support: Yes
>>>>>> Number of Bits in RainerScript integers: 64
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
it's worth noting that the logs do not come out in the order that they went in.
The logs in the memory queue will go out very quickly but the logs in the disk
queue will go out much more slowly, so if you have the logs with a number in
them, and just look at the end of the desination, the last number may be 1152,
but numbers 1153-2000 may still be in the file, just much earlier in the file.

David Lang


On Tue, 22 Feb 2022, David Lang wrote:

> Date: Tue, 22 Feb 2022 11:14:18 -0800 (PST)
> From: David Lang <david@lang.hm>
> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Cc: David Lang <david@lang.hm>,
> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> I'll have to look at the attachment later, but what does it show about the
> number of items processed by action 4? and can you get a similar stats dump
> from the system it's sending to?
>
> if you switch to the new format, one advantage is that the action() statement
> lets you give it a name, much easier to figure out what's what rather than
> just 'action 4'
>
> David Lang
>
> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>
>> Date: Tue, 22 Feb 2022 18:38:05 +0000
>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> To: David Lang <david@lang.hm>
>> Cc: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages
>> with
>> Rsyslog
>>
>> So in this case, I _think_ this shows the queue was holding 1152 messages,
>> the memory queue was holding 848, then after starting the server, the
>> memory queue appears to get emptied - if that's what this line means:
>>
>> Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
>> size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0 maxqsize=925
>>
>> But only the first 1152 messages ever come out on the server.
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, February 22, 2022 12:58 PM
>> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Cc: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages
>> with Rsyslog
>>
>> if you look there is an action 4 queue that also has 880 items in it,
>> that's the rest of them. That's the memory queue. those should also be
>> delivered once the link comes back up.
>>
>> what does pstats show after you bring the server back up?
>>
>> David Lang
>>
>> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>>
>>> Date: Tue, 22 Feb 2022 17:37:40 +0000
>>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> To: David Lang <david@lang.hm>,
>>> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages
>>> with
>>> Rsyslog
>>>
>>> I've attached the output of the impstat module for the following scenario:
>>>
>>> 1. impstat update rate is 30 seconds
>>>
>>> 2. restarted rsyslog on client, with server rsyslog is disabled
>>>
>>> 3. attempt to queue 2000 messages (just a simple 'C' program that
>>> calls syslog repeatedly)
>>>
>>> I can see where the DA queue only gets 1120 messages, in these two entries
>>> here:
>>>
>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]:
>>> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats:
>>> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=901
>>>
>>> ---------------impstat output----------------
>>>
>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0
>>> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21 AA3945
>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q:
>>> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 0: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 1: origin=core.action processed=16 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 2: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 3: origin=core.action processed=2000 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 4: origin=core.action processed=2000 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 5: origin=core.action processed=2025 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 6: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 7: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 8: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 9: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844
>>> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313
>>> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4
>>> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945
>>> rsyslogd-pstats: action 4 queue: origin=core.queue size=880
>>> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 Feb
>>> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue size=15
>>> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0
>>> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21 AA3945
>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: origin=core.queue
>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q:
>>> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=41
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, February 22, 2022 11:47 AM
>>> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> enable impstats and post the results so that we can see what's
>>> happening with the queues
>>>
>>> with a DA queue you have both a memory queue and a disk queue. did you
>>> restart the sending system while the server was down?
>>>
>>> David Lang
>>>
>>> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>>>
>>>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>>>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages
>>>> with
>>>> Rsyslog
>>>>
>>>> So there's still something I'm not understanding about DA queues.
>>>>
>>>> In my configuration, I have
>>>> $ActionQueueSize 1000
>>>> $WorkDirectory /var/spool/rsyslog
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>>>
>>>>
>>>> If I disable the server, queue < 1000 messages, then re-enable the
>>>> server, all messages are delivered.
>>>>
>>>> If I disable the server, queue 2000 messages, then re-enable the server,
>>>> only 1120 messages get delivered.
>>>>
>>>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets created,
>>>> but it seems as if it does not contain anything beyond message 1120.
>>>> It's like a lot of the messages didn't get flushed to the disk queue...
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Mariusz Kruk via rsyslog
>>>> Sent: Tuesday, February 22, 2022 8:44 AM
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> Not exactly, because with "creating a spearate ruleset" I meant a
>>>> completely different RainerScript-based configuration but this one should
>>>> also work as I wrote "somewhere around".
>>>>
>>>> Anyway, as David wrote somewhere in this thread - legacy config
>>>> format is OK for simple setups where it's more readable than Rainer
>>>> Script but if your config requires multiple directives modifying
>>>> functionality of the action, it's probably easier to write it as (in
>>>> your case)
>>>>
>>>> if ($syslogfacility == "local7") then
>>>>     action(type="omfwd" Target="wll" Port="2514"
>>>> action.resumeRetryCount="0" [... more action.parameters and
>>>> queue.parameters ...] )
>>>>
>>>> It's more obvious then what the parameters are for and you don't have
>>>> them scattered around (possibly intertwining with other parameters
>>>> modifying the resulting config).
>>>>
>>>> MK
>>>>
>>>> PS: I'm not sure if this condition will work this way; there was some bug
>>>> lately about textual representation but I don't recall if it was facility
>>>> or severity or both.
>>>>
>>>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>>>> What I found was that I had to do this in 50-default.conf:
>>>>>
>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>>>> $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
>>>>> down
>>>>>
>>>>> local7.* :omrelp:will:2514
>>>>>
>>>>> I believe that's what you meant here, yes?
>>>>>> I'd do a separate queue for this omfwd (or omrelp or whatever you're
>>>>>> gonna use in the end) action alone.
>>>>> When I did that, everything started to work properly. I can see the
>>>>> retries happening when rsyslogd is disabled on the server. Thanks for
>>>>> all your help.
>>>>>
>>>>> I wish I understood the configuration better. I have to admit, I find
>>>>> the documentation really confusing.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Mariusz Kruk via rsyslog
>>>>> Sent: Friday, February 18, 2022 3:22 PM
>>>>> To: rsyslog@lists.adiscon.com
>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>> Messages with Rsyslog
>>>>>
>>>>> If you run a client as
>>>>>
>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>>>
>>>>> You should see some text blob at the start but then, when the server is
>>>>> running, the client should not emit any more messages.
>>>>>
>>>>> But when you stop the server, the client should start emiting
>>>>> messages like
>>>>>
>>>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>>>> ResumeInRow 1
>>>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ]
>>>>> 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007
>>>>> 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>>>> iResumeRetryCount -1, iRetries 0
>>>>>
>>>>> And if you look for the string '<somenumber> messages' in debug log,
>>>>> if you close the client some time after stopping the server and
>>>>> pushing some more messages to the client, you should get something
>>>>> like
>>>>>
>>>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>>>> queue holds 2 messages after shutdown of workers.
>>>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>>>> [v8.2102.0-4.fc35 try
>>>>> https://www.rsyslog.com/e/2041 ]
>>>>>
>>>>> I'm not fully sure, however, since you use the legacy config format
>>>>> what's the interaction between both actions within the same queue. In
>>>>> order to be sure to have proper queueing _on the forwarding action_ I'd
>>>>> do a separate queue for this omfwd (or omrelp or whatever you're gonna
>>>>> use in the end) action alone.
>>>>>
>>>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>>>> So, following your advice, I've confirmed the following
>>>>>>
>>>>>> 1. I switched to RELP. as per the following:
>>>>>>
>>>>>> add the following to server rsyslog.conf
>>>>>>
>>>>>> module(load="imrelp")
>>>>>> input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
>>>>>>
>>>>>> add the following to server 50-default.conf:
>>>>>>
>>>>>> local7.* -/var/log/local7.log
>>>>>>
>>>>>> add the following to client 50-default.conf
>>>>>>
>>>>>> local7.* -/var/log/local7.log
>>>>>> local7.* :omrelp:<server>:2514
>>>>>>
>>>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was
>>>>>> only buffering one or two messages so the queue file would never be
>>>>>> created.
>>>>>>
>>>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had
>>>>>> to explicitly turn it off in rsyslog.conf so duplicates do not get
>>>>>> rolled up
>>>>>>
>>>>>> Here's exactly how I tested:
>>>>>>
>>>>>> 1. log a message from the client, verify that it shows up on the
>>>>>> server
>>>>>> # logger -p local7.info -s 'hello world'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the server
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> 2. disable rsyslog on the server
>>>>>> # systemctl stop syslog.socket rsyslog.service
>>>>>>
>>>>>> 3. log a message on the client
>>>>>> # logger -p local7.info -s 'hello world 2'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> 4. enable rsyslog on the server
>>>>>> # systemctl start syslog.socket rsyslog.service
>>>>>>
>>>>>> 5. log a message on the client
>>>>>> # logger -p local7.info -s 'hello world 3'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the server
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> "hello world 3" comes out on the server. "hello world 2" does not.
>>>>>> Note that the server is only down for a few seconds in this scenario.
>>>>>>
>>>>>> I tried setting $ActionResumeInterval 1 on the client, and I've tried
>>>>>> running syslogd in debug mode, but frankly I don't understand the
>>>>>> output very well and have no idea what I'm looking for. I don't see
>>>>>> anything that would suggest the message is being queued on the client
>>>>>> when the server is down as in step 3, but again, I'm not sure how that
>>>>>> would show up in the debug trace.
>>>>>>
>>>>>> There must be something I'm doing wrong, but what?
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>> Mariusz Kruk via rsyslog
>>>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>>>> To: rsyslog@lists.adiscon.com
>>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>>> Messages with Rsyslog
>>>>>>
>>>>>> Firstly, after you confirm that your queueing works properly, I'd
>>>>>> advise you to switch to RELP so you have "more reliability".
>>>>>>
>>>>>> But regarding your setup - as you defined
>>>>>>
>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>
>>>>>> Your queue should be placed there.
>>>>>>
>>>>>> Question is whether you do indeed have such directory in your system.
>>>>>> Because if you don't, the rsyslog daemon won't be able to save the
>>>>>> queue contents.
>>>>>>
>>>>>> But in case of just a few messages you shouldn't be saving the contents
>>>>>> do disk at all. (it would be saved when you have unsent messages and
>>>>>> shut down the rsyslog daemon).
>>>>>>
>>>>>> Also, notice that
>>>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_rep
>>>>>> e a t edmsgreduction.html "This parameter models old sysklogd
>>>>>> legacy.
>>>>>> *Note that many people, including the rsyslog authors, consider
>>>>>> this to be a misfeature.* See /Discussion/ below to learn why."
>>>>>>
>>>>>> But in general, the setup should work... with one caveat. Your "never"
>>>>>> might in fact not be "never". You didn't tweak the settings that
>>>>>> control action resuming so they are at default 30 second initial
>>>>>> interval which is getting raised after every 10 tries up to a default
>>>>>> 1800 seconds. So if the server was off for long enough, the client
>>>>>> might simply have paused sending for a really significant time.
>>>>>>
>>>>>> See the description of parameters at
>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>>>
>>>>>> You might set (just for test! you probably don't want to set it in
>>>>>> prod for that often)
>>>>>>
>>>>>> $ActionResumeInterval 1
>>>>>>
>>>>>> And then run your client instance in debug mode to see interactively
>>>>>> what it's trying to do.
>>>>>>
>>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding of
>>>>>>> syslog messages with rsyslog according to these instructions:
>>>>>>>
>>>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.h
>>>>>>> t
>>>>>>> m
>>>>>>> l
>>>>>>>
>>>>>>> I confirm that remote logging is working initially by doing
>>>>>>>
>>>>>>> # logger "hello, world"
>>>>>>>
>>>>>>> on the client, and verifying that this message shows up in the
>>>>>>> server (in this case in /var/log/syslog)
>>>>>>>
>>>>>>> I then shut down the rsyslog server, and log a few more messages on
>>>>>>> the client. As expected, these are not showing up on the server side
>>>>>>> any more. On the client, they seem to be going to its /var/log/syslog
>>>>>>> file; I have no idea where (if) they're being queued.
>>>>>>>
>>>>>>> I then re-enable the rsyslog server, but the entries that I wrote on
>>>>>>> the client never seem to make it back to the server. What am I doing
>>>>>>> wrong?
>>>>>>>
>>>>>>> Some configuration files:
>>>>>>>
>>>>>>> ------------------------------------------------------------------
>>>>>>> -
>>>>>>> -
>>>>>>> -
>>>>>>> -
>>>>>>> ----------------------
>>>>>>> client rsyslog.conf file:
>>>>>>>
>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>> #
>>>>>>> # For more information see
>>>>>>> #
>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>> #
>>>>>>> # Default logging rules can be found in
>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>
>>>>>>>
>>>>>>> #################
>>>>>>> #### MODULES ####
>>>>>>> #################
>>>>>>>
>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>> logging
>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>
>>>>>>> # provides UDP syslog reception
>>>>>>> #module(load="imudp")
>>>>>>> #input(type="imudp" port="514")
>>>>>>>
>>>>>>> # provides TCP syslog reception
>>>>>>> #module(load="imtcp")
>>>>>>> #input(type="imtcp" port="514")
>>>>>>>
>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>
>>>>>>> ###########################
>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>> ###########################
>>>>>>>
>>>>>>> #
>>>>>>> # Use traditional timestamp format.
>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>> #
>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>
>>>>>>> # Filter duplicated messages
>>>>>>> $RepeatedMsgReduction on
>>>>>>>
>>>>>>> #
>>>>>>> # Set the default permissions for all log files.
>>>>>>> #
>>>>>>> $FileOwner syslog
>>>>>>> $FileGroup adm
>>>>>>> $FileCreateMode 0640
>>>>>>> $DirCreateMode 0755
>>>>>>> $Umask 0022
>>>>>>> $PrivDropToUser syslog
>>>>>>> $PrivDropToGroup syslog
>>>>>>>
>>>>>>> #
>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>> /var/spool/rsyslog
>>>>>>>
>>>>>>> #
>>>>>>> # setup reliable local buffering
>>>>>>> #
>>>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>>>> mode $ActionResumeRetryCount -1 # infinite retries on insert
>>>>>>> failure $ActionQueueSaveOnShutdown on # save in-memory data if
>>>>>>> rsyslog shuts down
>>>>>>>
>>>>>>> #
>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>> *.* @@<redacted>:514
>>>>>>>
>>>>>>> ------------------------------------------------------------------
>>>>>>> server rsyslog.conf file
>>>>>>>
>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>> #
>>>>>>> # For more information see
>>>>>>> #
>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>> #
>>>>>>> # Default logging rules can be found in
>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>
>>>>>>>
>>>>>>> #################
>>>>>>> #### MODULES ####
>>>>>>> #################
>>>>>>>
>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>> logging
>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>
>>>>>>> # provides UDP syslog reception
>>>>>>> #module(load="imudp")
>>>>>>> #input(type="imudp" port="514")
>>>>>>>
>>>>>>> # provides TCP syslog reception
>>>>>>> module(load="imtcp")
>>>>>>> input(type="imtcp" port="514")
>>>>>>>
>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>
>>>>>>> ###########################
>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>> ###########################
>>>>>>>
>>>>>>> #
>>>>>>> # Use traditional timestamp format.
>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>> #
>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>
>>>>>>> # Filter duplicated messages
>>>>>>> $RepeatedMsgReduction on
>>>>>>>
>>>>>>> #
>>>>>>> # Set the default permissions for all log files.
>>>>>>> #
>>>>>>> $FileOwner syslog
>>>>>>> $FileGroup adm
>>>>>>> $FileCreateMode 0640
>>>>>>> $DirCreateMode 0755
>>>>>>> $Umask 0022
>>>>>>> $PrivDropToUser syslog
>>>>>>> $PrivDropToGroup syslog
>>>>>>>
>>>>>>> #
>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>> /var/spool/rsyslog
>>>>>>>
>>>>>>> #
>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>>
>>>>>>> ------------------------------------------------------------------
>>>>>>> version info for rsyslogd (both machines running Ubuntu 18.04,
>>>>>>> FWIW)
>>>>>>>
>>>>>>> # rsyslogd -version (same version for both client and server)
>>>>>>>
>>>>>>> rsyslogd 8.32.0, compiled with:
>>>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>>>> PLATFORM (lsb_release -d):
>>>>>>> FEATURE_REGEXP: Yes
>>>>>>> GSSAPI Kerberos 5 support: Yes
>>>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>>>> 32bit Atomic operations supported: Yes
>>>>>>> 64bit Atomic operations supported: Yes
>>>>>>> memory allocator: system default
>>>>>>> Runtime Instrumentation (slow code): No
>>>>>>> uuid support: Yes
>>>>>>> systemd support: Yes
>>>>>>> Number of Bits in RainerScript integers: 64
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>>>>> if you DON'T LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>>>>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>>>>> LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>>>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>>>>> THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>>>> THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>>> LIKE THAT.
>>>
>
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
Ah, I did not expect this. That is in fact what is happening. May I ask why this is the case? This is not a desirable behavior in my application - I'd have to attach sequence numbers to each message and reorder them later. Are there any options that would force in-order message delivery?

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Tuesday, February 22, 2022 2:16 PM
To: David Lang <david@lang.hm>
Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>; MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog

it's worth noting that the logs do not come out in the order that they went in.
The logs in the memory queue will go out very quickly but the logs in the disk queue will go out much more slowly, so if you have the logs with a number in them, and just look at the end of the desination, the last number may be 1152, but numbers 1153-2000 may still be in the file, just much earlier in the file.

David Lang


On Tue, 22 Feb 2022, David Lang wrote:

> Date: Tue, 22 Feb 2022 11:14:18 -0800 (PST)
> From: David Lang <david@lang.hm>
> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
> Cc: David Lang <david@lang.hm>,
> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
> Rsyslog
>
> I'll have to look at the attachment later, but what does it show about
> the number of items processed by action 4? and can you get a similar
> stats dump from the system it's sending to?
>
> if you switch to the new format, one advantage is that the action()
> statement lets you give it a name, much easier to figure out what's
> what rather than just 'action 4'
>
> David Lang
>
> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>
>> Date: Tue, 22 Feb 2022 18:38:05 +0000
>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> To: David Lang <david@lang.hm>
>> Cc: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>> Messages with
>> Rsyslog
>>
>> So in this case, I _think_ this shows the queue was holding 1152
>> messages, the memory queue was holding 848, then after starting the
>> server, the memory queue appears to get emptied - if that's what this line means:
>>
>> Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue:
>> origin=core.queue
>> size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0
>> maxqsize=925
>>
>> But only the first 1152 messages ever come out on the server.
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, February 22, 2022 12:58 PM
>> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Cc: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>> Messages with Rsyslog
>>
>> if you look there is an action 4 queue that also has 880 items in it,
>> that's the rest of them. That's the memory queue. those should also
>> be delivered once the link comes back up.
>>
>> what does pstats show after you bring the server back up?
>>
>> David Lang
>>
>> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>>
>>> Date: Tue, 22 Feb 2022 17:37:40 +0000
>>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> To: David Lang <david@lang.hm>,
>>> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with
>>> Rsyslog
>>>
>>> I've attached the output of the impstat module for the following scenario:
>>>
>>> 1. impstat update rate is 30 seconds
>>>
>>> 2. restarted rsyslog on client, with server rsyslog is disabled
>>>
>>> 3. attempt to queue 2000 messages (just a simple 'C' program that
>>> calls syslog repeatedly)
>>>
>>> I can see where the DA queue only gets 1120 messages, in these two
>>> entries
>>> here:
>>>
>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]:
>>> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats:
>>> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=901
>>>
>>> ---------------impstat output----------------
>>>
>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0
>>> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21
>>> AA3945
>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
>>> Feb
>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue:
>>> origin=core.queue
>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q:
>>> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0
>>> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 0: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 1: origin=core.action processed=16 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 2: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 3: origin=core.action processed=2000 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 4: origin=core.action processed=2000 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 5: origin=core.action processed=2025 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 6: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 7: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 8: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> action 9: origin=core.action processed=0 failed=0 suspended=0
>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844
>>> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313
>>> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4
>>> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0
>>> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945
>>> rsyslogd-pstats: action 4 queue: origin=core.queue size=880
>>> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
>>> Feb
>>> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue
>>> size=15
>>> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>>> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>> 22
>>> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0
>>> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21
>>> AA3945
>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
>>> Feb
>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue:
>>> origin=core.queue
>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q:
>>> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0
>>> discarded.nf=0 maxqsize=41
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, February 22, 2022 11:47 AM
>>> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> enable impstats and post the results so that we can see what's
>>> happening with the queues
>>>
>>> with a DA queue you have both a memory queue and a disk queue. did
>>> you restart the sending system while the server was down?
>>>
>>> David Lang
>>>
>>> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>>>
>>>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>>>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with
>>>> Rsyslog
>>>>
>>>> So there's still something I'm not understanding about DA queues.
>>>>
>>>> In my configuration, I have
>>>> $ActionQueueSize 1000
>>>> $WorkDirectory /var/spool/rsyslog
>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>> mode
>>>>
>>>>
>>>> If I disable the server, queue < 1000 messages, then re-enable the
>>>> server, all messages are delivered.
>>>>
>>>> If I disable the server, queue 2000 messages, then re-enable the
>>>> server, only 1120 messages get delivered.
>>>>
>>>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets
>>>> created, but it seems as if it does not contain anything beyond message 1120.
>>>> It's like a lot of the messages didn't get flushed to the disk queue...
>>>>
>>>> -----Original Message-----
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>> Mariusz Kruk via rsyslog
>>>> Sent: Tuesday, February 22, 2022 8:44 AM
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> Not exactly, because with "creating a spearate ruleset" I meant a
>>>> completely different RainerScript-based configuration but this one
>>>> should also work as I wrote "somewhere around".
>>>>
>>>> Anyway, as David wrote somewhere in this thread - legacy config
>>>> format is OK for simple setups where it's more readable than Rainer
>>>> Script but if your config requires multiple directives modifying
>>>> functionality of the action, it's probably easier to write it as
>>>> (in your case)
>>>>
>>>> if ($syslogfacility == "local7") then
>>>>     action(type="omfwd" Target="wll" Port="2514"
>>>> action.resumeRetryCount="0" [... more action.parameters and
>>>> queue.parameters ...] )
>>>>
>>>> It's more obvious then what the parameters are for and you don't
>>>> have them scattered around (possibly intertwining with other
>>>> parameters modifying the resulting config).
>>>>
>>>> MK
>>>>
>>>> PS: I'm not sure if this condition will work this way; there was
>>>> some bug lately about textual representation but I don't recall if
>>>> it was facility or severity or both.
>>>>
>>>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>>>> What I found was that I had to do this in 50-default.conf:
>>>>>
>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>> mode $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog
>>>>> shuts down
>>>>>
>>>>> local7.* :omrelp:will:2514
>>>>>
>>>>> I believe that's what you meant here, yes?
>>>>>> I'd do a separate queue for this omfwd (or omrelp or whatever
>>>>>> you're gonna use in the end) action alone.
>>>>> When I did that, everything started to work properly. I can see
>>>>> the retries happening when rsyslogd is disabled on the server.
>>>>> Thanks for all your help.
>>>>>
>>>>> I wish I understood the configuration better. I have to admit, I
>>>>> find the documentation really confusing.
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Mariusz Kruk via rsyslog
>>>>> Sent: Friday, February 18, 2022 3:22 PM
>>>>> To: rsyslog@lists.adiscon.com
>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>> Messages with Rsyslog
>>>>>
>>>>> If you run a client as
>>>>>
>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>>>
>>>>> You should see some text blob at the start but then, when the
>>>>> server is running, the client should not emit any more messages.
>>>>>
>>>>> But when you stop the server, the client should start emiting
>>>>> messages like
>>>>>
>>>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>>>> ResumeInRow 1
>>>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ]
>>>>> 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned
>>>>> -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>>>> iResumeRetryCount -1, iRetries 0
>>>>>
>>>>> And if you look for the string '<somenumber> messages' in debug
>>>>> log, if you close the client some time after stopping the server
>>>>> and pushing some more messages to the client, you should get
>>>>> something like
>>>>>
>>>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>>>> queue holds 2 messages after shutdown of workers.
>>>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>>>> [v8.2102.0-4.fc35 try
>>>>> https://www.rsyslog.com/e/2041 ]
>>>>>
>>>>> I'm not fully sure, however, since you use the legacy config
>>>>> format what's the interaction between both actions within the same
>>>>> queue. In order to be sure to have proper queueing _on the
>>>>> forwarding action_ I'd do a separate queue for this omfwd (or
>>>>> omrelp or whatever you're gonna use in the end) action alone.
>>>>>
>>>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>>>> So, following your advice, I've confirmed the following
>>>>>>
>>>>>> 1. I switched to RELP. as per the following:
>>>>>>
>>>>>> add the following to server rsyslog.conf
>>>>>>
>>>>>> module(load="imrelp")
>>>>>> input(type="imrelp" port="2514" maxDataSize="10k"
>>>>>> keepAlive="on")
>>>>>>
>>>>>> add the following to server 50-default.conf:
>>>>>>
>>>>>> local7.* -/var/log/local7.log
>>>>>>
>>>>>> add the following to client 50-default.conf
>>>>>>
>>>>>> local7.* -/var/log/local7.log
>>>>>> local7.* :omrelp:<server>:2514
>>>>>>
>>>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was
>>>>>> only buffering one or two messages so the queue file would never
>>>>>> be created.
>>>>>>
>>>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had
>>>>>> to explicitly turn it off in rsyslog.conf so duplicates do not
>>>>>> get rolled up
>>>>>>
>>>>>> Here's exactly how I tested:
>>>>>>
>>>>>> 1. log a message from the client, verify that it shows up on the
>>>>>> server
>>>>>> # logger -p local7.info -s 'hello world'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the server
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> 2. disable rsyslog on the server
>>>>>> # systemctl stop syslog.socket rsyslog.service
>>>>>>
>>>>>> 3. log a message on the client
>>>>>> # logger -p local7.info -s 'hello world 2'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> 4. enable rsyslog on the server
>>>>>> # systemctl start syslog.socket rsyslog.service
>>>>>>
>>>>>> 5. log a message on the client
>>>>>> # logger -p local7.info -s 'hello world 3'
>>>>>>
>>>>>> shows up in /var/log/local7.log on the server
>>>>>> shows up in /var/log/local7.log on the client
>>>>>>
>>>>>> "hello world 3" comes out on the server. "hello world 2" does not.
>>>>>> Note that the server is only down for a few seconds in this scenario.
>>>>>>
>>>>>> I tried setting $ActionResumeInterval 1 on the client, and I've
>>>>>> tried running syslogd in debug mode, but frankly I don't
>>>>>> understand the output very well and have no idea what I'm looking
>>>>>> for. I don't see anything that would suggest the message is
>>>>>> being queued on the client when the server is down as in step 3,
>>>>>> but again, I'm not sure how that would show up in the debug trace.
>>>>>>
>>>>>> There must be something I'm doing wrong, but what?
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>> Mariusz Kruk via rsyslog
>>>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>>>> To: rsyslog@lists.adiscon.com
>>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>>> Messages with Rsyslog
>>>>>>
>>>>>> Firstly, after you confirm that your queueing works properly, I'd
>>>>>> advise you to switch to RELP so you have "more reliability".
>>>>>>
>>>>>> But regarding your setup - as you defined
>>>>>>
>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>
>>>>>> Your queue should be placed there.
>>>>>>
>>>>>> Question is whether you do indeed have such directory in your system.
>>>>>> Because if you don't, the rsyslog daemon won't be able to save
>>>>>> the queue contents.
>>>>>>
>>>>>> But in case of just a few messages you shouldn't be saving the
>>>>>> contents do disk at all. (it would be saved when you have unsent
>>>>>> messages and shut down the rsyslog daemon).
>>>>>>
>>>>>> Also, notice that
>>>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_r
>>>>>> ep e a t edmsgreduction.html "This parameter models old sysklogd
>>>>>> legacy.
>>>>>> *Note that many people, including the rsyslog authors, consider
>>>>>> this to be a misfeature.* See /Discussion/ below to learn why."
>>>>>>
>>>>>> But in general, the setup should work... with one caveat. Your "never"
>>>>>> might in fact not be "never". You didn't tweak the settings that
>>>>>> control action resuming so they are at default 30 second initial
>>>>>> interval which is getting raised after every 10 tries up to a
>>>>>> default
>>>>>> 1800 seconds. So if the server was off for long enough, the
>>>>>> client might simply have paused sending for a really significant time.
>>>>>>
>>>>>> See the description of parameters at
>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>>>
>>>>>> You might set (just for test! you probably don't want to set it
>>>>>> in prod for that often)
>>>>>>
>>>>>> $ActionResumeInterval 1
>>>>>>
>>>>>> And then run your client instance in debug mode to see
>>>>>> interactively what it's trying to do.
>>>>>>
>>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding
>>>>>>> of syslog messages with rsyslog according to these instructions:
>>>>>>>
>>>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding
>>>>>>> .h
>>>>>>> t
>>>>>>> m
>>>>>>> l
>>>>>>>
>>>>>>> I confirm that remote logging is working initially by doing
>>>>>>>
>>>>>>> # logger "hello, world"
>>>>>>>
>>>>>>> on the client, and verifying that this message shows up in the
>>>>>>> server (in this case in /var/log/syslog)
>>>>>>>
>>>>>>> I then shut down the rsyslog server, and log a few more messages
>>>>>>> on the client. As expected, these are not showing up on the
>>>>>>> server side any more. On the client, they seem to be going to
>>>>>>> its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>>>>
>>>>>>> I then re-enable the rsyslog server, but the entries that I
>>>>>>> wrote on the client never seem to make it back to the server.
>>>>>>> What am I doing wrong?
>>>>>>>
>>>>>>> Some configuration files:
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> -
>>>>>>> -
>>>>>>> -
>>>>>>> -
>>>>>>> ----------------------
>>>>>>> client rsyslog.conf file:
>>>>>>>
>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>> #
>>>>>>> # For more information see
>>>>>>> #
>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>> #
>>>>>>> # Default logging rules can be found in
>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>
>>>>>>>
>>>>>>> #################
>>>>>>> #### MODULES ####
>>>>>>> #################
>>>>>>>
>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>> logging
>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>
>>>>>>> # provides UDP syslog reception
>>>>>>> #module(load="imudp")
>>>>>>> #input(type="imudp" port="514")
>>>>>>>
>>>>>>> # provides TCP syslog reception
>>>>>>> #module(load="imtcp")
>>>>>>> #input(type="imtcp" port="514")
>>>>>>>
>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>
>>>>>>> ###########################
>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>> ###########################
>>>>>>>
>>>>>>> #
>>>>>>> # Use traditional timestamp format.
>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>> #
>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>
>>>>>>> # Filter duplicated messages
>>>>>>> $RepeatedMsgReduction on
>>>>>>>
>>>>>>> #
>>>>>>> # Set the default permissions for all log files.
>>>>>>> #
>>>>>>> $FileOwner syslog
>>>>>>> $FileGroup adm
>>>>>>> $FileCreateMode 0640
>>>>>>> $DirCreateMode 0755
>>>>>>> $Umask 0022
>>>>>>> $PrivDropToUser syslog
>>>>>>> $PrivDropToGroup syslog
>>>>>>>
>>>>>>> #
>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>> /var/spool/rsyslog
>>>>>>>
>>>>>>> #
>>>>>>> # setup reliable local buffering # $ActionQueueType LinkedList #
>>>>>>> use asynchronous processing $ActionQueueFileName srvrfwd # set
>>>>>>> file name, also enables disk mode $ActionResumeRetryCount -1 #
>>>>>>> infinite retries on insert failure $ActionQueueSaveOnShutdown on
>>>>>>> # save in-memory data if rsyslog shuts down
>>>>>>>
>>>>>>> #
>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>> *.* @@<redacted>:514
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> server rsyslog.conf file
>>>>>>>
>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>> #
>>>>>>> # For more information see
>>>>>>> #
>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>> #
>>>>>>> # Default logging rules can be found in
>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>
>>>>>>>
>>>>>>> #################
>>>>>>> #### MODULES ####
>>>>>>> #################
>>>>>>>
>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>> logging
>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>
>>>>>>> # provides UDP syslog reception
>>>>>>> #module(load="imudp")
>>>>>>> #input(type="imudp" port="514")
>>>>>>>
>>>>>>> # provides TCP syslog reception
>>>>>>> module(load="imtcp")
>>>>>>> input(type="imtcp" port="514")
>>>>>>>
>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>
>>>>>>> ###########################
>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>> ###########################
>>>>>>>
>>>>>>> #
>>>>>>> # Use traditional timestamp format.
>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>> #
>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>
>>>>>>> # Filter duplicated messages
>>>>>>> $RepeatedMsgReduction on
>>>>>>>
>>>>>>> #
>>>>>>> # Set the default permissions for all log files.
>>>>>>> #
>>>>>>> $FileOwner syslog
>>>>>>> $FileGroup adm
>>>>>>> $FileCreateMode 0640
>>>>>>> $DirCreateMode 0755
>>>>>>> $Umask 0022
>>>>>>> $PrivDropToUser syslog
>>>>>>> $PrivDropToGroup syslog
>>>>>>>
>>>>>>> #
>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>> /var/spool/rsyslog
>>>>>>>
>>>>>>> #
>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> -- version info for rsyslogd (both machines running Ubuntu
>>>>>>> 18.04,
>>>>>>> FWIW)
>>>>>>>
>>>>>>> # rsyslogd -version (same version for both client and server)
>>>>>>>
>>>>>>> rsyslogd 8.32.0, compiled with:
>>>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>>>> PLATFORM (lsb_release -d):
>>>>>>> FEATURE_REGEXP: Yes
>>>>>>> GSSAPI Kerberos 5 support: Yes
>>>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>>>> 32bit Atomic operations supported: Yes
>>>>>>> 64bit Atomic operations supported: Yes
>>>>>>> memory allocator: system default
>>>>>>> Runtime Instrumentation (slow code): No
>>>>>>> uuid support: Yes
>>>>>>> systemd support: Yes
>>>>>>> Number of Bits in RainerScript integers: 64
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>>> you DON'T LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST if you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>> you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>> you DON'T LIKE THAT.
>>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: setting up reliable forwarding of syslog Messages with Rsyslog [ In reply to ]
to get strict in-order operation you have to cripple rsyslog performance, not
have any redundancy, and not have any relays. Even something as simple as UDP
forwarding is not guarnateed to not have packets take different paths and
therefor pass each other on the wire.

Way back in v3, rsyslog was making attempts to keep everything in-order, and I
pointed out all the things outside of rsyslog's control that could cause
problems, so we relaxed the ordering requirement and it allowed for
simplifcations that drastically improved performance.

In this case, the fact that the two queues are handled independently is why the
fast in-memory queue is able to empty much faster than the on-disk queue. (if
you think about the problem, if you have a queue that has spilled to disk and
have more logs arriving, should you try to deliver them? or should you take the
slow path to write them to disk, read other logs from disk and deliver those
instead? forcing new logs to take the slow path can result in so much overhead
that instead of catching up, you may fall further behind)

requiring strict in-order processing would also prevent you from having multiple
threads work on different log messages, at least without having so much overhead
in syncronizing the work that the multiple threads would probably slow
performance.

In general, the best thing that you can do is to use high resolution timestamps
(RFC5424) instead of the legacy 1-second-resolution timestamps. you can't trust
different systems (or even different programs on the same system) to be
accurarate to that level, but you can trust that a given piece of software will
see time move in one direction (well, most of the time, unless you have daylight
savings change, or correct the system clock...)

When analysing logs, you need to be careful about what assumptions you make
about the order of logs and instead of looking for log1 followed by log2, look
for log1 and log2 to happen within a short time window.

David Lang



On Tue, 22 Feb 2022, MACGREGOR Will wrote:

> Ah, I did not expect this. That is in fact what is happening. May I ask why this is the case? This is not a desirable behavior in my application - I'd have to attach sequence numbers to each message and reorder them later. Are there any options that would force in-order message delivery?
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Tuesday, February 22, 2022 2:16 PM
> To: David Lang <david@lang.hm>
> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>; MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog
>
> it's worth noting that the logs do not come out in the order that they went in.
> The logs in the memory queue will go out very quickly but the logs in the disk queue will go out much more slowly, so if you have the logs with a number in them, and just look at the end of the desination, the last number may be 1152, but numbers 1153-2000 may still be in the file, just much earlier in the file.
>
> David Lang
>
>
> On Tue, 22 Feb 2022, David Lang wrote:
>
>> Date: Tue, 22 Feb 2022 11:14:18 -0800 (PST)
>> From: David Lang <david@lang.hm>
>> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
>> Cc: David Lang <david@lang.hm>,
>> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with
>> Rsyslog
>>
>> I'll have to look at the attachment later, but what does it show about
>> the number of items processed by action 4? and can you get a similar
>> stats dump from the system it's sending to?
>>
>> if you switch to the new format, one advantage is that the action()
>> statement lets you give it a name, much easier to figure out what's
>> what rather than just 'action 4'
>>
>> David Lang
>>
>> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>>
>>> Date: Tue, 22 Feb 2022 18:38:05 +0000
>>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with
>>> Rsyslog
>>>
>>> So in this case, I _think_ this shows the queue was holding 1152
>>> messages, the memory queue was holding 848, then after starting the
>>> server, the memory queue appears to get emptied - if that's what this line means:
>>>
>>> Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue:
>>> origin=core.queue
>>> size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0
>>> maxqsize=925
>>>
>>> But only the first 1152 messages ever come out on the server.
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, February 22, 2022 12:58 PM
>>> To: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>> Cc: David Lang <david@lang.hm>; MACGREGOR Will via rsyslog
>>> <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>>> Messages with Rsyslog
>>>
>>> if you look there is an action 4 queue that also has 880 items in it,
>>> that's the rest of them. That's the memory queue. those should also
>>> be delivered once the link comes back up.
>>>
>>> what does pstats show after you bring the server back up?
>>>
>>> David Lang
>>>
>>> On Tue, 22 Feb 2022, MACGREGOR Will wrote:
>>>
>>>> Date: Tue, 22 Feb 2022 17:37:40 +0000
>>>> From: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>>> To: David Lang <david@lang.hm>,
>>>> MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with
>>>> Rsyslog
>>>>
>>>> I've attached the output of the impstat module for the following scenario:
>>>>
>>>> 1. impstat update rate is 30 seconds
>>>>
>>>> 2. restarted rsyslog on client, with server rsyslog is disabled
>>>>
>>>> 3. attempt to queue 2000 messages (just a simple 'C' program that
>>>> calls syslog repeatedly)
>>>>
>>>> I can see where the DA queue only gets 1120 messages, in these two
>>>> entries
>>>> here:
>>>>
>>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]:
>>>> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0
>>>> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats:
>>>> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0
>>>> discarded.full=0 discarded.nf=0 maxqsize=901
>>>>
>>>> ---------------impstat output----------------
>>>>
>>>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>>> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>>> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>>> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0
>>>> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21
>>>> AA3945
>>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
>>>> Feb
>>>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue:
>>>> origin=core.queue
>>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>>> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q:
>>>> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0
>>>> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0
>>>> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 0: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 1: origin=core.action processed=16 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 2: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 3: origin=core.action processed=2000 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 4: origin=core.action processed=2000 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 5: origin=core.action processed=2025 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 6: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 7: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 8: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> action 9: origin=core.action processed=0 failed=0 suspended=0
>>>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats:
>>>> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844
>>>> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313
>>>> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4
>>>> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0
>>>> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945
>>>> rsyslogd-pstats: action 4 queue: origin=core.queue size=880
>>>> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901
>>>> Feb
>>>> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue
>>>> size=15
>>>> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock
>>>> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action
>>>> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action
>>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action
>>>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action
>>>> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0
>>>> Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action
>>>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb
>>>> 22
>>>> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats
>>>> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0
>>>> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21
>>>> AA3945
>>>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117
>>>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117
>>>> Feb
>>>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue:
>>>> origin=core.queue
>>>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0
>>>> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q:
>>>> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0
>>>> discarded.nf=0 maxqsize=41
>>>>
>>>> -----Original Message-----
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Tuesday, February 22, 2022 11:47 AM
>>>> To: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>> Messages with Rsyslog
>>>>
>>>> enable impstats and post the results so that we can see what's
>>>> happening with the queues
>>>>
>>>> with a DA queue you have both a memory queue and a disk queue. did
>>>> you restart the sending system while the server was down?
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote:
>>>>
>>>>> Date: Tue, 22 Feb 2022 16:44:58 +0000
>>>>> From: MACGREGOR Will via rsyslog <rsyslog@lists.adiscon.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Cc: MACGREGOR Will <will.macgregor@thalesgroup.com>
>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>> Messages with
>>>>> Rsyslog
>>>>>
>>>>> So there's still something I'm not understanding about DA queues.
>>>>>
>>>>> In my configuration, I have
>>>>> $ActionQueueSize 1000
>>>>> $WorkDirectory /var/spool/rsyslog
>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>> mode
>>>>>
>>>>>
>>>>> If I disable the server, queue < 1000 messages, then re-enable the
>>>>> server, all messages are delivered.
>>>>>
>>>>> If I disable the server, queue 2000 messages, then re-enable the
>>>>> server, only 1120 messages get delivered.
>>>>>
>>>>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets
>>>>> created, but it seems as if it does not contain anything beyond message 1120.
>>>>> It's like a lot of the messages didn't get flushed to the disk queue...
>>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>> Mariusz Kruk via rsyslog
>>>>> Sent: Tuesday, February 22, 2022 8:44 AM
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>> Messages with Rsyslog
>>>>>
>>>>> Not exactly, because with "creating a spearate ruleset" I meant a
>>>>> completely different RainerScript-based configuration but this one
>>>>> should also work as I wrote "somewhere around".
>>>>>
>>>>> Anyway, as David wrote somewhere in this thread - legacy config
>>>>> format is OK for simple setups where it's more readable than Rainer
>>>>> Script but if your config requires multiple directives modifying
>>>>> functionality of the action, it's probably easier to write it as
>>>>> (in your case)
>>>>>
>>>>> if ($syslogfacility == "local7") then
>>>>>     action(type="omfwd" Target="wll" Port="2514"
>>>>> action.resumeRetryCount="0" [... more action.parameters and
>>>>> queue.parameters ...] )
>>>>>
>>>>> It's more obvious then what the parameters are for and you don't
>>>>> have them scattered around (possibly intertwining with other
>>>>> parameters modifying the resulting config).
>>>>>
>>>>> MK
>>>>>
>>>>> PS: I'm not sure if this condition will work this way; there was
>>>>> some bug lately about textual representation but I don't recall if
>>>>> it was facility or severity or both.
>>>>>
>>>>> On 22.02.2022 14:31, MACGREGOR Will wrote:
>>>>>> What I found was that I had to do this in 50-default.conf:
>>>>>>
>>>>>> $ActionQueueType LinkedList # use asynchronous processing
>>>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk
>>>>>> mode $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
>>>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog
>>>>>> shuts down
>>>>>>
>>>>>> local7.* :omrelp:will:2514
>>>>>>
>>>>>> I believe that's what you meant here, yes?
>>>>>>> I'd do a separate queue for this omfwd (or omrelp or whatever
>>>>>>> you're gonna use in the end) action alone.
>>>>>> When I did that, everything started to work properly. I can see
>>>>>> the retries happening when rsyslogd is disabled on the server.
>>>>>> Thanks for all your help.
>>>>>>
>>>>>> I wish I understood the configuration better. I have to admit, I
>>>>>> find the documentation really confusing.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>> Mariusz Kruk via rsyslog
>>>>>> Sent: Friday, February 18, 2022 3:22 PM
>>>>>> To: rsyslog@lists.adiscon.com
>>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>>> Messages with Rsyslog
>>>>>>
>>>>>> If you run a client as
>>>>>>
>>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
>>>>>>
>>>>>> You should see some text blob at the start but then, when the
>>>>>> server is running, the client should not emit any more messages.
>>>>>>
>>>>>> But when you stop the server, the client should start emiting
>>>>>> messages like
>>>>>>
>>>>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
>>>>>> ResumeInRow 1
>>>>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
>>>>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ]
>>>>>> 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned
>>>>>> -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
>>>>>> actionDoRetry: action-0-builtin:omfwd check for max retries,
>>>>>> iResumeRetryCount -1, iRetries 0
>>>>>>
>>>>>> And if you look for the string '<somenumber> messages' in debug
>>>>>> log, if you close the client some time after stopping the server
>>>>>> and pushing some more messages to the client, you should get
>>>>>> something like
>>>>>>
>>>>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
>>>>>> queue holds 2 messages after shutdown of workers.
>>>>>> queue.saveonshutdown is set, so data will now be spooled to disk
>>>>>> [v8.2102.0-4.fc35 try
>>>>>> https://www.rsyslog.com/e/2041 ]
>>>>>>
>>>>>> I'm not fully sure, however, since you use the legacy config
>>>>>> format what's the interaction between both actions within the same
>>>>>> queue. In order to be sure to have proper queueing _on the
>>>>>> forwarding action_ I'd do a separate queue for this omfwd (or
>>>>>> omrelp or whatever you're gonna use in the end) action alone.
>>>>>>
>>>>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
>>>>>>> So, following your advice, I've confirmed the following
>>>>>>>
>>>>>>> 1. I switched to RELP. as per the following:
>>>>>>>
>>>>>>> add the following to server rsyslog.conf
>>>>>>>
>>>>>>> module(load="imrelp")
>>>>>>> input(type="imrelp" port="2514" maxDataSize="10k"
>>>>>>> keepAlive="on")
>>>>>>>
>>>>>>> add the following to server 50-default.conf:
>>>>>>>
>>>>>>> local7.* -/var/log/local7.log
>>>>>>>
>>>>>>> add the following to client 50-default.conf
>>>>>>>
>>>>>>> local7.* -/var/log/local7.log
>>>>>>> local7.* :omrelp:<server>:2514
>>>>>>>
>>>>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was
>>>>>>> only buffering one or two messages so the queue file would never
>>>>>>> be created.
>>>>>>>
>>>>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had
>>>>>>> to explicitly turn it off in rsyslog.conf so duplicates do not
>>>>>>> get rolled up
>>>>>>>
>>>>>>> Here's exactly how I tested:
>>>>>>>
>>>>>>> 1. log a message from the client, verify that it shows up on the
>>>>>>> server
>>>>>>> # logger -p local7.info -s 'hello world'
>>>>>>>
>>>>>>> shows up in /var/log/local7.log on the server
>>>>>>> shows up in /var/log/local7.log on the client
>>>>>>>
>>>>>>> 2. disable rsyslog on the server
>>>>>>> # systemctl stop syslog.socket rsyslog.service
>>>>>>>
>>>>>>> 3. log a message on the client
>>>>>>> # logger -p local7.info -s 'hello world 2'
>>>>>>>
>>>>>>> shows up in /var/log/local7.log on the client
>>>>>>>
>>>>>>> 4. enable rsyslog on the server
>>>>>>> # systemctl start syslog.socket rsyslog.service
>>>>>>>
>>>>>>> 5. log a message on the client
>>>>>>> # logger -p local7.info -s 'hello world 3'
>>>>>>>
>>>>>>> shows up in /var/log/local7.log on the server
>>>>>>> shows up in /var/log/local7.log on the client
>>>>>>>
>>>>>>> "hello world 3" comes out on the server. "hello world 2" does not.
>>>>>>> Note that the server is only down for a few seconds in this scenario.
>>>>>>>
>>>>>>> I tried setting $ActionResumeInterval 1 on the client, and I've
>>>>>>> tried running syslogd in debug mode, but frankly I don't
>>>>>>> understand the output very well and have no idea what I'm looking
>>>>>>> for. I don't see anything that would suggest the message is
>>>>>>> being queued on the client when the server is down as in step 3,
>>>>>>> but again, I'm not sure how that would show up in the debug trace.
>>>>>>>
>>>>>>> There must be something I'm doing wrong, but what?
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>>>>>> Mariusz Kruk via rsyslog
>>>>>>> Sent: Friday, February 18, 2022 4:18 AM
>>>>>>> To: rsyslog@lists.adiscon.com
>>>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog
>>>>>>> Messages with Rsyslog
>>>>>>>
>>>>>>> Firstly, after you confirm that your queueing works properly, I'd
>>>>>>> advise you to switch to RELP so you have "more reliability".
>>>>>>>
>>>>>>> But regarding your setup - as you defined
>>>>>>>
>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>>
>>>>>>> Your queue should be placed there.
>>>>>>>
>>>>>>> Question is whether you do indeed have such directory in your system.
>>>>>>> Because if you don't, the rsyslog daemon won't be able to save
>>>>>>> the queue contents.
>>>>>>>
>>>>>>> But in case of just a few messages you shouldn't be saving the
>>>>>>> contents do disk at all. (it would be saved when you have unsent
>>>>>>> messages and shut down the rsyslog daemon).
>>>>>>>
>>>>>>> Also, notice that
>>>>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_r
>>>>>>> ep e a t edmsgreduction.html "This parameter models old sysklogd
>>>>>>> legacy.
>>>>>>> *Note that many people, including the rsyslog authors, consider
>>>>>>> this to be a misfeature.* See /Discussion/ below to learn why."
>>>>>>>
>>>>>>> But in general, the setup should work... with one caveat. Your "never"
>>>>>>> might in fact not be "never". You didn't tweak the settings that
>>>>>>> control action resuming so they are at default 30 second initial
>>>>>>> interval which is getting raised after every 10 tries up to a
>>>>>>> default
>>>>>>> 1800 seconds. So if the server was off for long enough, the
>>>>>>> client might simply have paused sending for a really significant time.
>>>>>>>
>>>>>>> See the description of parameters at
>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
>>>>>>>
>>>>>>> You might set (just for test! you probably don't want to set it
>>>>>>> in prod for that often)
>>>>>>>
>>>>>>> $ActionResumeInterval 1
>>>>>>>
>>>>>>> And then run your client instance in debug mode to see
>>>>>>> interactively what it's trying to do.
>>>>>>>
>>>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
>>>>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding
>>>>>>>> of syslog messages with rsyslog according to these instructions:
>>>>>>>>
>>>>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding
>>>>>>>> .h
>>>>>>>> t
>>>>>>>> m
>>>>>>>> l
>>>>>>>>
>>>>>>>> I confirm that remote logging is working initially by doing
>>>>>>>>
>>>>>>>> # logger "hello, world"
>>>>>>>>
>>>>>>>> on the client, and verifying that this message shows up in the
>>>>>>>> server (in this case in /var/log/syslog)
>>>>>>>>
>>>>>>>> I then shut down the rsyslog server, and log a few more messages
>>>>>>>> on the client. As expected, these are not showing up on the
>>>>>>>> server side any more. On the client, they seem to be going to
>>>>>>>> its /var/log/syslog file; I have no idea where (if) they're being queued.
>>>>>>>>
>>>>>>>> I then re-enable the rsyslog server, but the entries that I
>>>>>>>> wrote on the client never seem to make it back to the server.
>>>>>>>> What am I doing wrong?
>>>>>>>>
>>>>>>>> Some configuration files:
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------
>>>>>>>> --
>>>>>>>> -
>>>>>>>> -
>>>>>>>> -
>>>>>>>> -
>>>>>>>> ----------------------
>>>>>>>> client rsyslog.conf file:
>>>>>>>>
>>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>>> #
>>>>>>>> # For more information see
>>>>>>>> #
>>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>>> #
>>>>>>>> # Default logging rules can be found in
>>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>>
>>>>>>>>
>>>>>>>> #################
>>>>>>>> #### MODULES ####
>>>>>>>> #################
>>>>>>>>
>>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>>> logging
>>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>>
>>>>>>>> # provides UDP syslog reception
>>>>>>>> #module(load="imudp")
>>>>>>>> #input(type="imudp" port="514")
>>>>>>>>
>>>>>>>> # provides TCP syslog reception
>>>>>>>> #module(load="imtcp")
>>>>>>>> #input(type="imtcp" port="514")
>>>>>>>>
>>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>>
>>>>>>>> ###########################
>>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>>> ###########################
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Use traditional timestamp format.
>>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>>> #
>>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>>
>>>>>>>> # Filter duplicated messages
>>>>>>>> $RepeatedMsgReduction on
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Set the default permissions for all log files.
>>>>>>>> #
>>>>>>>> $FileOwner syslog
>>>>>>>> $FileGroup adm
>>>>>>>> $FileCreateMode 0640
>>>>>>>> $DirCreateMode 0755
>>>>>>>> $Umask 0022
>>>>>>>> $PrivDropToUser syslog
>>>>>>>> $PrivDropToGroup syslog
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>>> /var/spool/rsyslog
>>>>>>>>
>>>>>>>> #
>>>>>>>> # setup reliable local buffering # $ActionQueueType LinkedList #
>>>>>>>> use asynchronous processing $ActionQueueFileName srvrfwd # set
>>>>>>>> file name, also enables disk mode $ActionResumeRetryCount -1 #
>>>>>>>> infinite retries on insert failure $ActionQueueSaveOnShutdown on
>>>>>>>> # save in-memory data if rsyslog shuts down
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>>> *.* @@<redacted>:514
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------
>>>>>>>> --
>>>>>>>> server rsyslog.conf file
>>>>>>>>
>>>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog.
>>>>>>>> #
>>>>>>>> # For more information see
>>>>>>>> #
>>>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>>>>>>>> #
>>>>>>>> # Default logging rules can be found in
>>>>>>>> /etc/rsyslog.d/50-default.conf
>>>>>>>>
>>>>>>>>
>>>>>>>> #################
>>>>>>>> #### MODULES ####
>>>>>>>> #################
>>>>>>>>
>>>>>>>> module(load="imuxsock") # provides support for local system
>>>>>>>> logging
>>>>>>>> #module(load="immark") # provides --MARK-- message capability
>>>>>>>>
>>>>>>>> # provides UDP syslog reception
>>>>>>>> #module(load="imudp")
>>>>>>>> #input(type="imudp" port="514")
>>>>>>>>
>>>>>>>> # provides TCP syslog reception
>>>>>>>> module(load="imtcp")
>>>>>>>> input(type="imtcp" port="514")
>>>>>>>>
>>>>>>>> # provides kernel logging support and enable non-kernel klog
>>>>>>>> messages module(load="imklog" permitnonkernelfacility="on")
>>>>>>>>
>>>>>>>> ###########################
>>>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>>>> ###########################
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Use traditional timestamp format.
>>>>>>>> # To enable high precision timestamps, comment out the following line.
>>>>>>>> #
>>>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>>>
>>>>>>>> # Filter duplicated messages
>>>>>>>> $RepeatedMsgReduction on
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Set the default permissions for all log files.
>>>>>>>> #
>>>>>>>> $FileOwner syslog
>>>>>>>> $FileGroup adm
>>>>>>>> $FileCreateMode 0640
>>>>>>>> $DirCreateMode 0755
>>>>>>>> $Umask 0022
>>>>>>>> $PrivDropToUser syslog
>>>>>>>> $PrivDropToGroup syslog
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Where to place spool and state files # $WorkDirectory
>>>>>>>> /var/spool/rsyslog
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig
>>>>>>>> /etc/rsyslog.d/*.conf
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------
>>>>>>>> -- version info for rsyslogd (both machines running Ubuntu
>>>>>>>> 18.04,
>>>>>>>> FWIW)
>>>>>>>>
>>>>>>>> # rsyslogd -version (same version for both client and server)
>>>>>>>>
>>>>>>>> rsyslogd 8.32.0, compiled with:
>>>>>>>> PLATFORM: x86_64-pc-linux-gnu
>>>>>>>> PLATFORM (lsb_release -d):
>>>>>>>> FEATURE_REGEXP: Yes
>>>>>>>> GSSAPI Kerberos 5 support: Yes
>>>>>>>> FEATURE_DEBUG (debug build, slow code): No
>>>>>>>> 32bit Atomic operations supported: Yes
>>>>>>>> 64bit Atomic operations supported: Yes
>>>>>>>> memory allocator: system default
>>>>>>>> Runtime Instrumentation (slow code): No
>>>>>>>> uuid support: Yes
>>>>>>>> systemd support: Yes
>>>>>>>> Number of Bits in RainerScript integers: 64
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>>>> you DON'T LIKE THAT.
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>>> POST if you DON'T LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>>> you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>> you DON'T LIKE THAT.
>>>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.