Mailing List Archive

sending via tcp means that when the receiver can't handle the messages, the
sender starts queueing them for later delivery and when the queue is full, will
suspend sending as well

it would probably make things much clearer if you enabled impstats on both
systems.

beyond that, I think we would need to see both configs, impstats from both, and
examples of messages that you think are wrong to try and understand this.

David Lang


On Wed, 19 Feb 2020, dgermanrsysl--- via rsyslog wrote:

> Date: Wed, 19 Feb 2020 13:09:54 -0500
> From: dgermanrsysl--- via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: dgermanrsysl@real-world-systems.com
> Subject: [rsyslog] Suspended entry occurs at remote syslog
>
> We frequently (BUT not always) see (identical) "action suspended"
> messages both on the originating host and on  the "forwarded to" host.
>
> 2 questions:
>
> 1) Why are the suspensions happening at all?
>
> 2) If there is a problem forwarding the message, how does it end up at
> the "forwarded to" host? (with the same timestamp as the original message)
>
> The environment is:
> Both systems are on the same LAN
>
> pi93graf (192.168.1.51) wifi connected raspberry Pi Zero W Raspbian
> GNU/Linux 9 (stretch)
> rsyslogd 8.24.0 at a fixed location less than 15 meters from the verizon
> router.
>
> dalogger ethernet connected Raspberry Pi Model B Rev 2; Raspbian
> GNU/Linux 10 (buster)
> rsyslogd  8.1901.0 (aka 2019.01)
> dalogger.dns04.com( a dynamic DNS resolved name ending up at
> 108.35.223.94 port forwarded to ethernet connected 192.168.1.14)
>
> on pi93graf local7.log AND 05_notice.log AND 04_warn.log
>
> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
>
> on dalogger local7.log.1: AND 05_notice.log.1 AND 04_warn.log.1
>
> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
>
> on pi93graf rsyslogd -N 1 -d 2 includes:
>
> 5241.693867466:main thread    : PRIFILT '*.warn'
> 5241.700666318:main thread    :   ACTION 11
> [builtin:omfwd:action(type="builtin:omfwd" ...)]
>
> /etc/rsyslog.conf includes:
>
> # all unexpected messages (warn,err,crit... ) go to dalogger
> *.warn                        @dalogger.dns04.com
> *.warn action(type="omfwd" target="dalogger.dns04.com" protocol="tcp"
> action.resumeRetryCount="20" queue.type="linkedList" queue.size="1000")
>
> If complete rsyslog.conf would be helpful I will make them available but
> I didn't want to clutter up this email with them.
>
> Thanks for you help and for a GREAT package.
>
> Dennis
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Thank you.

I have added impstats as per example 'Load module, send stats data to
local file' from
https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
(A crazy idea would be to send them to a server you have access to!)
They are at:
https://real-world-systems.com/rsyslog/dalogger.impstats.022020
https://real-world-systems.com/rsyslog/pi93graf.impsatts.022020

pi93graf s includes:

Thu Feb 20 13:21:36 2020: action 10: origin=core.action processed=15
failed=4 suspended=1 suspended.duration=30 resumed=0
...
Thu Feb 20 13:21:36 2020: action 23: origin=core.action processed=14
failed=4 suspended=1 suspended.duration=30 resumed=0

pi93graf s 05_notice.log includes

2020-02-20 13:18:46 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available (multiple times)
2020-02-20 13:19:41 pi93graf local7.err liblogging-stdlog: omfwd: error
101 sending via udp: Network is unreachable [v8.24.0 try
http://www.rsyslog.com/e/2354 ]
2020-02-20 13:19:41 pi93graf local7.warning liblogging-stdlog: action
'action 23' suspended, next retry is Thu Feb 20 13:20:11 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]
2020-02-20 13:19:41 pi93graf local7.err liblogging-stdlog: omfwd: error
101 sending via udp: Network is unreachable [v8.24.0 try
http://www.rsyslog.com/e/2354 ]
2020-02-20 13:19:41 pi93graf local7.warning liblogging-stdlog: action
'action 10' suspended, next retry is Thu Feb 20 13:20:11 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]

2020-02-20 13:24:10 pi93graf local7.warning liblogging-stdlog: action
'action 11' suspended, next retry is Thu Feb 20 13:24:40 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]
2020-02-20 13:34:13 pi93graf local7.warning liblogging-stdlog: action
'action 11' suspended, next retry is Thu Feb 20 13:34:43 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]

            The "Network is unreachable" message is unusual.

dalogger s 05_notice.log includes

2020-02-20 13:18:46 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available (((Notice no messages timestamped 13:19 )))
2020-02-20 13:20:37 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available
2020-02-20 13:21:16 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available
2020-02-20 13:21:48 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available
2020-02-20 13:22:20 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
Routers available
2020-02-20 13:24:10 pi93graf local7.warning liblogging-stdlog: action
'action 11' suspended, next retry is Thu Feb 20 13:24:40 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]
2020-02-20 13:34:13 pi93graf local7.warning liblogging-stdlog: action
'action 11' suspended, next retry is Thu Feb 20 13:34:43 2020 [v8.24.0
try http://www.rsyslog.com/e/2007 ]

++++
I have placed a copy of rsyslog.conf s at
https://real-world-systems.com/rsyslog/dalogger.rsyslog.022020.conf and
https://real-world-systems.com/rsyslog/pi93graf.rsyslog.022020.conf
https://real-world-systems.com/rsyslog/pi93graf.sysylog.d.debug.022020.conf
https://real-world-systems.com/rsyslog/pi93graf.sysylog.d.info.022020.conf

(PS Any unrelated comments regarding my misunderstanding of the
configuration are welcome!)
++++
Although in the pi93graf config there exists a line
*.warn action(type="omfwd" target="dalogger.dns04.com" protocol="tcp" ...
I don't believe it is honored in version 8.24

The line
*.warn                        @dalogger.dns04.com
uses UDP if I am not mistaken?

++++++++++++++
As Rainer noted in a recent post, 8.24 running on pi93graf is old, but
it seems no more recent version is available for raspberrypi stretch via
aptitude. I attempted to copy the version from dalogger running buster
but that needed a library that was unavailable and I wasn't sure if
going down that road was a good idea. Should I build a newer version
from source?

Thanks again for your assistance.

Dennis

On 2/19/20 1:29 PM, David Lang wrote:
> sending via tcp means that when the receiver can't handle the
> messages, the sender starts queueing them for later delivery and when
> the queue is full, will suspend sending as well
>
> it would probably make things much clearer if you enabled impstats on
> both systems.
>
> beyond that, I think we would need to see both configs, impstats from
> both, and examples of messages that you think are wrong to try and
> understand this.
>
> David Lang
>
>
>  On Wed, 19 Feb 2020, dgermanrsysl--- via rsyslog wrote:
>
>> Date: Wed, 19 Feb 2020 13:09:54 -0500
>> From: dgermanrsysl--- via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: dgermanrsysl@real-world-systems.com
>> Subject: [rsyslog] Suspended entry occurs at remote syslog
>>
>> We frequently (BUT not always) see (identical) "action suspended"
>> messages both on the originating host and on  the "forwarded to" host.
>>
>> 2 questions:
>>
>> 1) Why are the suspensions happening at all?
>>
>> 2) If there is a problem forwarding the message, how does it end up
>> at the "forwarded to" host? (with the same timestamp as the original
>> message)
>>
>> The environment is:
>> Both systems are on the same LAN
>>
>> pi93graf (192.168.1.51) wifi connected raspberry Pi Zero W Raspbian
>> GNU/Linux 9 (stretch)
>> rsyslogd 8.24.0 at a fixed location less than 15 meters from the
>> verizon router.
>>
>> dalogger ethernet connected Raspberry Pi Model B Rev 2; Raspbian
>> GNU/Linux 10 (buster)
>> rsyslogd  8.1901.0 (aka 2019.01)
>> dalogger.dns04.com( a dynamic DNS resolved name ending up at
>> 108.35.223.94 port forwarded to ethernet connected 192.168.1.14)
>>
>> on pi93graf local7.log AND 05_notice.log AND 04_warn.log
>>
>> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
>> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020
>> [v8.24.0 try http://www.rsyslog.com/e/2007 ]
>>
>> on dalogger local7.log.1: AND 05_notice.log.1 AND 04_warn.log.1
>>
>> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
>> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020
>> [v8.24.0 try http://www.rsyslog.com/e/2007 ]
>>
>> on pi93graf rsyslogd -N 1 -d 2 includes:
>>
>> 5241.693867466:main thread    : PRIFILT '*.warn'
>> 5241.700666318:main thread    :   ACTION 11
>> [builtin:omfwd:action(type="builtin:omfwd" ...)]
>>
>> /etc/rsyslog.conf includes:
>>
>> # all unexpected messages (warn,err,crit... ) go to dalogger
>> *.warn                        @dalogger.dns04.com
>> *.warn action(type="omfwd" target="dalogger.dns04.com" protocol="tcp"
>> action.resumeRetryCount="20" queue.type="linkedList" queue.size="1000")
>>
>> If complete rsyslog.conf would be helpful I will make them available
>> but I didn't want to clutter up this email with them.
>>
>> Thanks for you help and for a GREAT package.
>>
>> Dennis
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

try the current packages at
https://software.opensuse.org//download.html?project=home%3Argerhards&package=rsyslog

under debian there is a raspian option.

David Lang

On Thu, 20 Feb 2020, dgermanrsysl--- via rsyslog wrote:

> Date: Thu, 20 Feb 2020 15:09:28 -0500
> From: dgermanrsysl--- via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: dgermanrsysl@real-world-systems.com
> Subject: [rsyslog] Suspended entry occurs at remote syslog
>
> Thank you.
>
> I have added impstats as per example 'Load module, send stats data to
> local file' from
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
> (A crazy idea would be to send them to a server you have access to!)
> They are at:
> https://real-world-systems.com/rsyslog/dalogger.impstats.022020
> https://real-world-systems.com/rsyslog/pi93graf.impsatts.022020
>
> pi93graf s includes:
>
> Thu Feb 20 13:21:36 2020: action 10: origin=core.action processed=15
> failed=4 suspended=1 suspended.duration=30 resumed=0
> ...
> Thu Feb 20 13:21:36 2020: action 23: origin=core.action processed=14
> failed=4 suspended=1 suspended.duration=30 resumed=0
>
> pi93graf s 05_notice.log includes
>
> 2020-02-20 13:18:46 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available (multiple times)
> 2020-02-20 13:19:41 pi93graf local7.err liblogging-stdlog: omfwd: error
> 101 sending via udp: Network is unreachable [v8.24.0 try
> http://www.rsyslog.com/e/2354 ]
> 2020-02-20 13:19:41 pi93graf local7.warning liblogging-stdlog: action
> 'action 23' suspended, next retry is Thu Feb 20 13:20:11 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
> 2020-02-20 13:19:41 pi93graf local7.err liblogging-stdlog: omfwd: error
> 101 sending via udp: Network is unreachable [v8.24.0 try
> http://www.rsyslog.com/e/2354 ]
> 2020-02-20 13:19:41 pi93graf local7.warning liblogging-stdlog: action
> 'action 10' suspended, next retry is Thu Feb 20 13:20:11 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
>
> 2020-02-20 13:24:10 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Thu Feb 20 13:24:40 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
> 2020-02-20 13:34:13 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Thu Feb 20 13:34:43 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
>
>             The "Network is unreachable" message is unusual.
>
> dalogger s 05_notice.log includes
>
> 2020-02-20 13:18:46 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available (((Notice no messages timestamped 13:19 )))
> 2020-02-20 13:20:37 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available
> 2020-02-20 13:21:16 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available
> 2020-02-20 13:21:48 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available
> 2020-02-20 13:22:20 pi93graf daemon.warning dhcpcd[291]: wlan0: no IPv6
> Routers available
> 2020-02-20 13:24:10 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Thu Feb 20 13:24:40 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
> 2020-02-20 13:34:13 pi93graf local7.warning liblogging-stdlog: action
> 'action 11' suspended, next retry is Thu Feb 20 13:34:43 2020 [v8.24.0
> try http://www.rsyslog.com/e/2007 ]
>
> ++++
> I have placed a copy of rsyslog.conf s at
> https://real-world-systems.com/rsyslog/dalogger.rsyslog.022020.conf and
> https://real-world-systems.com/rsyslog/pi93graf.rsyslog.022020.conf
> https://real-world-systems.com/rsyslog/pi93graf.sysylog.d.debug.022020.conf
> https://real-world-systems.com/rsyslog/pi93graf.sysylog.d.info.022020.conf
>
> (PS Any unrelated comments regarding my misunderstanding of the
> configuration are welcome!)
> ++++
> Although in the pi93graf config there exists a line
> *.warn action(type="omfwd" target="dalogger.dns04.com" protocol="tcp" ...
> I don't believe it is honored in version 8.24
>
> The line
> *.warn                        @dalogger.dns04.com
> uses UDP if I am not mistaken?
>
> ++++++++++++++
> As Rainer noted in a recent post, 8.24 running on pi93graf is old, but
> it seems no more recent version is available for raspberrypi stretch via
> aptitude. I attempted to copy the version from dalogger running buster
> but that needed a library that was unavailable and I wasn't sure if
> going down that road was a good idea. Should I build a newer version
> from source?
>
> Thanks again for your assistance.
>
> Dennis
>
> On 2/19/20 1:29 PM, David Lang wrote:
>> sending via tcp means that when the receiver can't handle the
>> messages, the sender starts queueing them for later delivery and when
>> the queue is full, will suspend sending as well
>>
>> it would probably make things much clearer if you enabled impstats on
>> both systems.
>>
>> beyond that, I think we would need to see both configs, impstats from
>> both, and examples of messages that you think are wrong to try and
>> understand this.
>>
>> David Lang
>>
>>
>>  On Wed, 19 Feb 2020, dgermanrsysl--- via rsyslog wrote:
>>
>>> Date: Wed, 19 Feb 2020 13:09:54 -0500
>>> From: dgermanrsysl--- via rsyslog <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: dgermanrsysl@real-world-systems.com
>>> Subject: [rsyslog] Suspended entry occurs at remote syslog
>>>
>>> We frequently (BUT not always) see (identical) "action suspended"
>>> messages both on the originating host and on  the "forwarded to" host.
>>>
>>> 2 questions:
>>>
>>> 1) Why are the suspensions happening at all?
>>>
>>> 2) If there is a problem forwarding the message, how does it end up
>>> at the "forwarded to" host? (with the same timestamp as the original
>>> message)
>>>
>>> The environment is:
>>> Both systems are on the same LAN
>>>
>>> pi93graf (192.168.1.51) wifi connected raspberry Pi Zero W Raspbian
>>> GNU/Linux 9 (stretch)
>>> rsyslogd 8.24.0 at a fixed location less than 15 meters from the
>>> verizon router.
>>>
>>> dalogger ethernet connected Raspberry Pi Model B Rev 2; Raspbian
>>> GNU/Linux 10 (buster)
>>> rsyslogd  8.1901.0 (aka 2019.01)
>>> dalogger.dns04.com( a dynamic DNS resolved name ending up at
>>> 108.35.223.94 port forwarded to ethernet connected 192.168.1.14)
>>>
>>> on pi93graf local7.log AND 05_notice.log AND 04_warn.log
>>>
>>> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
>>> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020
>>> [v8.24.0 try http://www.rsyslog.com/e/2007 ]
>>>
>>> on dalogger local7.log.1: AND 05_notice.log.1 AND 04_warn.log.1
>>>
>>> 2020-02-18 19:57:20 pi93graf local7.warning liblogging-stdlog: action
>>> 'action 11' suspended, next retry is Tue Feb 18 19:58:50 2020
>>> [v8.24.0 try http://www.rsyslog.com/e/2007 ]
>>>
>>> on pi93graf rsyslogd -N 1 -d 2 includes:
>>>
>>> 5241.693867466:main thread    : PRIFILT '*.warn'
>>> 5241.700666318:main thread    :   ACTION 11
>>> [builtin:omfwd:action(type="builtin:omfwd" ...)]
>>>
>>> /etc/rsyslog.conf includes:
>>>
>>> # all unexpected messages (warn,err,crit... ) go to dalogger
>>> *.warn                        @dalogger.dns04.com
>>> *.warn action(type="omfwd" target="dalogger.dns04.com" protocol="tcp"
>>> action.resumeRetryCount="20" queue.type="linkedList" queue.size="1000")
>>>
>>> If complete rsyslog.conf would be helpful I will make them available
>>> but I didn't want to clutter up this email with them.
>>>
>>> Thanks for you help and for a GREAT package.
>>>
>>> Dennis
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.