Mailing List Archive: Rsyslog stop working when output destination (rabbitmq) is unavialable

Rsyslog stop working when output destination (rabbitmq) is unavialable

Feb 5, 2024, 5:53 AM

Post #1 of 6 (118 views)

I have rsyslog (8.2310.0) in docker container from latest alpine image as a
syslog collector with forwarding to logstash.local and rabbitmq.local

Rsyslog stops working when rabbitmq is unavailable and work fine when
rabbitmq server is up

How to configure rsyslog to work when the destination is unavailable ?

module(load="omrabbitmq")
module(load="imptcp" threads="3")
input(type="imptcp" port="514" ruleset="syslogCollector")

template(
name="json_syslog"
type="list"
option.json="on"
) {
constant(value="{")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"type\":\"syslog_json")
constant(value="\",\"tag\":\"")
property(name="syslogtag" format="json")
constant(value="\",\"relayhost\":\"") property(name="fromhost")
constant(value="\",\"relayip\":\"") property(name="fromhost-ip")
constant(value="\",\"logsource\":\"") property(name="source")
constant(value="\",\"hostname\":\"") property(name="hostname"
caseconversion="lower")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"source\":\"") property(name="app-name"
caseConversion="lower" onEmpty="null")
constant(value="\",\"priority\":\"") property(name="pri")
constant(value="\",\"severity\":\"")
property(name="syslogseverity" caseConversion="upper")
constant(value="\",\"facility\":\"") property(name="syslogfacility")
constant(value="\",\"severity_label\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility_label\":\"")
property(name="syslogfacility-text")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"end_msg\":\"")
constant(value="\"}\n")
}

ruleset(
name="syslogCollector"
) {
@@elk.local:5000;json_syslog
call send2mqtt
}

ruleset (
name="send2mqtt"

queue.type="LinkedList"
queue.size="1024"
queue.dequeueBatchSize="512"
queue.filename="q_mqtt"
queue.saveonshutdown="off"
queue.highwatermark="450"
queue.lowwatermark="50"

) {
action(
type="omrabbitmq"
host="rabbitmq.local"
verify_peer="off"
verify_hostname="off"
virtual_host="/"
user="xxx"
password="xxx"
exchange="syslog"
routing_key="messages"
body_template="json_syslog"

action.resumeRetryCount="0"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
action.resumeInterval="10"
)
}
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog stop working when output destination (rabbitmq) is unavialable [ In reply to ]

rsyslog at lists

Feb 5, 2024, 11:11 AM

Post #2 of 6 (117 views)

Permalink

you have a queue of 1024 for rabbitmq, if there are more messages than that
pending, other processing will stop until the queue can accept more messages.
Setup a larger queue (potentially a disk assisted queue) to handle longer
outages.

you may also want to consider configuring the queue to throw away messages if it
gets too full.

David Lang

On Mon, 5 Feb 2024, Alex via rsyslog wrote:

> I have rsyslog (8.2310.0) in docker container from latest alpine image as a
> syslog collector with forwarding to logstash.local and rabbitmq.local
>
> Rsyslog stops working when rabbitmq is unavailable and work fine when
> rabbitmq server is up
>
> How to configure rsyslog to work when the destination is unavailable ?
>
>
> module(load="omrabbitmq")
> module(load="imptcp" threads="3")
> input(type="imptcp" port="514" ruleset="syslogCollector")
>
> template(
> name="json_syslog"
> type="list"
> option.json="on"
> ) {
> constant(value="{")
> constant(value="\"@timestamp\":\"")
> property(name="timereported" dateFormat="rfc3339")
> constant(value="\",\"type\":\"syslog_json")
> constant(value="\",\"tag\":\"")
> property(name="syslogtag" format="json")
> constant(value="\",\"relayhost\":\"") property(name="fromhost")
> constant(value="\",\"relayip\":\"") property(name="fromhost-ip")
> constant(value="\",\"logsource\":\"") property(name="source")
> constant(value="\",\"hostname\":\"") property(name="hostname"
> caseconversion="lower")
> constant(value="\",\"program\":\"") property(name="programname")
> constant(value="\",\"source\":\"") property(name="app-name"
> caseConversion="lower" onEmpty="null")
> constant(value="\",\"priority\":\"") property(name="pri")
> constant(value="\",\"severity\":\"")
> property(name="syslogseverity" caseConversion="upper")
> constant(value="\",\"facility\":\"") property(name="syslogfacility")
> constant(value="\",\"severity_label\":\"")
> property(name="syslogseverity-text")
> constant(value="\",\"facility_label\":\"")
> property(name="syslogfacility-text")
> constant(value="\",\"message\":\"") property(name="msg" format="json")
> constant(value="\",\"end_msg\":\"")
> constant(value="\"}\n")
> }
>
> ruleset(
> name="syslogCollector"
> ) {
> @@elk.local:5000;json_syslog
> call send2mqtt
> }
>
> ruleset (
> name="send2mqtt"
>
> queue.type="LinkedList"
> queue.size="1024"
> queue.dequeueBatchSize="512"
> queue.filename="q_mqtt"
> queue.saveonshutdown="off"
> queue.highwatermark="450"
> queue.lowwatermark="50"
>
> ) {
> action(
> type="omrabbitmq"
> host="rabbitmq.local"
> verify_peer="off"
> verify_hostname="off"
> virtual_host="/"
> user="xxx"
> password="xxx"
> exchange="syslog"
> routing_key="messages"
> body_template="json_syslog"
>
> action.resumeRetryCount="0"
> action.reportSuspension="on"
> action.reportSuspensionContinuation="on"
> action.resumeInterval="10"
> )
> }
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog stop working when output destination (rabbitmq) is unavialable [ In reply to ]

rsyslog at lists

Feb 6, 2024, 10:03 PM

Post #3 of 6 (115 views)

Permalink

Alex, excuse me for using this thread for posting my question. But my
question is also on the same topic.

I have configured rsyslogd (v8.2102.0) to forward syslogs to two remote
servers, one using TCP protocol and second one using UDP. Besides
forwarding, I also have a rule to log to a local file. Here are the
relevant sections from my rsyslog.conf. I can post the full file if needed.

$MainMsgQueueTimeoutEnqueue 0

local4.* action(type="omfwd" target="10.240.219.64"
action.resumeRetryCount="10" protocol="tcp" port="36456"
template="Syslogs_ForwardFormat")
local4.* action(type="omfwd" target="10.16.39.124"
protocol="udp" port="36456" template="Syslogs_ForwardFormat")

*.* /var/log/syslogs

After the TCP connection is established and messages are getting forwarded,
say someone pulls out the ethernet cable at the remote syslog collector 1
that is using tcp protocol. Some number of syslogs from this point are
still forwarded to the server 2 using udp protocol and written to the local
file as well. However if the TCP connection to the first server does not
recover then after some time, we do not see any syslogs in the local file
or at the second syslog server. I straced rsyslogd and all I see is the
recvmsg() calls on the /dev/log socket.

How do we tell rsyslog to drop the TCP connection if the other end is no
longer receiving the syslogs. Perhaps it can drop the connection and retry
after say X minutes. I tried using resumeRetryCount as documented at
https://www.rsyslog.com/doc/configuration/actions.html, but that is not
helping in my situation. Perhaps I'm missing some other "action" setting?
Appreciate if you can give me a pointer to a sample configuration or point
me to relevant documentation.

Thank you.

On Mon, Feb 5, 2024 at 11:11?AM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> you have a queue of 1024 for rabbitmq, if there are more messages than
> that
> pending, other processing will stop until the queue can accept more
> messages.
> Setup a larger queue (potentially a disk assisted queue) to handle longer
> outages.
>
> you may also want to consider configuring the queue to throw away messages
> if it
> gets too full.
>
> David Lang
>
> On Mon, 5 Feb 2024, Alex via rsyslog wrote:
>
> > I have rsyslog (8.2310.0) in docker container from latest alpine image
> as a
> > syslog collector with forwarding to logstash.local and rabbitmq.local
> >
> > Rsyslog stops working when rabbitmq is unavailable and work fine when
> > rabbitmq server is up
> >
> > How to configure rsyslog to work when the destination is unavailable ?
> >
> >
> > module(load="omrabbitmq")
> > module(load="imptcp" threads="3")
> > input(type="imptcp" port="514" ruleset="syslogCollector")
> >
> > template(
> > name="json_syslog"
> > type="list"
> > option.json="on"
> > ) {
> > constant(value="{")
> > constant(value="\"@timestamp\":\"")
> > property(name="timereported" dateFormat="rfc3339")
> > constant(value="\",\"type\":\"syslog_json")
> > constant(value="\",\"tag\":\"")
> > property(name="syslogtag" format="json")
> > constant(value="\",\"relayhost\":\"") property(name="fromhost")
> > constant(value="\",\"relayip\":\"") property(name="fromhost-ip")
> > constant(value="\",\"logsource\":\"") property(name="source")
> > constant(value="\",\"hostname\":\"") property(name="hostname"
> > caseconversion="lower")
> > constant(value="\",\"program\":\"") property(name="programname")
> > constant(value="\",\"source\":\"") property(name="app-name"
> > caseConversion="lower" onEmpty="null")
> > constant(value="\",\"priority\":\"") property(name="pri")
> > constant(value="\",\"severity\":\"")
> > property(name="syslogseverity" caseConversion="upper")
> > constant(value="\",\"facility\":\"")
> property(name="syslogfacility")
> > constant(value="\",\"severity_label\":\"")
> > property(name="syslogseverity-text")
> > constant(value="\",\"facility_label\":\"")
> > property(name="syslogfacility-text")
> > constant(value="\",\"message\":\"") property(name="msg"
> format="json")
> > constant(value="\",\"end_msg\":\"")
> > constant(value="\"}\n")
> > }
> >
> > ruleset(
> > name="syslogCollector"
> > ) {
> > @@elk.local:5000;json_syslog
> > call send2mqtt
> > }
> >
> > ruleset (
> > name="send2mqtt"
> >
> > queue.type="LinkedList"
> > queue.size="1024"
> > queue.dequeueBatchSize="512"
> > queue.filename="q_mqtt"
> > queue.saveonshutdown="off"
> > queue.highwatermark="450"
> > queue.lowwatermark="50"
> >
> > ) {
> > action(
> > type="omrabbitmq"
> > host="rabbitmq.local"
> > verify_peer="off"
> > verify_hostname="off"
> > virtual_host="/"
> > user="xxx"
> > password="xxx"
> > exchange="syslog"
> > routing_key="messages"
> > body_template="json_syslog"
> >
> > action.resumeRetryCount="0"
> > action.reportSuspension="on"
> > action.reportSuspensionContinuation="on"
> > action.resumeInterval="10"
> > )
> > }
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog stop working when output destination (rabbitmq) is unavialable [ In reply to ]

rsyslog at lists

Feb 6, 2024, 10:15 PM

Post #4 of 6 (115 views)

Permalink

read the documents I posted above and they will explain most of what you are
seeing.

the reason it doesn't fail instantly is that the OS has a buffer for the TCP
connection, and that buffer takes a little time to fill up.

retrycount tells rsyslog how many times to retry each message before dropping
it.

closing and re-opening the connection won't help you if the network is down, as
everything will block waiting for the connection to get re-established.

If you enable impstats and have it write directly to a file (not send it though
the normal queue) then you will see the failures and see the action being
suspended after some failures.

David Lang

On Tue, 6 Feb 2024, Prasad Koya wrote:

> Alex, excuse me for using this thread for posting my question. But my
> question is also on the same topic.
>
> I have configured rsyslogd (v8.2102.0) to forward syslogs to two remote
> servers, one using TCP protocol and second one using UDP. Besides
> forwarding, I also have a rule to log to a local file. Here are the
> relevant sections from my rsyslog.conf. I can post the full file if needed.
>
>
>
> $MainMsgQueueTimeoutEnqueue 0
>
> local4.* action(type="omfwd" target="10.240.219.64"
> action.resumeRetryCount="10" protocol="tcp" port="36456"
> template="Syslogs_ForwardFormat")
> local4.* action(type="omfwd" target="10.16.39.124"
> protocol="udp" port="36456" template="Syslogs_ForwardFormat")
>
> *.* /var/log/syslogs
>
> After the TCP connection is established and messages are getting forwarded,
> say someone pulls out the ethernet cable at the remote syslog collector 1
> that is using tcp protocol. Some number of syslogs from this point are
> still forwarded to the server 2 using udp protocol and written to the local
> file as well. However if the TCP connection to the first server does not
> recover then after some time, we do not see any syslogs in the local file
> or at the second syslog server. I straced rsyslogd and all I see is the
> recvmsg() calls on the /dev/log socket.
>
> How do we tell rsyslog to drop the TCP connection if the other end is no
> longer receiving the syslogs. Perhaps it can drop the connection and retry
> after say X minutes. I tried using resumeRetryCount as documented at
> https://www.rsyslog.com/doc/configuration/actions.html, but that is not
> helping in my situation. Perhaps I'm missing some other "action" setting?
> Appreciate if you can give me a pointer to a sample configuration or point
> me to relevant documentation.
>
> Thank you.
>
>
>
>
> On Mon, Feb 5, 2024 at 11:11?AM David Lang via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
>> you have a queue of 1024 for rabbitmq, if there are more messages than
>> that
>> pending, other processing will stop until the queue can accept more
>> messages.
>> Setup a larger queue (potentially a disk assisted queue) to handle longer
>> outages.
>>
>> you may also want to consider configuring the queue to throw away messages
>> if it
>> gets too full.
>>
>> David Lang
>>
>> On Mon, 5 Feb 2024, Alex via rsyslog wrote:
>>
>>> I have rsyslog (8.2310.0) in docker container from latest alpine image
>> as a
>>> syslog collector with forwarding to logstash.local and rabbitmq.local
>>>
>>> Rsyslog stops working when rabbitmq is unavailable and work fine when
>>> rabbitmq server is up
>>>
>>> How to configure rsyslog to work when the destination is unavailable ?
>>>
>>>
>>> module(load="omrabbitmq")
>>> module(load="imptcp" threads="3")
>>> input(type="imptcp" port="514" ruleset="syslogCollector")
>>>
>>> template(
>>> name="json_syslog"
>>> type="list"
>>> option.json="on"
>>> ) {
>>> constant(value="{")
>>> constant(value="\"@timestamp\":\"")
>>> property(name="timereported" dateFormat="rfc3339")
>>> constant(value="\",\"type\":\"syslog_json")
>>> constant(value="\",\"tag\":\"")
>>> property(name="syslogtag" format="json")
>>> constant(value="\",\"relayhost\":\"") property(name="fromhost")
>>> constant(value="\",\"relayip\":\"") property(name="fromhost-ip")
>>> constant(value="\",\"logsource\":\"") property(name="source")
>>> constant(value="\",\"hostname\":\"") property(name="hostname"
>>> caseconversion="lower")
>>> constant(value="\",\"program\":\"") property(name="programname")
>>> constant(value="\",\"source\":\"") property(name="app-name"
>>> caseConversion="lower" onEmpty="null")
>>> constant(value="\",\"priority\":\"") property(name="pri")
>>> constant(value="\",\"severity\":\"")
>>> property(name="syslogseverity" caseConversion="upper")
>>> constant(value="\",\"facility\":\"")
>> property(name="syslogfacility")
>>> constant(value="\",\"severity_label\":\"")
>>> property(name="syslogseverity-text")
>>> constant(value="\",\"facility_label\":\"")
>>> property(name="syslogfacility-text")
>>> constant(value="\",\"message\":\"") property(name="msg"
>> format="json")
>>> constant(value="\",\"end_msg\":\"")
>>> constant(value="\"}\n")
>>> }
>>>
>>> ruleset(
>>> name="syslogCollector"
>>> ) {
>>> @@elk.local:5000;json_syslog
>>> call send2mqtt
>>> }
>>>
>>> ruleset (
>>> name="send2mqtt"
>>>
>>> queue.type="LinkedList"
>>> queue.size="1024"
>>> queue.dequeueBatchSize="512"
>>> queue.filename="q_mqtt"
>>> queue.saveonshutdown="off"
>>> queue.highwatermark="450"
>>> queue.lowwatermark="50"
>>>
>>> ) {
>>> action(
>>> type="omrabbitmq"
>>> host="rabbitmq.local"
>>> verify_peer="off"
>>> verify_hostname="off"
>>> virtual_host="/"
>>> user="xxx"
>>> password="xxx"
>>> exchange="syslog"
>>> routing_key="messages"
>>> body_template="json_syslog"
>>>
>>> action.resumeRetryCount="0"
>>> action.reportSuspension="on"
>>> action.reportSuspensionContinuation="on"
>>> action.resumeInterval="10"
>>> )
>>> }
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog stop working when output destination (rabbitmq) is unavialable [ In reply to ]

rsyslog at lists

Feb 6, 2024, 10:37 PM

Post #5 of 6 (115 views)

Permalink

My solution is

queue.type="LinkedList"
queue.size="4096"
queue.timeoutEnqueue="0" # timeout for reject new messages if queue
is full
queue.dequeuebatchSize="4000"

??, 7 ????. 2024??. ? 09:03, Prasad Koya via rsyslog <
rsyslog@lists.adiscon.com>:

> Alex, excuse me for using this thread for posting my question. But my
> question is also on the same topic.
>
> I have configured rsyslogd (v8.2102.0) to forward syslogs to two remote
> servers, one using TCP protocol and second one using UDP. Besides
> forwarding, I also have a rule to log to a local file. Here are the
> relevant sections from my rsyslog.conf. I can post the full file if needed.
>
>
>
> $MainMsgQueueTimeoutEnqueue 0
>
> local4.* action(type="omfwd" target="10.240.219.64"
> action.resumeRetryCount="10" protocol="tcp" port="36456"
> template="Syslogs_ForwardFormat")
> local4.* action(type="omfwd" target="10.16.39.124"
> protocol="udp" port="36456" template="Syslogs_ForwardFormat")
>
> *.* /var/log/syslogs
>
> After the TCP connection is established and messages are getting forwarded,
> say someone pulls out the ethernet cable at the remote syslog collector 1
> that is using tcp protocol. Some number of syslogs from this point are
> still forwarded to the server 2 using udp protocol and written to the local
> file as well. However if the TCP connection to the first server does not
> recover then after some time, we do not see any syslogs in the local file
> or at the second syslog server. I straced rsyslogd and all I see is the
> recvmsg() calls on the /dev/log socket.
>
> How do we tell rsyslog to drop the TCP connection if the other end is no
> longer receiving the syslogs. Perhaps it can drop the connection and retry
> after say X minutes. I tried using resumeRetryCount as documented at
> https://www.rsyslog.com/doc/configuration/actions.html, but that is not
> helping in my situation. Perhaps I'm missing some other "action" setting?
> Appreciate if you can give me a pointer to a sample configuration or point
> me to relevant documentation.
>
> Thank you.
>
>
>
>
> On Mon, Feb 5, 2024 at 11:11?AM David Lang via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> > you have a queue of 1024 for rabbitmq, if there are more messages than
> > that
> > pending, other processing will stop until the queue can accept more
> > messages.
> > Setup a larger queue (potentially a disk assisted queue) to handle longer
> > outages.
> >
> > you may also want to consider configuring the queue to throw away
> messages
> > if it
> > gets too full.
> >
> > David Lang
> >
> > On Mon, 5 Feb 2024, Alex via rsyslog wrote:
> >
> > > I have rsyslog (8.2310.0) in docker container from latest alpine image
> > as a
> > > syslog collector with forwarding to logstash.local and rabbitmq.local
> > >
> > > Rsyslog stops working when rabbitmq is unavailable and work fine when
> > > rabbitmq server is up
> > >
> > > How to configure rsyslog to work when the destination is unavailable ?
> > >
> > >
> > > module(load="omrabbitmq")
> > > module(load="imptcp" threads="3")
> > > input(type="imptcp" port="514" ruleset="syslogCollector")
> > >
> > > template(
> > > name="json_syslog"
> > > type="list"
> > > option.json="on"
> > > ) {
> > > constant(value="{")
> > > constant(value="\"@timestamp\":\"")
> > > property(name="timereported" dateFormat="rfc3339")
> > > constant(value="\",\"type\":\"syslog_json")
> > > constant(value="\",\"tag\":\"")
> > > property(name="syslogtag" format="json")
> > > constant(value="\",\"relayhost\":\"") property(name="fromhost")
> > > constant(value="\",\"relayip\":\"")
> property(name="fromhost-ip")
> > > constant(value="\",\"logsource\":\"") property(name="source")
> > > constant(value="\",\"hostname\":\"") property(name="hostname"
> > > caseconversion="lower")
> > > constant(value="\",\"program\":\"")
> property(name="programname")
> > > constant(value="\",\"source\":\"") property(name="app-name"
> > > caseConversion="lower" onEmpty="null")
> > > constant(value="\",\"priority\":\"") property(name="pri")
> > > constant(value="\",\"severity\":\"")
> > > property(name="syslogseverity" caseConversion="upper")
> > > constant(value="\",\"facility\":\"")
> > property(name="syslogfacility")
> > > constant(value="\",\"severity_label\":\"")
> > > property(name="syslogseverity-text")
> > > constant(value="\",\"facility_label\":\"")
> > > property(name="syslogfacility-text")
> > > constant(value="\",\"message\":\"") property(name="msg"
> > format="json")
> > > constant(value="\",\"end_msg\":\"")
> > > constant(value="\"}\n")
> > > }
> > >
> > > ruleset(
> > > name="syslogCollector"
> > > ) {
> > > @@elk.local:5000;json_syslog
> > > call send2mqtt
> > > }
> > >
> > > ruleset (
> > > name="send2mqtt"
> > >
> > > queue.type="LinkedList"
> > > queue.size="1024"
> > > queue.dequeueBatchSize="512"
> > > queue.filename="q_mqtt"
> > > queue.saveonshutdown="off"
> > > queue.highwatermark="450"
> > > queue.lowwatermark="50"
> > >
> > > ) {
> > > action(
> > > type="omrabbitmq"
> > > host="rabbitmq.local"
> > > verify_peer="off"
> > > verify_hostname="off"
> > > virtual_host="/"
> > > user="xxx"
> > > password="xxx"
> > > exchange="syslog"
> > > routing_key="messages"
> > > body_template="json_syslog"
> > >
> > > action.resumeRetryCount="0"
> > > action.reportSuspension="on"
> > > action.reportSuspensionContinuation="on"
> > > action.resumeInterval="10"
> > > )
> > > }
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> > >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Rsyslog stop working when output destination (rabbitmq) is unavialable [ In reply to ]

rsyslog at lists

Feb 7, 2024, 2:20 AM

Post #6 of 6 (115 views)

Permalink

just a note that a batch size of 4000 doesn't make a lot of sense when the queue
size is only 4096 :-)

batch sizes are a case of diminishing returns. There is a huge advantage of
going from to around 100 or so, but far less gains in going from 100 to 1000
(unless you are doing database inserts or similar)

On Wed, 7 Feb 2024, Alex via rsyslog wrote:

> My solution is
>
> queue.type="LinkedList"
> queue.size="4096"
> queue.timeoutEnqueue="0" # timeout for reject new messages if queue
> is full
> queue.dequeuebatchSize="4000"
>
>
> ??, 7 ????. 2024??. ? 09:03, Prasad Koya via rsyslog <
> rsyslog@lists.adiscon.com>:
>
>> Alex, excuse me for using this thread for posting my question. But my
>> question is also on the same topic.
>>
>> I have configured rsyslogd (v8.2102.0) to forward syslogs to two remote
>> servers, one using TCP protocol and second one using UDP. Besides
>> forwarding, I also have a rule to log to a local file. Here are the
>> relevant sections from my rsyslog.conf. I can post the full file if needed.
>>
>>
>>
>> $MainMsgQueueTimeoutEnqueue 0
>>
>> local4.* action(type="omfwd" target="10.240.219.64"
>> action.resumeRetryCount="10" protocol="tcp" port="36456"
>> template="Syslogs_ForwardFormat")
>> local4.* action(type="omfwd" target="10.16.39.124"
>> protocol="udp" port="36456" template="Syslogs_ForwardFormat")
>>
>> *.* /var/log/syslogs
>>
>> After the TCP connection is established and messages are getting forwarded,
>> say someone pulls out the ethernet cable at the remote syslog collector 1
>> that is using tcp protocol. Some number of syslogs from this point are
>> still forwarded to the server 2 using udp protocol and written to the local
>> file as well. However if the TCP connection to the first server does not
>> recover then after some time, we do not see any syslogs in the local file
>> or at the second syslog server. I straced rsyslogd and all I see is the
>> recvmsg() calls on the /dev/log socket.
>>
>> How do we tell rsyslog to drop the TCP connection if the other end is no
>> longer receiving the syslogs. Perhaps it can drop the connection and retry
>> after say X minutes. I tried using resumeRetryCount as documented at
>> https://www.rsyslog.com/doc/configuration/actions.html, but that is not
>> helping in my situation. Perhaps I'm missing some other "action" setting?
>> Appreciate if you can give me a pointer to a sample configuration or point
>> me to relevant documentation.
>>
>> Thank you.
>>
>>
>>
>>
>> On Mon, Feb 5, 2024 at 11:11?AM David Lang via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>>
>> > you have a queue of 1024 for rabbitmq, if there are more messages than
>> > that
>> > pending, other processing will stop until the queue can accept more
>> > messages.
>> > Setup a larger queue (potentially a disk assisted queue) to handle longer
>> > outages.
>> >
>> > you may also want to consider configuring the queue to throw away
>> messages
>> > if it
>> > gets too full.
>> >
>> > David Lang
>> >
>> > On Mon, 5 Feb 2024, Alex via rsyslog wrote:
>> >
>> > > I have rsyslog (8.2310.0) in docker container from latest alpine image
>> > as a
>> > > syslog collector with forwarding to logstash.local and rabbitmq.local
>> > >
>> > > Rsyslog stops working when rabbitmq is unavailable and work fine when
>> > > rabbitmq server is up
>> > >
>> > > How to configure rsyslog to work when the destination is unavailable ?
>> > >
>> > >
>> > > module(load="omrabbitmq")
>> > > module(load="imptcp" threads="3")
>> > > input(type="imptcp" port="514" ruleset="syslogCollector")
>> > >
>> > > template(
>> > > name="json_syslog"
>> > > type="list"
>> > > option.json="on"
>> > > ) {
>> > > constant(value="{")
>> > > constant(value="\"@timestamp\":\"")
>> > > property(name="timereported" dateFormat="rfc3339")
>> > > constant(value="\",\"type\":\"syslog_json")
>> > > constant(value="\",\"tag\":\"")
>> > > property(name="syslogtag" format="json")
>> > > constant(value="\",\"relayhost\":\"") property(name="fromhost")
>> > > constant(value="\",\"relayip\":\"")
>> property(name="fromhost-ip")
>> > > constant(value="\",\"logsource\":\"") property(name="source")
>> > > constant(value="\",\"hostname\":\"") property(name="hostname"
>> > > caseconversion="lower")
>> > > constant(value="\",\"program\":\"")
>> property(name="programname")
>> > > constant(value="\",\"source\":\"") property(name="app-name"
>> > > caseConversion="lower" onEmpty="null")
>> > > constant(value="\",\"priority\":\"") property(name="pri")
>> > > constant(value="\",\"severity\":\"")
>> > > property(name="syslogseverity" caseConversion="upper")
>> > > constant(value="\",\"facility\":\"")
>> > property(name="syslogfacility")
>> > > constant(value="\",\"severity_label\":\"")
>> > > property(name="syslogseverity-text")
>> > > constant(value="\",\"facility_label\":\"")
>> > > property(name="syslogfacility-text")
>> > > constant(value="\",\"message\":\"") property(name="msg"
>> > format="json")
>> > > constant(value="\",\"end_msg\":\"")
>> > > constant(value="\"}\n")
>> > > }
>> > >
>> > > ruleset(
>> > > name="syslogCollector"
>> > > ) {
>> > > @@elk.local:5000;json_syslog
>> > > call send2mqtt
>> > > }
>> > >
>> > > ruleset (
>> > > name="send2mqtt"
>> > >
>> > > queue.type="LinkedList"
>> > > queue.size="1024"
>> > > queue.dequeueBatchSize="512"
>> > > queue.filename="q_mqtt"
>> > > queue.saveonshutdown="off"
>> > > queue.highwatermark="450"
>> > > queue.lowwatermark="50"
>> > >
>> > > ) {
>> > > action(
>> > > type="omrabbitmq"
>> > > host="rabbitmq.local"
>> > > verify_peer="off"
>> > > verify_hostname="off"
>> > > virtual_host="/"
>> > > user="xxx"
>> > > password="xxx"
>> > > exchange="syslog"
>> > > routing_key="messages"
>> > > body_template="json_syslog"
>> > >
>> > > action.resumeRetryCount="0"
>> > > action.reportSuspension="on"
>> > > action.reportSuspensionContinuation="on"
>> > > action.resumeInterval="10"
>> > > )
>> > > }
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > DON'T LIKE THAT.
>> > >
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > DON'T LIKE THAT.
>> >
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.