Mailing List Archive: No longer writing to file when forwarding fails (zero window)

No longer writing to file when forwarding fails (zero window)

Sep 17, 2019, 3:04 AM

Post #1 of 7 (645 views)

Hello,

We have an rsyslog instance in front of our ELK
(Elasticsearch/Logstash/Kibana) cluster. (-> Rsyslog -> Logstash ->
Elasticsearch)

The main reason for this is to still have plain-text logs in case the
more complex ELK stuff fails.

When the disk fills up, Elasticsearch stops indexing. Logstash (in front
of Elasticsearch) keeps the TCP connections from Rsyslog open, but sets
zero window. As a consequence, everything is coming to a grinding halt.

Rsyslog version is v8.33.1, and the configuration is pretty
straightforward:

-------------------------------------
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")

template(name="RSYSLOG_SyslogProtocolFormatFallback" type="string"
string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\n")
template(name="FileNameByHost" type="string"
string="/syslog/%fromhost-ip%.log")

*.* action(type="omfile" dynaFile="FileNameByHost"
template="RSYSLOG_FileFormat")

action(type="omfwd" target="log02e.secu.lu" port="1514" protocol="tcp"
template="RSYSLOG_ForwardFormat")
action(type="omfwd" target="log03e.secu.lu" port="1514" protocol="tcp"
template="RSYSLOG_ForwardFormat"
action.execOnlyWhenPreviousIsSuspended="on")
action(type="omfile" file="/syslog/localbuffer"
template="RSYSLOG_SyslogProtocolFormatFallback"
action.execOnlyWhenPreviousIsSuspended="on")
-------------------------------------

Neither the first nor the last call to omfile has any effect if omfwd
stalls...

What can be done about this?

Thanks,
Marki
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 17, 2019, 5:30 AM

Post #2 of 7 (645 views)

Permalink

This is expected behavior, when one output stalls, all log processing in that
ruleset pauses. The fix for this is to setup a queue for that action (probably a
disk assisted queue since ES may be down for a while)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 17, 2019, 7:27 AM

Post #3 of 7 (645 views)

Permalink

Hmm ok, that means I can't force it to just write a plaintext log in any
case, no matter what happens to the following output modules?

Would there be some creative but not too complex way to accomplish that
without needing more disk space for queueing the output going toward ES?
In the worst case it can be lost and we would be able to reinject it
from the plaintext logs probably...

The main reason is that I want the plaintext output to be out of rsyslog
as fast as possible. Imagine for example the disk fills up while
queueing, then we have nothing at all. No plaintext logs, and nothing
else either.

Thanks.

On 2019-09-17 14:30, David Lang wrote:
> This is expected behavior, when one output stalls, all log processing
> in that ruleset pauses. The fix for this is to setup a queue for that
> action (probably a disk assisted queue since ES may be down for a
> while)
>
> David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 17, 2019, 9:14 AM

Post #4 of 7 (645 views)

Permalink

Ok, so I've done more tests with this: there is also a conflict here,
because I'm using this in a failover scenario:

(a) action(type="omfile" dynaFile="FileNameByHost")
(b) action(type="omfwd" target="log02" port="1514" protocol="tcp")
(c) action(type="omfwd" target="log03" port="1514" protocol="tcp"
action.execOnlyWhenPreviousIsSuspended="on")
(d) action(type="omfile" file="/rsyslogq/localbuffer"
action.execOnlyWhenPreviousIsSuspended="on")

If I assign a (DA-)queue to (b), it will not fail over to (c), which I
want it to. Instead, it will start queueing.

If I /do not/ assign a queue to (b), it will only fail over to (c), if
the connection is refused (action becomes suspended), not if it
/stalls/.

In the end, I want
1. (b) to ail over to (c) no matter why (b) is currently failing.
2. (a) to write a plaintext log in any case. We could consider (b) and
(c) luxury and ignore them if they fail.

On 2019-09-17 14:30, David Lang wrote:
> This is expected behavior, when one output stalls, all log processing
> in that ruleset pauses. The fix for this is to setup a queue for that
> action (probably a disk assisted queue since ES may be down for a
> while)
>
> David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 17, 2019, 1:50 PM

Post #5 of 7 (645 views)

Permalink

put b and c in a separate ruleset and put the queue on that ruleset rather than
the individual actions

David Lang

On Tue, 17 Sep 2019, Marki via rsyslog wrote:

> Date: Tue, 17 Sep 2019 18:14:08 +0200
> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> To: Marki via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Marki <rsyslog@lists.roth.lu>
> Subject: Re: [rsyslog] No longer writing to file when forwarding fails (zero
> window)
>
> Ok, so I've done more tests with this: there is also a conflict here, because
> I'm using this in a failover scenario:
>
> (a) action(type="omfile" dynaFile="FileNameByHost")
> (b) action(type="omfwd" target="log02" port="1514" protocol="tcp")
> (c) action(type="omfwd" target="log03" port="1514" protocol="tcp"
> action.execOnlyWhenPreviousIsSuspended="on")
> (d) action(type="omfile" file="/rsyslogq/localbuffer"
> action.execOnlyWhenPreviousIsSuspended="on")
>
> If I assign a (DA-)queue to (b), it will not fail over to (c), which I want
> it to. Instead, it will start queueing.
>
> If I /do not/ assign a queue to (b), it will only fail over to (c), if the
> connection is refused (action becomes suspended), not if it /stalls/.
>
> In the end, I want
> 1. (b) to ail over to (c) no matter why (b) is currently failing.
> 2. (a) to write a plaintext log in any case. We could consider (b) and (c)
> luxury and ignore them if they fail.
>
>
> On 2019-09-17 14:30, David Lang wrote:
>> This is expected behavior, when one output stalls, all log processing
>> in that ruleset pauses. The fix for this is to setup a queue for that
>> action (probably a disk assisted queue since ES may be down for a
>> while)
>>
>> David Lang
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 17, 2019, 11:30 PM

Post #6 of 7 (645 views)

Permalink

Yeah, somehow creating another instance was my conclusion too so I tried
using UDP forwarding to feed a second instance with a dedicated queue.

It does indeed work that way but what I don't understand so far is: Why
is output to file still stalling when for example the queue of the
second instance is full? It looks like the UDP connection is being
throttled too in that case, while I thought/hoped messages would simply
be lost (which would actually be the preferred way).

---------------
module(load="imtcp")
module(load="imudp")

input(type="imtcp" port="514")
input(type="imudp" port="514")

template(name="FileNameByHost" type="string"
string="/syslog/%fromhost-ip%.log")
action(type="omfile" dynaFile="FileNameByHost"
template="RSYSLOG_FileFormat")
action(type="omfwd" target="localhost" port="515" protocol="udp"
template="RSYSLOG_ForwardFormat")

ruleset(name="asyncelk" queue.type="linkedlist" queue.filename="rsysq")
{
action(type="omfwd" target="log02" port="1514" protocol="tcp"
template="RSYSLOG_ForwardFormat")
action(type="omfwd" target="log03" port="1514" protocol="tcp"
template="RSYSLOG_ForwardFormat"
action.execOnlyWhenPreviousIsSuspended="on")
}
input(type="imudp" port="515" address="127.0.0.1" ruleset="asyncelk")
---------------

On 2019-09-17 22:50, David Lang wrote:
> put b and c in a separate ruleset and put the queue on that ruleset
> rather than the individual actions
>
> David Lang
>
> On Tue, 17 Sep 2019, Marki via rsyslog wrote:
>
>> Ok, so I've done more tests with this: there is also a conflict here,
>> because I'm using this in a failover scenario:
>>
>> (a) action(type="omfile" dynaFile="FileNameByHost")
>> (b) action(type="omfwd" target="log02" port="1514" protocol="tcp")
>> (c) action(type="omfwd" target="log03" port="1514" protocol="tcp"
>> action.execOnlyWhenPreviousIsSuspended="on")
>> (d) action(type="omfile" file="/rsyslogq/localbuffer"
>> action.execOnlyWhenPreviousIsSuspended="on")
>>
>> If I assign a (DA-)queue to (b), it will not fail over to (c), which I
>> want it to. Instead, it will start queueing.
>>
>> If I /do not/ assign a queue to (b), it will only fail over to (c), if
>> the connection is refused (action becomes suspended), not if it
>> /stalls/.
>>
>> In the end, I want
>> 1. (b) to ail over to (c) no matter why (b) is currently failing.
>> 2. (a) to write a plaintext log in any case. We could consider (b) and
>> (c) luxury and ignore them if they fail.
>>
>>
>> On 2019-09-17 14:30, David Lang wrote:
>>> This is expected behavior, when one output stalls, all log processing
>>> in that ruleset pauses. The fix for this is to setup a queue for that
>>> action (probably a disk assisted queue since ES may be down for a
>>> while)
>>>
>>> David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: No longer writing to file when forwarding fails (zero window) [ In reply to ]

rsyslog at lists

Sep 18, 2019, 4:20 AM

Post #7 of 7 (645 views)

Permalink

sorry, I'm not sure where a second instance comes in here. but UDP forwarding
should not ever detect that what it's sending to is not processing, although
localhost is funny so there may be some special case that's happening there. But
there's no need to forward to yourself to invoke the ruleset

This is what I would have expected

module(load="imtcp")
module(load="imudp")

input(type="imtcp" port="514")
input(type="imudp" port="514")

template(name="FileNameByHost" type="string"
string="/syslog/%fromhost-ip%.log")
action(type="omfile" dynaFile="FileNameByHost" template="RSYSLOG_FileFormat")
call asyncelk

ruleset(name="asyncelk" queue.type="linkedlist" queue.filename="rsysq") {
action(type="omfwd" target="log02" port="1514" protocol="tcp" template="RSYSLOG_ForwardFormat")
action(type="omfwd" target="log03" port="1514" protocol="tcp" template="RSYSLOG_ForwardFormat" action.execOnlyWhenPreviousIsSuspended="on")
}

On Wed, 18 Sep 2019, Marki via rsyslog wrote:

> Date: Wed, 18 Sep 2019 08:30:11 +0200
> From: Marki via rsyslog <rsyslog@lists.adiscon.com>
> To: Marki via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Marki <rsyslog@lists.roth.lu>
> Subject: Re: [rsyslog] No longer writing to file when forwarding fails (zero
> window)
>
> Yeah, somehow creating another instance was my conclusion too so I tried
> using UDP forwarding to feed a second instance with a dedicated queue.
>
> It does indeed work that way but what I don't understand so far is: Why is
> output to file still stalling when for example the queue of the second
> instance is full? It looks like the UDP connection is being throttled too in
> that case, while I thought/hoped messages would simply be lost (which would
> actually be the preferred way).
>
> ---------------
> module(load="imtcp")
> module(load="imudp")
>
> input(type="imtcp" port="514")
> input(type="imudp" port="514")
>
> template(name="FileNameByHost" type="string"
> string="/syslog/%fromhost-ip%.log")
> action(type="omfile" dynaFile="FileNameByHost" template="RSYSLOG_FileFormat")
> action(type="omfwd" target="localhost" port="515" protocol="udp"
> template="RSYSLOG_ForwardFormat")
>
> ruleset(name="asyncelk" queue.type="linkedlist" queue.filename="rsysq") {
> action(type="omfwd" target="log02" port="1514" protocol="tcp"
> template="RSYSLOG_ForwardFormat")
> action(type="omfwd" target="log03" port="1514" protocol="tcp"
> template="RSYSLOG_ForwardFormat" action.execOnlyWhenPreviousIsSuspended="on")
> }
> input(type="imudp" port="515" address="127.0.0.1" ruleset="asyncelk")
> ---------------
>
> On 2019-09-17 22:50, David Lang wrote:
>> put b and c in a separate ruleset and put the queue on that ruleset
>> rather than the individual actions
>>
>> David Lang
>>
>> On Tue, 17 Sep 2019, Marki via rsyslog wrote:
>>
>>> Ok, so I've done more tests with this: there is also a conflict here,
>>> because I'm using this in a failover scenario:
>>>
>>> (a) action(type="omfile" dynaFile="FileNameByHost")
>>> (b) action(type="omfwd" target="log02" port="1514" protocol="tcp")
>>> (c) action(type="omfwd" target="log03" port="1514" protocol="tcp"
>>> action.execOnlyWhenPreviousIsSuspended="on")
>>> (d) action(type="omfile" file="/rsyslogq/localbuffer"
>>> action.execOnlyWhenPreviousIsSuspended="on")
>>>
>>> If I assign a (DA-)queue to (b), it will not fail over to (c), which I
>>> want it to. Instead, it will start queueing.
>>>
>>> If I /do not/ assign a queue to (b), it will only fail over to (c), if the
>>> connection is refused (action becomes suspended), not if it /stalls/.
>>>
>>> In the end, I want
>>> 1. (b) to ail over to (c) no matter why (b) is currently failing.
>>> 2. (a) to write a plaintext log in any case. We could consider (b) and (c)
>>> luxury and ignore them if they fail.
>>>
>>>
>>> On 2019-09-17 14:30, David Lang wrote:
>>>> This is expected behavior, when one output stalls, all log processing
>>>> in that ruleset pauses. The fix for this is to setup a queue for that
>>>> action (probably a disk assisted queue since ES may be down for a
>>>> while)
>>>>
>>>> David Lang
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.