Mailing List Archive: bind one queue to a ruleset for all syslogs going to one server.

bind one queue to a ruleset for all syslogs going to one server.

Feb 13, 2024, 1:43 PM

Post #1 of 5 (98 views)

Hi

Before I explain what I'm trying to solve, here is our rsyslog.conf:

module( load="imuxsock" )
module( load="imklog" )
module( load="impstats" interval="60" severity="7" log.syslog="off"
log.file="/var/log/rsyslog_stats")

$MainMsgQueueTimeoutEnqueue 0

template( name="Msg_ForwardFormat_info" type="list" ) {
constant( value="<166>" )
property( name="timestamp" dateFormat="rfc3164" )
constant( value=" al210 " )
property( name="syslogtag" position.from="1" position.to="32" )
property( name="msg" spifno1stsp="on" )
property( name="msg" )
}
template( name="Msg_ForwardFormat_notice" type="list" ) {
constant( value="<166>" )
property( name="timestamp" dateFormat="rfc3164" )
constant( value=" al210 " )
property( name="syslogtag" position.from="1" position.to="32" )
property( name="msg" spifno1stsp="on" )
property( name="msg" )
}

# Forward to syslog server 1
local4.=info action(type="omfwd" target="10.240.127.159"
protocol="tcp" queue.type="LinkedList" port="34567"
template="Msg_ForwardFormat_info")
local4.=notice action(type="omfwd" target="10.240.127.159"
protocol="tcp" queue.type="LinkedList" port="34567"
template="Msg_ForwardFormat_notice")

# Forward to syslog server 2
local4.=info action(type="omfwd" target="10.240.127.199"
protocol="tcp" queue.type="LinkedList" port="34567"
template="Msg_ForwardFormat_info")
local4.=notice action(type="omfwd" target="10.240.127.199"
protocol="tcp" queue.type="LinkedList" port="34567"
template="Msg_ForwardFormat_notice")

Except for the prefix, the templates Msg_ForwardFormat_Info,
Msg_ForwardFormat_notice are exactly the same. We defined them separately
so we can prefix the message being forwarded with the original
facility/severity tag.

We'd like to use queues in front of the actions so if one TCP connection is
stalled, we do not want forwarding to another TCP connection or logging to
a local file being stalled.

If rsyslog receives 2 messages from 2 tasks in the system, where one is "
local4.info" and the other is "local4.notice", that would result in 4 TCP
connections (due to 4 different action queues/threads above).

Consider this case where an application sends 2 syslogs:
at time t1, a local4.info message
at time t1+few microseconds, a local4.notice message

While these messages are deep copied and enqueued to respective action
queues, it's possible that the second message may reach the remote syslog
server before the first message. I'd like to avoid that situation.

How do I define one queue for all syslogs forwarded to a server
by preserving the facility/severity in each message.

Thank you.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: bind one queue to a ruleset for all syslogs going to one server. [ In reply to ]

rsyslog at lists

Feb 13, 2024, 11:29 PM

Post #2 of 5 (98 views)

Permalink

There is no guarantee either in the syslog itself that the delivery of
the events will be in the order they were sent. Especially if you're
using UDP.

Also - I don't think there is any guarantee for two queues to be
processed "in parallel".

Regardless of whether you have rsyslog or anything else in the middle,
if you need this level of precision, you need to have sufficiently
precise timestamp so you can order your events by timestamps. Otherwise
it's just an educated guess. That's why TCP has sequence numbers - you
can't rely on the order by which you received packets from the network.

That's the general comment.

In a very particular case - if you're receiving the events from a single
source over a single TCP connection, you can have a reasonable
expectations for rsyslog to receive them in order but I'm not 100% sure
if/how you can make rsyslog make sure it doesn't do any "interleaving"
in terms of output actions. Someone knowing the internals better would
need to elaborate on this more.

You could try pushing all events to a single ruleset with a queue with
just one worker thread. I suppose then the events dequeued from the main
queue and enqueued into the ruleset queue would be in order. I'm not
fully sure about the order of events enqueued into action queues though.
I'm curious myself.

MK

On 13.02.2024 22:43, Prasad Koya via rsyslog wrote:
> Hi
>
> Before I explain what I'm trying to solve, here is our rsyslog.conf:
>
> module( load="imuxsock" )
> module( load="imklog" )
> module( load="impstats" interval="60" severity="7" log.syslog="off"
> log.file="/var/log/rsyslog_stats")
>
> $MainMsgQueueTimeoutEnqueue 0
>
> template( name="Msg_ForwardFormat_info" type="list" ) {
> constant( value="<166>" )
> property( name="timestamp" dateFormat="rfc3164" )
> constant( value=" al210 " )
> property( name="syslogtag" position.from="1" position.to="32" )
> property( name="msg" spifno1stsp="on" )
> property( name="msg" )
> }
> template( name="Msg_ForwardFormat_notice" type="list" ) {
> constant( value="<166>" )
> property( name="timestamp" dateFormat="rfc3164" )
> constant( value=" al210 " )
> property( name="syslogtag" position.from="1" position.to="32" )
> property( name="msg" spifno1stsp="on" )
> property( name="msg" )
> }
>
> # Forward to syslog server 1
> local4.=info action(type="omfwd" target="10.240.127.159"
> protocol="tcp" queue.type="LinkedList" port="34567"
> template="Msg_ForwardFormat_info")
> local4.=notice action(type="omfwd" target="10.240.127.159"
> protocol="tcp" queue.type="LinkedList" port="34567"
> template="Msg_ForwardFormat_notice")
>
> # Forward to syslog server 2
> local4.=info action(type="omfwd" target="10.240.127.199"
> protocol="tcp" queue.type="LinkedList" port="34567"
> template="Msg_ForwardFormat_info")
> local4.=notice action(type="omfwd" target="10.240.127.199"
> protocol="tcp" queue.type="LinkedList" port="34567"
> template="Msg_ForwardFormat_notice")
>
> Except for the prefix, the templates Msg_ForwardFormat_Info,
> Msg_ForwardFormat_notice are exactly the same. We defined them separately
> so we can prefix the message being forwarded with the original
> facility/severity tag.
>
> We'd like to use queues in front of the actions so if one TCP connection is
> stalled, we do not want forwarding to another TCP connection or logging to
> a local file being stalled.
>
> If rsyslog receives 2 messages from 2 tasks in the system, where one is "
> local4.info" and the other is "local4.notice", that would result in 4 TCP
> connections (due to 4 different action queues/threads above).
>
> Consider this case where an application sends 2 syslogs:
> at time t1, a local4.info message
> at time t1+few microseconds, a local4.notice message
>
> While these messages are deep copied and enqueued to respective action
> queues, it's possible that the second message may reach the remote syslog
> server before the first message. I'd like to avoid that situation.
>
> How do I define one queue for all syslogs forwarded to a server
> by preserving the facility/severity in each message.
>
> Thank you.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: bind one queue to a ruleset for all syslogs going to one server. [ In reply to ]

rsyslog at lists

Feb 14, 2024, 12:49 AM

Post #3 of 5 (98 views)

Permalink

> You could try pushing all events to a single ruleset with a queue with
> just one worker thread. I suppose then the events dequeued from the main
> queue and enqueued into the ruleset queue would be in order. I'm not
> fully sure about the order of events enqueued into action queues though.
> I'm curious myself.

In this case, it still depends on the main queue actions and settings.
To be pretty sure, it is best to bind the input directly to the
ruleset you mention. The only problem that can occur is outages of the
receiving server. During recovery, some messages may be requeued, and
this at a different queue position.

I have seen some extreme cases of "I always need to preserve
sequence". You can ensure this via

Bind input to a ruleset with a DIRECT mode queue, do not use any other
queue definitions in that ruleset, set potentially failing actions to
retry eternally.

Obviously, this will deliver pretty bad performance.

Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: bind one queue to a ruleset for all syslogs going to one server. [ In reply to ]

rsyslog at lists

Feb 14, 2024, 10:04 AM

Post #4 of 5 (98 views)

Permalink

Thanks for replying.

The problem I'm trying to solve is when forwarding syslogs to multiple
servers over TCP. Yes, UDP is unreliable and out of order messages are
expected.

I'm understanding more about rulesets. I'll get back to the thread after
some more work on this.

On Wed, Feb 14, 2024 at 12:50?AM Rainer Gerhards via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> > You could try pushing all events to a single ruleset with a queue with
> > just one worker thread. I suppose then the events dequeued from the main
> > queue and enqueued into the ruleset queue would be in order. I'm not
> > fully sure about the order of events enqueued into action queues though.
> > I'm curious myself.
>
> In this case, it still depends on the main queue actions and settings.
> To be pretty sure, it is best to bind the input directly to the
> ruleset you mention. The only problem that can occur is outages of the
> receiving server. During recovery, some messages may be requeued, and
> this at a different queue position.
>
> I have seen some extreme cases of "I always need to preserve
> sequence". You can ensure this via
>
> Bind input to a ruleset with a DIRECT mode queue, do not use any other
> queue definitions in that ruleset, set potentially failing actions to
> retry eternally.
>
> Obviously, this will deliver pretty bad performance.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: bind one queue to a ruleset for all syslogs going to one server. [ In reply to ]

rsyslog at lists

Feb 14, 2024, 1:29 PM

Post #5 of 5 (94 views)

Permalink

On Tue, 13 Feb 2024, Prasad Koya via rsyslog wrote:

> While these messages are deep copied and enqueued to respective action
> queues, it's possible that the second message may reach the remote syslog
> server before the first message. I'd like to avoid that situation.
>
> How do I define one queue for all syslogs forwarded to a server
> by preserving the facility/severity in each message.

you can't. Rsyslog can use multiple threads, and the threads do not even try to
corrdinate between them to make sure they are sending messages in order.

by putting the forward action in a single ruleset, and calling that ruleset
instead of invoking the action, you will make it less likely that things will be
out of order, but only less likely.

At one point in the distant past, rsyslog tried really hard to send the messages
out in the order that they were received, but I pointed out quite a few corner
cases where that would be defeated by the network (especially if sending through
relay servers) and rsyslog abandoned that effort, and in the process increased
it's throughput by several orders of magnatude.

In general, messages still probably arrive in order, but not always. And when
you are talking about logs sent from multiple machines, it becomes even
impossible to guarantee reception of logs from multiple machines in the order
they were generated (and in this case anything you do to make log delivery more
reliable like TCP or RELP, makes the possible delays in message delivery worse)

As noted, you if you care about absolute ordering of logs, you need good, high
precision timestamps, especially across multiple machines.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.