No sir, no encryption is in play. Just plain ol TCP syslog. We're running rsyslog v8.1911
I don't think we've enforced any additional dynafile settings, but i'll double check there's not an override I don't know about. Do youz have a rough order guidance for dynafilecachesize settings?
After getting thoroughly lost in the Google rabbit hole, I had other questions;
Would using imptcp over imtcp help?
* I saw mentions that imptcp handles connections better/performs better.
Would setting the KeepAlives in the rsyslog config on the server-side help to manage the (zombie?) TCP connections.?
* The load balancer being in the middle feels like it's the cause of repeated ESTABLISHED connections, but to keep HA/redundancy it's kind of a necessary evil.
For reference, our main input config looks like this. There are a number of other input entries, but they're all variations on this for different log source types (servers, apps, etc).
module(load="imudp")
module(load="imtcp" MaxListeners="100" AddtlFrameDelimiter="000")
input(type="imudp" port="10514" ruleset="firewall_rule")
input(type="imtcp" port="10514" ruleset="firewall_rule")
template(name="firewall_logs" type="string" string="/data/logs/firewall/10514/%fromhost-ip%/syslog.log")
ruleset(name="firewall_rule") {
action(type="omfile"
FileCreateMode="0744"
DirCreateMode="0755"
FileOwner="loguser"
FileGroup="loguser"
DirOwner="loguser"
DirGroup="loguser"
DynaFile="firewall_logs")
}
.
.
.
[snip]
Thank again, really appreciate the insight.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
Sent: Sunday, April 24, 2022 8:57 AM
To: David Lang <david@lang.hm>; Steven D via rsyslog <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerless@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
David
Thanks for the depth of this reply. Let me feed back in some additional info.
We've removed the load balancer from the syslog path as part of troubleshooting and the behavior didn't change. We continued to see log loss using TCP(most notably for our firewalls), even when directing to a single receiver server.
So far as the network goes there is a clean path from sending log source (firewall) to the rsyslog receiver. So I guess I'm looking for guidance on what knobs I should look at turning on the RHEL/rsyslog side.
As far as RELP, will that receive standard inbound TCP syslog on the receiver server? I was under the impression (possibly mistaken) both sender/receiver needed to use RELP. I'm happy to test using that module versus imtcp if I'm wrong.
Thanks again.
Regards,
Steven.
-------- Original message --------
From: David Lang <david@lang.hm>
Date: 4/24/22 8:27 AM (GMT-05:00)
To: Steven D via rsyslog <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerless@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
One problem with TCP load balancing of syslog messages is that the load
balancers do not understand the syslog protocol, so they can't rebalance at a
message boundry.
A second problem is that when a firewall or load balancer drops a connection,
the sender doesn't know that it's dropped until the next time it tries to
deliver a message. Since TCP doesn't have any way for the OS TCP stack to tell
software "that message that you submitted to an open connection, and I accepted,
it can't be delivered" (once the OS accepts the message, the sender has to
assume that it will be delivered)
As a result, it's very easy for TCP syslog to be less reliable than UDP syslog.
The 'common sense wisdom' is that TCP is reliable because dropped packets inside
an ongoing connection will get retried, but dropped packets are actually very
uncommon inside a datacenter. They may happen when a firewall/router is
overloaded, but it's not very common. Back in 2006 or so I did testing and found
that within a local network, UDP was almost perfectly reliable (as long as the
receiver could keep up and not overflow the OS buffers)
Rsyslog has the rebindinterval feature, which tells the sender to disconnect and
reconnect periodically so that the load balancer has a chance to make a new
balancing decision.
you also want to make sure that the log stram is not idle for too long ('mark'
was the historical method of doing that, I prefer vmstat 60 |logger -t vmstat as
it's not much larger and an extremely dense set of information that can be very
useful when troubleshooting)
The other thing to look at is the RELP protocol, it was developed specifically
because TCP was designed to be reliable over an unreliable wire, but assumes
that both ends will remain up and the connection will not be cut by a middlebox.
RELP does full application level acks so that the sender knows that the receiver
rsyslog actually processed the message
with plain TCP, once the sending software submits data to the OS stack and the
OS stack says it's accepted the data, the data then sits in a buffer on the
sending machine, then gets sent over the wire (with retries), then sits in a
buffer on the receiving machine until the receiving software reads it. If
anything causes the connection to be terminated (firewall, load balancer, crash
on the receiving machine, etc) the data will be lost and the sending software
has no way of learning about it.
David Lang
On Sun, 24 Apr 2022, Steven D via rsyslog wrote:
> Date: Sun, 24 Apr 2022 12:14:35 +0000
> From: Steven D via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: Steven D <pheerless@hotmail.com>
> Subject: [rsyslog] Basic Rsyslog Troubleshooting
>
> Greetings list
>
> New to rsyslog list, not new to logging. We're experiencing an odd issue where TCP syslog messages are being dropped at seemingly random intervals...hoping to get some input.
>
> The TLDR on our architecture is we have set up a couple rsyslog receivers behind a Netscaler Load balancer. Multiple platforms/devices are configured to send syslog to the load balancer, which distributes to the receivers. Receivers are running RHEL v8 and rsyslog v8.1911. Receivers write files to disk, which we then read with a SIEM agent.
>
> We've got a modestly sized environment with a syslog client base of 200-300 servers, 30 networking devices (including firewalls) and some applications all directing logging to the load balancer.
>
> Our config file is pretty vanilla, no cache, or advanced tweaks. Just using the "imtcp" and "imudp" modules and rulesets to write files to disk based on the sending host IP/port.
>
> The first problem we're seeing is that hosts sending via TCP have log messages missed (never written to disk), where UDP seems more reliable. When switching the firewalls to UDP, throughput nearly doubles and message loss is less noticeable (yeah I know it's still UDP).
>
> Possibly related is that we've noticed that each receiver also holds a lot of "Established" connections for back to the clients, but different ports. (Possible session/connection exhaustion?)
>
> Any guidance on how we can approach and troubleshoot this issue would be appreciated. Commands, dummy guides, sarcasm all welcome.
>
> Thanks much
>
> Regards,
>
> Steven.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.