Mailing List Archive

1 2  View All
Re: Basic Rsyslog Troubleshooting [ In reply to ]
On Mon, 25 Apr 2022, Mariusz Kruk via rsyslog wrote:

> Sure. I work with them, I know ;-)
>
> It's just that for some, you can do the same but using rsyslog to process the
> message (even filter some events out or trim them or do many other fancy
> stuff) an send them directly to SIEM (by means of native SIEM API, not by
> syslog)
>
> instead of killing the server with IOPS. That's all.

fair point, but I'll point out that since rsyslog is buffering the writes to
disk, the IOPS load is not as high as you would think. It's sequential writes
and reads, and in practice should be just sequential writes as the reads should
be able to be served from ram (the files should be read shortly after they are
written, so should still be cached)

It would be interesting to look into the performance (CPU and I/O) of writing to
disk and having the files read in batches vs posting each message to the SIEM. I
think there are enough variables involved that it's not an obvious win either
way.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Thx David for the excellent thread!

Just a side-note, nothing more to add.

Dynafile performance depends on the number of *really* active files. I
have had quite some installations where the cache size was much lower
than active files in an overall look. But in these cases many of the
"active" files were written to only very seldomly. So evicting a file
and re-opening didn't really hurt. But it saved quite a lot of
resources. I think the bottom line to see is when the relative number
of cache evictions is high. "A couple" of them does not really hurt.

Just wanted to make sure this use-case does not get unmentioned.

Rainer

El dom, 24 abr 2022 a las 16:30, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> On Sun, 24 Apr 2022, Steven D wrote:
>
> > No sir, no encryption is in play. Just plain ol TCP syslog. We're running rsyslog v8.1911
> >
> > I don't think we've enforced any additional dynafile settings, but i'll double check there's not an override I don't know about. Do youz have a rough order guidance for dynafilecachesize settings?
>
> larger than the number of files that will be open at any time. If this is
> smaller than the number of files being actively written to, every time you get a
> log for a new file, rsyslog has to first close (and flush writes to) an old
> file. performance utterly collapses.
>
> > After getting thoroughly lost in the Google rabbit hole, I had other questions;
> >
> > Would using imptcp over imtcp help?
> >
> > * I saw mentions that imptcp handles connections better/performs better.
>
> should not be a significant difference for the volume you describe, but won't
> hurt to try
>
> > Would setting the KeepAlives in the rsyslog config on the server-side help to manage the (zombie?) TCP connections.?
> >
> > * The load balancer being in the middle feels like it's the cause of repeated ESTABLISHED connections, but to keep HA/redundancy it's kind of a necessary evil.
>
> yes, that should help avoid the connection being idle long enough for the load
> balancer to break the connection (and each break probably causes a log loss), I
> would also enable rebind interval. I like to set it to reconnect every few
> seconds under high log volume to give the load balancers the best chance to work
>
> you can use a different method for failover. I like using pacemaker/corosync to
> move an IP between the two systems (it has the added advantage that you can use
> the CLUSTERIP feature to do rough load balancing between systems without an
> external loadbalancer)
>
> David Lang
>
> > For reference, our main input config looks like this. There are a number of other input entries, but they're all variations on this for different log source types (servers, apps, etc).
> >
> > module(load="imudp")
> > module(load="imtcp" MaxListeners="100" AddtlFrameDelimiter="000")
> >
> > input(type="imudp" port="10514" ruleset="firewall_rule")
> > input(type="imtcp" port="10514" ruleset="firewall_rule")
> > template(name="firewall_logs" type="string" string="/data/logs/firewall/10514/%fromhost-ip%/syslog.log")
> > ruleset(name="firewall_rule") {
> > action(type="omfile"
> > FileCreateMode="0744"
> > DirCreateMode="0755"
> > FileOwner="loguser"
> > FileGroup="loguser"
> > DirOwner="loguser"
> > DirGroup="loguser"
> > DynaFile="firewall_logs")
> > }
> > .
> > .
> > .
> > [snip]
> >
> > Thank again, really appreciate the insight.
> > ________________________________
> > From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
> > Sent: Sunday, April 24, 2022 8:57 AM
> > To: David Lang <david@lang.hm>; Steven D via rsyslog <rsyslog@lists.adiscon.com>
> > Cc: Steven D <pheerless@hotmail.com>
> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
> >
> > David
> >
> > Thanks for the depth of this reply. Let me feed back in some additional info.
> >
> > We've removed the load balancer from the syslog path as part of troubleshooting and the behavior didn't change. We continued to see log loss using TCP(most notably for our firewalls), even when directing to a single receiver server.
> >
> > So far as the network goes there is a clean path from sending log source (firewall) to the rsyslog receiver. So I guess I'm looking for guidance on what knobs I should look at turning on the RHEL/rsyslog side.
> >
> > As far as RELP, will that receive standard inbound TCP syslog on the receiver server? I was under the impression (possibly mistaken) both sender/receiver needed to use RELP. I'm happy to test using that module versus imtcp if I'm wrong.
> >
> > Thanks again.
> >
> > Regards,
> >
> > Steven.
> >
> >
> >
> > -------- Original message --------
> > From: David Lang <david@lang.hm>
> > Date: 4/24/22 8:27 AM (GMT-05:00)
> > To: Steven D via rsyslog <rsyslog@lists.adiscon.com>
> > Cc: Steven D <pheerless@hotmail.com>
> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
> >
> > One problem with TCP load balancing of syslog messages is that the load
> > balancers do not understand the syslog protocol, so they can't rebalance at a
> > message boundry.
> >
> > A second problem is that when a firewall or load balancer drops a connection,
> > the sender doesn't know that it's dropped until the next time it tries to
> > deliver a message. Since TCP doesn't have any way for the OS TCP stack to tell
> > software "that message that you submitted to an open connection, and I accepted,
> > it can't be delivered" (once the OS accepts the message, the sender has to
> > assume that it will be delivered)
> >
> > As a result, it's very easy for TCP syslog to be less reliable than UDP syslog.
> > The 'common sense wisdom' is that TCP is reliable because dropped packets inside
> > an ongoing connection will get retried, but dropped packets are actually very
> > uncommon inside a datacenter. They may happen when a firewall/router is
> > overloaded, but it's not very common. Back in 2006 or so I did testing and found
> > that within a local network, UDP was almost perfectly reliable (as long as the
> > receiver could keep up and not overflow the OS buffers)
> >
> > Rsyslog has the rebindinterval feature, which tells the sender to disconnect and
> > reconnect periodically so that the load balancer has a chance to make a new
> > balancing decision.
> >
> > you also want to make sure that the log stram is not idle for too long ('mark'
> > was the historical method of doing that, I prefer vmstat 60 |logger -t vmstat as
> > it's not much larger and an extremely dense set of information that can be very
> > useful when troubleshooting)
> >
> > The other thing to look at is the RELP protocol, it was developed specifically
> > because TCP was designed to be reliable over an unreliable wire, but assumes
> > that both ends will remain up and the connection will not be cut by a middlebox.
> > RELP does full application level acks so that the sender knows that the receiver
> > rsyslog actually processed the message
> >
> > with plain TCP, once the sending software submits data to the OS stack and the
> > OS stack says it's accepted the data, the data then sits in a buffer on the
> > sending machine, then gets sent over the wire (with retries), then sits in a
> > buffer on the receiving machine until the receiving software reads it. If
> > anything causes the connection to be terminated (firewall, load balancer, crash
> > on the receiving machine, etc) the data will be lost and the sending software
> > has no way of learning about it.
> >
> > David Lang
> >
> >
> > On Sun, 24 Apr 2022, Steven D via rsyslog wrote:
> >
> >> Date: Sun, 24 Apr 2022 12:14:35 +0000
> >> From: Steven D via rsyslog <rsyslog@lists.adiscon.com>
> >> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> >> Cc: Steven D <pheerless@hotmail.com>
> >> Subject: [rsyslog] Basic Rsyslog Troubleshooting
> >>
> >> Greetings list
> >>
> >> New to rsyslog list, not new to logging. We're experiencing an odd issue where TCP syslog messages are being dropped at seemingly random intervals...hoping to get some input.
> >>
> >> The TLDR on our architecture is we have set up a couple rsyslog receivers behind a Netscaler Load balancer. Multiple platforms/devices are configured to send syslog to the load balancer, which distributes to the receivers. Receivers are running RHEL v8 and rsyslog v8.1911. Receivers write files to disk, which we then read with a SIEM agent.
> >>
> >> We've got a modestly sized environment with a syslog client base of 200-300 servers, 30 networking devices (including firewalls) and some applications all directing logging to the load balancer.
> >>
> >> Our config file is pretty vanilla, no cache, or advanced tweaks. Just using the "imtcp" and "imudp" modules and rulesets to write files to disk based on the sending host IP/port.
> >>
> >> The first problem we're seeing is that hosts sending via TCP have log messages missed (never written to disk), where UDP seems more reliable. When switching the firewalls to UDP, throughput nearly doubles and message loss is less noticeable (yeah I know it's still UDP).
> >>
> >> Possibly related is that we've noticed that each receiver also holds a lot of "Established" connections for back to the clients, but different ports. (Possible session/connection exhaustion?)
> >>
> >> Any guidance on how we can approach and troubleshoot this issue would be appreciated. Commands, dummy guides, sarcasm all welcome.
> >>
> >> Thanks much
> >>
> >> Regards,
> >>
> >> Steven.
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
On 25.04.2022 14:41, David Lang wrote:
> On Mon, 25 Apr 2022, Mariusz Kruk via rsyslog wrote:
>
>> Sure. I work with them, I know ;-)
>>
>> It's just that for some, you can do the same but using rsyslog to
>> process the message (even filter some events out or trim them or do
>> many other fancy stuff) an send them directly to SIEM (by means of
>> native SIEM API, not by syslog)
>>
>> instead of killing the server with IOPS. That's all.
>
> fair point, but I'll point out that since rsyslog is buffering the
> writes to disk, the IOPS load is not as high as you would think. It's
> sequential writes and reads, and in practice should be just sequential
> writes as the reads should be able to be served from ram (the files
> should be read shortly after they are written, so should still be cached)
>
> It would be interesting to look into the performance (CPU and I/O) of
> writing to disk and having the files read in batches vs posting each
> message to the SIEM. I think there are enough variables involved that
> it's not an obvious win either way.
>
Sure, there is also another layer of OS-level caching/buffering. But in
the end it all has to get written to the disk. ;-) I just think that
passing the events straight through rulesets from input to output (and I
mean the real destination, not to the files) is simply more
straightforward and easier to manage (after all you don't have to - for
example - deal with file rotation and so on). On the other hand, you may
run into some other problems when your rulesets get too complicated and
"run away".


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
On Mon, 25 Apr 2022, Mariusz Kruk wrote:

> On 25.04.2022 14:41, David Lang wrote:
>> On Mon, 25 Apr 2022, Mariusz Kruk via rsyslog wrote:
>>
>>> Sure. I work with them, I know ;-)
>>>
>>> It's just that for some, you can do the same but using rsyslog to process
>>> the message (even filter some events out or trim them or do many other
>>> fancy stuff) an send them directly to SIEM (by means of native SIEM API,
>>> not by syslog)
>>>
>>> instead of killing the server with IOPS. That's all.
>>
>> fair point, but I'll point out that since rsyslog is buffering the writes
>> to disk, the IOPS load is not as high as you would think. It's sequential
>> writes and reads, and in practice should be just sequential writes as the
>> reads should be able to be served from ram (the files should be read
>> shortly after they are written, so should still be cached)
>>
>> It would be interesting to look into the performance (CPU and I/O) of
>> writing to disk and having the files read in batches vs posting each
>> message to the SIEM. I think there are enough variables involved that it's
>> not an obvious win either way.
>>
> Sure, there is also another layer of OS-level caching/buffering. But in the
> end it all has to get written to the disk. ;-) I just think that passing the
> events straight through rulesets from input to output (and I mean the real
> destination, not to the files) is simply more straightforward and easier to
> manage (after all you don't have to - for example - deal with file rotation
> and so on). On the other hand, you may run into some other problems when your
> rulesets get too complicated and "run away".

buffered, sequential writes to disk are pretty cheap and perform well even on
slow spinning rust, and file rotation (should) happen during off-peak periods,
and as previously mentioned gives you a handy backup for those cases where
something goes wrong with your SIEM. Yes, you could have disk backed queues in
rsyslog, but the performance of such queues is horrid and significantly more I/O
load than writing to disk.

Not saying that sending directly is wrong, just listing things I've run into
that make me default the other way.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
On 25.04.2022 15:04, David Lang wrote:
>>>> Sure. I work with them, I know ;-)
>>>>
>>>> It's just that for some, you can do the same but using rsyslog to
>>>> process the message (even filter some events out or trim them or do
>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>> native SIEM API, not by syslog)
>>>>
>>>> instead of killing the server with IOPS. That's all.
>>>
>>> fair point, but I'll point out that since rsyslog is buffering the
>>> writes to disk, the IOPS load is not as high as you would think.
>>> It's sequential writes and reads, and in practice should be just
>>> sequential writes as the reads should be able to be served from ram
>>> (the files should be read shortly after they are written, so should
>>> still be cached)
>>>
>>> It would be interesting to look into the performance (CPU and I/O)
>>> of writing to disk and having the files read in batches vs posting
>>> each message to the SIEM. I think there are enough variables
>>> involved that it's not an obvious win either way.
>>>
>> Sure, there is also another layer of OS-level caching/buffering. But
>> in the end it all has to get written to the disk. ;-) I just think
>> that passing the events straight through rulesets from input to
>> output (and I mean the real destination, not to the files) is simply
>> more straightforward and easier to manage (after all you don't have
>> to - for example - deal with file rotation and so on). On the other
>> hand, you may run into some other problems when your rulesets get too
>> complicated and "run away".
>
> buffered, sequential writes to disk are pretty cheap and perform well
> even on slow spinning rust, and file rotation (should) happen during
> off-peak periods, and as previously mentioned gives you a handy backup
> for those cases where something goes wrong with your SIEM. Yes, you
> could have disk backed queues in rsyslog, but the performance of such
> queues is horrid and significantly more I/O load than writing to disk.
>
> Not saying that sending directly is wrong, just listing things I've
> run into that make me default the other way.
>
Yep. I wasn't talking about file rotation in terms of performance (since
it should be relatively cheap anyway, especially if you don't want to
compress old files). I was referring more to the manageability of the
whole solution.

Adding another ruleset seems easier than making sure that you have a
proper directory prepared for the files, possibly reconfiguring your
SIEM agent, making sure the files are included in the rotation scheme...
Seems like a headache.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.

Glad there are other SIEM jockeys on here... haha.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
Sent: Monday, April 25, 2022 9:08 AM
To: David Lang <david@lang.hm>
Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting


On 25.04.2022 15:04, David Lang wrote:
>>>> Sure. I work with them, I know ;-)
>>>>
>>>> It's just that for some, you can do the same but using rsyslog to
>>>> process the message (even filter some events out or trim them or do
>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>> native SIEM API, not by syslog)
>>>>
>>>> instead of killing the server with IOPS. That's all.
>>>
>>> fair point, but I'll point out that since rsyslog is buffering the
>>> writes to disk, the IOPS load is not as high as you would think.
>>> It's sequential writes and reads, and in practice should be just
>>> sequential writes as the reads should be able to be served from ram
>>> (the files should be read shortly after they are written, so should
>>> still be cached)
>>>
>>> It would be interesting to look into the performance (CPU and I/O)
>>> of writing to disk and having the files read in batches vs posting
>>> each message to the SIEM. I think there are enough variables
>>> involved that it's not an obvious win either way.
>>>
>> Sure, there is also another layer of OS-level caching/buffering. But
>> in the end it all has to get written to the disk. ;-) I just think
>> that passing the events straight through rulesets from input to
>> output (and I mean the real destination, not to the files) is simply
>> more straightforward and easier to manage (after all you don't have
>> to - for example - deal with file rotation and so on). On the other
>> hand, you may run into some other problems when your rulesets get too
>> complicated and "run away".
>
> buffered, sequential writes to disk are pretty cheap and perform well
> even on slow spinning rust, and file rotation (should) happen during
> off-peak periods, and as previously mentioned gives you a handy backup
> for those cases where something goes wrong with your SIEM. Yes, you
> could have disk backed queues in rsyslog, but the performance of such
> queues is horrid and significantly more I/O load than writing to disk.
>
> Not saying that sending directly is wrong, just listing things I've
> run into that make me default the other way.
>
Yep. I wasn't talking about file rotation in terms of performance (since
it should be relatively cheap anyway, especially if you don't want to
compress old files). I was referring more to the manageability of the
whole solution.

Adding another ruleset seems easier than making sure that you have a
proper directory prepared for the files, possibly reconfiguring your
SIEM agent, making sure the files are included in the rotation scheme...
Seems like a headache.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Here's most recent few rotations of pstats data, any additional input would be appreciated.

With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?

Regards,
Steven
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
Sent: Monday, April 25, 2022 9:52 AM
To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerless@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.

Glad there are other SIEM jockeys on here... haha.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
Sent: Monday, April 25, 2022 9:08 AM
To: David Lang <david@lang.hm>
Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting


On 25.04.2022 15:04, David Lang wrote:
>>>> Sure. I work with them, I know ;-)
>>>>
>>>> It's just that for some, you can do the same but using rsyslog to
>>>> process the message (even filter some events out or trim them or do
>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>> native SIEM API, not by syslog)
>>>>
>>>> instead of killing the server with IOPS. That's all.
>>>
>>> fair point, but I'll point out that since rsyslog is buffering the
>>> writes to disk, the IOPS load is not as high as you would think.
>>> It's sequential writes and reads, and in practice should be just
>>> sequential writes as the reads should be able to be served from ram
>>> (the files should be read shortly after they are written, so should
>>> still be cached)
>>>
>>> It would be interesting to look into the performance (CPU and I/O)
>>> of writing to disk and having the files read in batches vs posting
>>> each message to the SIEM. I think there are enough variables
>>> involved that it's not an obvious win either way.
>>>
>> Sure, there is also another layer of OS-level caching/buffering. But
>> in the end it all has to get written to the disk. ;-) I just think
>> that passing the events straight through rulesets from input to
>> output (and I mean the real destination, not to the files) is simply
>> more straightforward and easier to manage (after all you don't have
>> to - for example - deal with file rotation and so on). On the other
>> hand, you may run into some other problems when your rulesets get too
>> complicated and "run away".
>
> buffered, sequential writes to disk are pretty cheap and perform well
> even on slow spinning rust, and file rotation (should) happen during
> off-peak periods, and as previously mentioned gives you a handy backup
> for those cases where something goes wrong with your SIEM. Yes, you
> could have disk backed queues in rsyslog, but the performance of such
> queues is horrid and significantly more I/O load than writing to disk.
>
> Not saying that sending directly is wrong, just listing things I've
> run into that make me default the other way.
>
Yep. I wasn't talking about file rotation in terms of performance (since
it should be relatively cheap anyway, especially if you don't want to
compress old files). I was referring more to the manageability of the
whole solution.

Adding another ruleset seems easier than making sure that you have a
proper directory prepared for the files, possibly reconfiguring your
SIEM agent, making sure the files are included in the rotation scheme...
Seems like a headache.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
hmm, looks like it didn't make it through the mail server

https://cromwell-intl.com/open-source/performance-tuning/tcp.html

(also note there are similar UDP buffer tuning parameters, if those buffers fill
up udp logs will be dropped silently)

David Lang

On Tue, 26 Apr 2022, Steven D wrote:

> Date: Tue, 26 Apr 2022 11:00:16 +0000
> From: Steven D <pheerless@hotmail.com>
> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>
> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>
> Regards,
> Steven
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
> Sent: Monday, April 25, 2022 9:52 AM
> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Steven D <pheerless@hotmail.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>
> Glad there are other SIEM jockeys on here... haha.
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Sent: Monday, April 25, 2022 9:08 AM
> To: David Lang <david@lang.hm>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
>
> On 25.04.2022 15:04, David Lang wrote:
>>>>> Sure. I work with them, I know ;-)
>>>>>
>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>> process the message (even filter some events out or trim them or do
>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>> native SIEM API, not by syslog)
>>>>>
>>>>> instead of killing the server with IOPS. That's all.
>>>>
>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>> writes to disk, the IOPS load is not as high as you would think.
>>>> It's sequential writes and reads, and in practice should be just
>>>> sequential writes as the reads should be able to be served from ram
>>>> (the files should be read shortly after they are written, so should
>>>> still be cached)
>>>>
>>>> It would be interesting to look into the performance (CPU and I/O)
>>>> of writing to disk and having the files read in batches vs posting
>>>> each message to the SIEM. I think there are enough variables
>>>> involved that it's not an obvious win either way.
>>>>
>>> Sure, there is also another layer of OS-level caching/buffering. But
>>> in the end it all has to get written to the disk. ;-) I just think
>>> that passing the events straight through rulesets from input to
>>> output (and I mean the real destination, not to the files) is simply
>>> more straightforward and easier to manage (after all you don't have
>>> to - for example - deal with file rotation and so on). On the other
>>> hand, you may run into some other problems when your rulesets get too
>>> complicated and "run away".
>>
>> buffered, sequential writes to disk are pretty cheap and perform well
>> even on slow spinning rust, and file rotation (should) happen during
>> off-peak periods, and as previously mentioned gives you a handy backup
>> for those cases where something goes wrong with your SIEM. Yes, you
>> could have disk backed queues in rsyslog, but the performance of such
>> queues is horrid and significantly more I/O load than writing to disk.
>>
>> Not saying that sending directly is wrong, just listing things I've
>> run into that make me default the other way.
>>
> Yep. I wasn't talking about file rotation in terms of performance (since
> it should be relatively cheap anyway, especially if you don't want to
> compress old files). I was referring more to the manageability of the
> whole solution.
>
> Adding another ruleset seems easier than making sure that you have a
> proper directory prepared for the files, possibly reconfiguring your
> SIEM agent, making sure the files are included in the rotation scheme...
> Seems like a headache.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
________________________________
From: David Lang <david@lang.hm>
Sent: Tuesday, April 26, 2022 9:57 AM
To: Steven D <pheerless@hotmail.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

hmm, looks like it didn't make it through the mail server

https://cromwell-intl.com/open-source/performance-tuning/tcp.html

(also note there are similar UDP buffer tuning parameters, if those buffers fill
up udp logs will be dropped silently)

David Lang

On Tue, 26 Apr 2022, Steven D wrote:

> Date: Tue, 26 Apr 2022 11:00:16 +0000
> From: Steven D <pheerless@hotmail.com>
> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>
> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>
> Regards,
> Steven
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
> Sent: Monday, April 25, 2022 9:52 AM
> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Steven D <pheerless@hotmail.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>
> Glad there are other SIEM jockeys on here... haha.
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Sent: Monday, April 25, 2022 9:08 AM
> To: David Lang <david@lang.hm>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
>
> On 25.04.2022 15:04, David Lang wrote:
>>>>> Sure. I work with them, I know ;-)
>>>>>
>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>> process the message (even filter some events out or trim them or do
>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>> native SIEM API, not by syslog)
>>>>>
>>>>> instead of killing the server with IOPS. That's all.
>>>>
>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>> writes to disk, the IOPS load is not as high as you would think.
>>>> It's sequential writes and reads, and in practice should be just
>>>> sequential writes as the reads should be able to be served from ram
>>>> (the files should be read shortly after they are written, so should
>>>> still be cached)
>>>>
>>>> It would be interesting to look into the performance (CPU and I/O)
>>>> of writing to disk and having the files read in batches vs posting
>>>> each message to the SIEM. I think there are enough variables
>>>> involved that it's not an obvious win either way.
>>>>
>>> Sure, there is also another layer of OS-level caching/buffering. But
>>> in the end it all has to get written to the disk. ;-) I just think
>>> that passing the events straight through rulesets from input to
>>> output (and I mean the real destination, not to the files) is simply
>>> more straightforward and easier to manage (after all you don't have
>>> to - for example - deal with file rotation and so on). On the other
>>> hand, you may run into some other problems when your rulesets get too
>>> complicated and "run away".
>>
>> buffered, sequential writes to disk are pretty cheap and perform well
>> even on slow spinning rust, and file rotation (should) happen during
>> off-peak periods, and as previously mentioned gives you a handy backup
>> for those cases where something goes wrong with your SIEM. Yes, you
>> could have disk backed queues in rsyslog, but the performance of such
>> queues is horrid and significantly more I/O load than writing to disk.
>>
>> Not saying that sending directly is wrong, just listing things I've
>> run into that make me default the other way.
>>
> Yep. I wasn't talking about file rotation in terms of performance (since
> it should be relatively cheap anyway, especially if you don't want to
> compress old files). I was referring more to the manageability of the
> whole solution.
>
> Adding another ruleset seems easier than making sure that you have a
> proper directory prepared for the files, possibly reconfiguring your
> SIEM agent, making sure the files are included in the rotation scheme...
> Seems like a headache.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
Re: Basic Rsyslog Troubleshooting [ In reply to ]
that looks pretty good to me, if it's still running slow, I would look at the
threads in top to see what their utilization is, and try the imptcp vs imtcp and
see if there's a difference

the way you are specifying a ruleset in an input means that you don't have lots
of rule processing happening.

one thing you could have is contention for the lock on the main queue. By
default, all messages go to the main queue. If you are tieing an input to a
ruleset, you can give that ruleset a queue and the messages will never touch the
main queue (in effect, it's like you were running a separate rsyslog instance
for that input and ruleset)

David Lang


On Tue, 26 Apr 2022, Steven D wrote:

> Date: Tue, 26 Apr 2022 14:12:36 +0000
> From: Steven D <pheerless@hotmail.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Tuesday, April 26, 2022 9:57 AM
> To: Steven D <pheerless@hotmail.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> hmm, looks like it didn't make it through the mail server
>
> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>
> (also note there are similar UDP buffer tuning parameters, if those buffers fill
> up udp logs will be dropped silently)
>
> David Lang
>
> On Tue, 26 Apr 2022, Steven D wrote:
>
>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>> From: Steven D <pheerless@hotmail.com>
>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>
>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>
>> Regards,
>> Steven
>> ________________________________
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Monday, April 25, 2022 9:52 AM
>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Steven D <pheerless@hotmail.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>
>> Glad there are other SIEM jockeys on here... haha.
>> ________________________________
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Monday, April 25, 2022 9:08 AM
>> To: David Lang <david@lang.hm>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>>
>> On 25.04.2022 15:04, David Lang wrote:
>>>>>> Sure. I work with them, I know ;-)
>>>>>>
>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>> process the message (even filter some events out or trim them or do
>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>> native SIEM API, not by syslog)
>>>>>>
>>>>>> instead of killing the server with IOPS. That's all.
>>>>>
>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>> It's sequential writes and reads, and in practice should be just
>>>>> sequential writes as the reads should be able to be served from ram
>>>>> (the files should be read shortly after they are written, so should
>>>>> still be cached)
>>>>>
>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>> of writing to disk and having the files read in batches vs posting
>>>>> each message to the SIEM. I think there are enough variables
>>>>> involved that it's not an obvious win either way.
>>>>>
>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>> in the end it all has to get written to the disk. ;-) I just think
>>>> that passing the events straight through rulesets from input to
>>>> output (and I mean the real destination, not to the files) is simply
>>>> more straightforward and easier to manage (after all you don't have
>>>> to - for example - deal with file rotation and so on). On the other
>>>> hand, you may run into some other problems when your rulesets get too
>>>> complicated and "run away".
>>>
>>> buffered, sequential writes to disk are pretty cheap and perform well
>>> even on slow spinning rust, and file rotation (should) happen during
>>> off-peak periods, and as previously mentioned gives you a handy backup
>>> for those cases where something goes wrong with your SIEM. Yes, you
>>> could have disk backed queues in rsyslog, but the performance of such
>>> queues is horrid and significantly more I/O load than writing to disk.
>>>
>>> Not saying that sending directly is wrong, just listing things I've
>>> run into that make me default the other way.
>>>
>> Yep. I wasn't talking about file rotation in terms of performance (since
>> it should be relatively cheap anyway, especially if you don't want to
>> compress old files). I was referring more to the manageability of the
>> whole solution.
>>
>> Adding another ruleset seems easier than making sure that you have a
>> proper directory prepared for the files, possibly reconfiguring your
>> SIEM agent, making sure the files are included in the rotation scheme...
>> Seems like a headache.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????

I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
________________________________
From: David Lang <david@lang.hm>
Sent: Tuesday, April 26, 2022 10:23 AM
To: Steven D <pheerless@hotmail.com>
Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

that looks pretty good to me, if it's still running slow, I would look at the
threads in top to see what their utilization is, and try the imptcp vs imtcp and
see if there's a difference

the way you are specifying a ruleset in an input means that you don't have lots
of rule processing happening.

one thing you could have is contention for the lock on the main queue. By
default, all messages go to the main queue. If you are tieing an input to a
ruleset, you can give that ruleset a queue and the messages will never touch the
main queue (in effect, it's like you were running a separate rsyslog instance
for that input and ruleset)

David Lang


On Tue, 26 Apr 2022, Steven D wrote:

> Date: Tue, 26 Apr 2022 14:12:36 +0000
> From: Steven D <pheerless@hotmail.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Tuesday, April 26, 2022 9:57 AM
> To: Steven D <pheerless@hotmail.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> hmm, looks like it didn't make it through the mail server
>
> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>
> (also note there are similar UDP buffer tuning parameters, if those buffers fill
> up udp logs will be dropped silently)
>
> David Lang
>
> On Tue, 26 Apr 2022, Steven D wrote:
>
>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>> From: Steven D <pheerless@hotmail.com>
>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>
>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>
>> Regards,
>> Steven
>> ________________________________
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Monday, April 25, 2022 9:52 AM
>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Steven D <pheerless@hotmail.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>
>> Glad there are other SIEM jockeys on here... haha.
>> ________________________________
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Monday, April 25, 2022 9:08 AM
>> To: David Lang <david@lang.hm>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>>
>> On 25.04.2022 15:04, David Lang wrote:
>>>>>> Sure. I work with them, I know ;-)
>>>>>>
>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>> process the message (even filter some events out or trim them or do
>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>> native SIEM API, not by syslog)
>>>>>>
>>>>>> instead of killing the server with IOPS. That's all.
>>>>>
>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>> It's sequential writes and reads, and in practice should be just
>>>>> sequential writes as the reads should be able to be served from ram
>>>>> (the files should be read shortly after they are written, so should
>>>>> still be cached)
>>>>>
>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>> of writing to disk and having the files read in batches vs posting
>>>>> each message to the SIEM. I think there are enough variables
>>>>> involved that it's not an obvious win either way.
>>>>>
>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>> in the end it all has to get written to the disk. ;-) I just think
>>>> that passing the events straight through rulesets from input to
>>>> output (and I mean the real destination, not to the files) is simply
>>>> more straightforward and easier to manage (after all you don't have
>>>> to - for example - deal with file rotation and so on). On the other
>>>> hand, you may run into some other problems when your rulesets get too
>>>> complicated and "run away".
>>>
>>> buffered, sequential writes to disk are pretty cheap and perform well
>>> even on slow spinning rust, and file rotation (should) happen during
>>> off-peak periods, and as previously mentioned gives you a handy backup
>>> for those cases where something goes wrong with your SIEM. Yes, you
>>> could have disk backed queues in rsyslog, but the performance of such
>>> queues is horrid and significantly more I/O load than writing to disk.
>>>
>>> Not saying that sending directly is wrong, just listing things I've
>>> run into that make me default the other way.
>>>
>> Yep. I wasn't talking about file rotation in terms of performance (since
>> it should be relatively cheap anyway, especially if you don't want to
>> compress old files). I was referring more to the manageability of the
>> whole solution.
>>
>> Adding another ruleset seems easier than making sure that you have a
>> proper directory prepared for the files, possibly reconfiguring your
>> SIEM agent, making sure the files are included in the rotation scheme...
>> Seems like a headache.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
I second David's thought to switch to imptcp. It is designed for plain TCP performance.

I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…”

Thanks Gents!


> On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????
>
> I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Tuesday, April 26, 2022 10:23 AM
> To: Steven D <pheerless@hotmail.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> that looks pretty good to me, if it's still running slow, I would look at the
> threads in top to see what their utilization is, and try the imptcp vs imtcp and
> see if there's a difference
>
> the way you are specifying a ruleset in an input means that you don't have lots
> of rule processing happening.
>
> one thing you could have is contention for the lock on the main queue. By
> default, all messages go to the main queue. If you are tieing an input to a
> ruleset, you can give that ruleset a queue and the messages will never touch the
> main queue (in effect, it's like you were running a separate rsyslog instance
> for that input and ruleset)
>
> David Lang
>
>
> On Tue, 26 Apr 2022, Steven D wrote:
>
>> Date: Tue, 26 Apr 2022 14:12:36 +0000
>> From: Steven D <pheerless@hotmail.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, April 26, 2022 9:57 AM
>> To: Steven D <pheerless@hotmail.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> hmm, looks like it didn't make it through the mail server
>>
>> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>>
>> (also note there are similar UDP buffer tuning parameters, if those buffers fill
>> up udp logs will be dropped silently)
>>
>> David Lang
>>
>> On Tue, 26 Apr 2022, Steven D wrote:
>>
>>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>>> From: Steven D <pheerless@hotmail.com>
>>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>>
>>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>>
>>> Regards,
>>> Steven
>>> ________________________________
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>>> Sent: Monday, April 25, 2022 9:52 AM
>>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: Steven D <pheerless@hotmail.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>>
>>> Glad there are other SIEM jockeys on here... haha.
>>> ________________________________
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>> Sent: Monday, April 25, 2022 9:08 AM
>>> To: David Lang <david@lang.hm>
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>>
>>> On 25.04.2022 15:04, David Lang wrote:
>>>>>>> Sure. I work with them, I know ;-)
>>>>>>>
>>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>>> process the message (even filter some events out or trim them or do
>>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>>> native SIEM API, not by syslog)
>>>>>>>
>>>>>>> instead of killing the server with IOPS. That's all.
>>>>>>
>>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>>> It's sequential writes and reads, and in practice should be just
>>>>>> sequential writes as the reads should be able to be served from ram
>>>>>> (the files should be read shortly after they are written, so should
>>>>>> still be cached)
>>>>>>
>>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>>> of writing to disk and having the files read in batches vs posting
>>>>>> each message to the SIEM. I think there are enough variables
>>>>>> involved that it's not an obvious win either way.
>>>>>>
>>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>>> in the end it all has to get written to the disk. ;-) I just think
>>>>> that passing the events straight through rulesets from input to
>>>>> output (and I mean the real destination, not to the files) is simply
>>>>> more straightforward and easier to manage (after all you don't have
>>>>> to - for example - deal with file rotation and so on). On the other
>>>>> hand, you may run into some other problems when your rulesets get too
>>>>> complicated and "run away".
>>>>
>>>> buffered, sequential writes to disk are pretty cheap and perform well
>>>> even on slow spinning rust, and file rotation (should) happen during
>>>> off-peak periods, and as previously mentioned gives you a handy backup
>>>> for those cases where something goes wrong with your SIEM. Yes, you
>>>> could have disk backed queues in rsyslog, but the performance of such
>>>> queues is horrid and significantly more I/O load than writing to disk.
>>>>
>>>> Not saying that sending directly is wrong, just listing things I've
>>>> run into that make me default the other way.
>>>>
>>> Yep. I wasn't talking about file rotation in terms of performance (since
>>> it should be relatively cheap anyway, especially if you don't want to
>>> compress old files). I was referring more to the manageability of the
>>> whole solution.
>>>
>>> Adding another ruleset seems easier than making sure that you have a
>>> proper directory prepared for the files, possibly reconfiguring your
>>> SIEM agent, making sure the files are included in the rotation scheme...
>>> Seems like a headache.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
Wanted to circle back around to this topic for a closure update.

We were able to diagnose that one particular data source (a specific firewall vendor) appears to have trouble consistently sending TCP ... So we switched it to UDP and problem is gone. Ultimately no issues with rsyslog or the OS were at fault.

The rest of the diagnostic help from here helped us locate a couple other issues we wouldnt have noticed otherwise.

Thanks a lot all!



Regards,

Steven.



-------- Original message --------
From: John Chivian <jchivian@chivian.com>
Date: 4/26/22 12:58 PM (GMT-05:00)
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>, Steven D <pheerless@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting

I second David's thought to switch to imptcp. It is designed for plain TCP performance.

I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…”

Thanks Gents!


> On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????
>
> I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Tuesday, April 26, 2022 10:23 AM
> To: Steven D <pheerless@hotmail.com>
> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> that looks pretty good to me, if it's still running slow, I would look at the
> threads in top to see what their utilization is, and try the imptcp vs imtcp and
> see if there's a difference
>
> the way you are specifying a ruleset in an input means that you don't have lots
> of rule processing happening.
>
> one thing you could have is contention for the lock on the main queue. By
> default, all messages go to the main queue. If you are tieing an input to a
> ruleset, you can give that ruleset a queue and the messages will never touch the
> main queue (in effect, it's like you were running a separate rsyslog instance
> for that input and ruleset)
>
> David Lang
>
>
> On Tue, 26 Apr 2022, Steven D wrote:
>
>> Date: Tue, 26 Apr 2022 14:12:36 +0000
>> From: Steven D <pheerless@hotmail.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, April 26, 2022 9:57 AM
>> To: Steven D <pheerless@hotmail.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> hmm, looks like it didn't make it through the mail server
>>
>> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>>
>> (also note there are similar UDP buffer tuning parameters, if those buffers fill
>> up udp logs will be dropped silently)
>>
>> David Lang
>>
>> On Tue, 26 Apr 2022, Steven D wrote:
>>
>>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>>> From: Steven D <pheerless@hotmail.com>
>>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>>
>>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>>
>>> Regards,
>>> Steven
>>> ________________________________
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>>> Sent: Monday, April 25, 2022 9:52 AM
>>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: Steven D <pheerless@hotmail.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>>
>>> Glad there are other SIEM jockeys on here... haha.
>>> ________________________________
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>> Sent: Monday, April 25, 2022 9:08 AM
>>> To: David Lang <david@lang.hm>
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>>
>>> On 25.04.2022 15:04, David Lang wrote:
>>>>>>> Sure. I work with them, I know ;-)
>>>>>>>
>>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>>> process the message (even filter some events out or trim them or do
>>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>>> native SIEM API, not by syslog)
>>>>>>>
>>>>>>> instead of killing the server with IOPS. That's all.
>>>>>>
>>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>>> It's sequential writes and reads, and in practice should be just
>>>>>> sequential writes as the reads should be able to be served from ram
>>>>>> (the files should be read shortly after they are written, so should
>>>>>> still be cached)
>>>>>>
>>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>>> of writing to disk and having the files read in batches vs posting
>>>>>> each message to the SIEM. I think there are enough variables
>>>>>> involved that it's not an obvious win either way.
>>>>>>
>>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>>> in the end it all has to get written to the disk. ;-) I just think
>>>>> that passing the events straight through rulesets from input to
>>>>> output (and I mean the real destination, not to the files) is simply
>>>>> more straightforward and easier to manage (after all you don't have
>>>>> to - for example - deal with file rotation and so on). On the other
>>>>> hand, you may run into some other problems when your rulesets get too
>>>>> complicated and "run away".
>>>>
>>>> buffered, sequential writes to disk are pretty cheap and perform well
>>>> even on slow spinning rust, and file rotation (should) happen during
>>>> off-peak periods, and as previously mentioned gives you a handy backup
>>>> for those cases where something goes wrong with your SIEM. Yes, you
>>>> could have disk backed queues in rsyslog, but the performance of such
>>>> queues is horrid and significantly more I/O load than writing to disk.
>>>>
>>>> Not saying that sending directly is wrong, just listing things I've
>>>> run into that make me default the other way.
>>>>
>>> Yep. I wasn't talking about file rotation in terms of performance (since
>>> it should be relatively cheap anyway, especially if you don't want to
>>> compress old files). I was referring more to the manageability of the
>>> whole solution.
>>>
>>> Adding another ruleset seems easier than making sure that you have a
>>> proper directory prepared for the files, possibly reconfiguring your
>>> SIEM agent, making sure the files are included in the rotation scheme...
>>> Seems like a headache.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
rsyslog documentation suffers from being written by people too close to it's
development (too many assumptions baked in), given the troubleshooting excercise
that you just went through, would it be reasonable for you to try and writeup a
guide based on what we talked through with you to help the next person?

David Lang

On Wed, 11 May 2022, Steven D wrote:

> Date: Wed, 11 May 2022 22:56:38 +0000
> From: Steven D <pheerless@hotmail.com>
> To: John Chivian <jchivian@chivian.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
>
> Wanted to circle back around to this topic for a closure update.
>
> We were able to diagnose that one particular data source (a specific firewall vendor) appears to have trouble consistently sending TCP ... So we switched it to UDP and problem is gone. Ultimately no issues with rsyslog or the OS were at fault.
>
> The rest of the diagnostic help from here helped us locate a couple other issues we wouldnt have noticed otherwise.
>
> Thanks a lot all!
>
>
>
> Regards,
>
> Steven.
>
>
>
> -------- Original message --------
> From: John Chivian <jchivian@chivian.com>
> Date: 4/26/22 12:58 PM (GMT-05:00)
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>, Steven D <pheerless@hotmail.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> I second David's thought to switch to imptcp. It is designed for plain TCP performance.
>
> I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…”
>
> Thanks Gents!
>
>
>> On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????
>>
>> I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, April 26, 2022 10:23 AM
>> To: Steven D <pheerless@hotmail.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> that looks pretty good to me, if it's still running slow, I would look at the
>> threads in top to see what their utilization is, and try the imptcp vs imtcp and
>> see if there's a difference
>>
>> the way you are specifying a ruleset in an input means that you don't have lots
>> of rule processing happening.
>>
>> one thing you could have is contention for the lock on the main queue. By
>> default, all messages go to the main queue. If you are tieing an input to a
>> ruleset, you can give that ruleset a queue and the messages will never touch the
>> main queue (in effect, it's like you were running a separate rsyslog instance
>> for that input and ruleset)
>>
>> David Lang
>>
>>
>> On Tue, 26 Apr 2022, Steven D wrote:
>>
>>> Date: Tue, 26 Apr 2022 14:12:36 +0000
>>> From: Steven D <pheerless@hotmail.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
>>> ________________________________
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, April 26, 2022 9:57 AM
>>> To: Steven D <pheerless@hotmail.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> hmm, looks like it didn't make it through the mail server
>>>
>>> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>>>
>>> (also note there are similar UDP buffer tuning parameters, if those buffers fill
>>> up udp logs will be dropped silently)
>>>
>>> David Lang
>>>
>>> On Tue, 26 Apr 2022, Steven D wrote:
>>>
>>>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>>>> From: Steven D <pheerless@hotmail.com>
>>>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>>>
>>>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>>>
>>>> Regards,
>>>> Steven
>>>> ________________________________
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>>>> Sent: Monday, April 25, 2022 9:52 AM
>>>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Steven D <pheerless@hotmail.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>>>
>>>> Glad there are other SIEM jockeys on here... haha.
>>>> ________________________________
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>> Sent: Monday, April 25, 2022 9:08 AM
>>>> To: David Lang <david@lang.hm>
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>>
>>>> On 25.04.2022 15:04, David Lang wrote:
>>>>>>>> Sure. I work with them, I know ;-)
>>>>>>>>
>>>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>>>> process the message (even filter some events out or trim them or do
>>>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>>>> native SIEM API, not by syslog)
>>>>>>>>
>>>>>>>> instead of killing the server with IOPS. That's all.
>>>>>>>
>>>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>>>> It's sequential writes and reads, and in practice should be just
>>>>>>> sequential writes as the reads should be able to be served from ram
>>>>>>> (the files should be read shortly after they are written, so should
>>>>>>> still be cached)
>>>>>>>
>>>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>>>> of writing to disk and having the files read in batches vs posting
>>>>>>> each message to the SIEM. I think there are enough variables
>>>>>>> involved that it's not an obvious win either way.
>>>>>>>
>>>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>>>> in the end it all has to get written to the disk. ;-) I just think
>>>>>> that passing the events straight through rulesets from input to
>>>>>> output (and I mean the real destination, not to the files) is simply
>>>>>> more straightforward and easier to manage (after all you don't have
>>>>>> to - for example - deal with file rotation and so on). On the other
>>>>>> hand, you may run into some other problems when your rulesets get too
>>>>>> complicated and "run away".
>>>>>
>>>>> buffered, sequential writes to disk are pretty cheap and perform well
>>>>> even on slow spinning rust, and file rotation (should) happen during
>>>>> off-peak periods, and as previously mentioned gives you a handy backup
>>>>> for those cases where something goes wrong with your SIEM. Yes, you
>>>>> could have disk backed queues in rsyslog, but the performance of such
>>>>> queues is horrid and significantly more I/O load than writing to disk.
>>>>>
>>>>> Not saying that sending directly is wrong, just listing things I've
>>>>> run into that make me default the other way.
>>>>>
>>>> Yep. I wasn't talking about file rotation in terms of performance (since
>>>> it should be relatively cheap anyway, especially if you don't want to
>>>> compress old files). I was referring more to the manageability of the
>>>> whole solution.
>>>>
>>>> Adding another ruleset seems easier than making sure that you have a
>>>> proper directory prepared for the files, possibly reconfiguring your
>>>> SIEM agent, making sure the files are included in the rotation scheme...
>>>> Seems like a headache.
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
I can probably put some thoughts down...

I will admit that I was finally able to prove it wasn't rsyslog/OS by using a syslog stress/testing tool called "loggen"... Used it to replicate a log and exceed the data rates I was seeing from the problematic source... Basically proving reception side was working fine.

It's from syslog-ng, so I apologize for my blasphemy... I couldn't be bothered to build tcpflood.

Lol



Regards,

Steven.



-------- Original message --------
From: David Lang <david@lang.hm>
Date: 5/11/22 7:03 PM (GMT-05:00)
To: Steven D <pheerless@hotmail.com>
Cc: John Chivian <jchivian@chivian.com>, rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm>
Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting

rsyslog documentation suffers from being written by people too close to it's
development (too many assumptions baked in), given the troubleshooting excercise
that you just went through, would it be reasonable for you to try and writeup a
guide based on what we talked through with you to help the next person?

David Lang

On Wed, 11 May 2022, Steven D wrote:

> Date: Wed, 11 May 2022 22:56:38 +0000
> From: Steven D <pheerless@hotmail.com>
> To: John Chivian <jchivian@chivian.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
>
> Wanted to circle back around to this topic for a closure update.
>
> We were able to diagnose that one particular data source (a specific firewall vendor) appears to have trouble consistently sending TCP ... So we switched it to UDP and problem is gone. Ultimately no issues with rsyslog or the OS were at fault.
>
> The rest of the diagnostic help from here helped us locate a couple other issues we wouldnt have noticed otherwise.
>
> Thanks a lot all!
>
>
>
> Regards,
>
> Steven.
>
>
>
> -------- Original message --------
> From: John Chivian <jchivian@chivian.com>
> Date: 4/26/22 12:58 PM (GMT-05:00)
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>, Steven D <pheerless@hotmail.com>
> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>
> I second David's thought to switch to imptcp. It is designed for plain TCP performance.
>
> I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…”
>
> Thanks Gents!
>
>
>> On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????
>>
>> I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Tuesday, April 26, 2022 10:23 AM
>> To: Steven D <pheerless@hotmail.com>
>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> that looks pretty good to me, if it's still running slow, I would look at the
>> threads in top to see what their utilization is, and try the imptcp vs imtcp and
>> see if there's a difference
>>
>> the way you are specifying a ruleset in an input means that you don't have lots
>> of rule processing happening.
>>
>> one thing you could have is contention for the lock on the main queue. By
>> default, all messages go to the main queue. If you are tieing an input to a
>> ruleset, you can give that ruleset a queue and the messages will never touch the
>> main queue (in effect, it's like you were running a separate rsyslog instance
>> for that input and ruleset)
>>
>> David Lang
>>
>>
>> On Tue, 26 Apr 2022, Steven D wrote:
>>
>>> Date: Tue, 26 Apr 2022 14:12:36 +0000
>>> From: Steven D <pheerless@hotmail.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
>>> ________________________________
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, April 26, 2022 9:57 AM
>>> To: Steven D <pheerless@hotmail.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> hmm, looks like it didn't make it through the mail server
>>>
>>> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>>>
>>> (also note there are similar UDP buffer tuning parameters, if those buffers fill
>>> up udp logs will be dropped silently)
>>>
>>> David Lang
>>>
>>> On Tue, 26 Apr 2022, Steven D wrote:
>>>
>>>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>>>> From: Steven D <pheerless@hotmail.com>
>>>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>>>
>>>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>>>
>>>> Regards,
>>>> Steven
>>>> ________________________________
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>>>> Sent: Monday, April 25, 2022 9:52 AM
>>>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Steven D <pheerless@hotmail.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>>>
>>>> Glad there are other SIEM jockeys on here... haha.
>>>> ________________________________
>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>> Sent: Monday, April 25, 2022 9:08 AM
>>>> To: David Lang <david@lang.hm>
>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>>
>>>> On 25.04.2022 15:04, David Lang wrote:
>>>>>>>> Sure. I work with them, I know ;-)
>>>>>>>>
>>>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>>>> process the message (even filter some events out or trim them or do
>>>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>>>> native SIEM API, not by syslog)
>>>>>>>>
>>>>>>>> instead of killing the server with IOPS. That's all.
>>>>>>>
>>>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>>>> It's sequential writes and reads, and in practice should be just
>>>>>>> sequential writes as the reads should be able to be served from ram
>>>>>>> (the files should be read shortly after they are written, so should
>>>>>>> still be cached)
>>>>>>>
>>>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>>>> of writing to disk and having the files read in batches vs posting
>>>>>>> each message to the SIEM. I think there are enough variables
>>>>>>> involved that it's not an obvious win either way.
>>>>>>>
>>>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>>>> in the end it all has to get written to the disk. ;-) I just think
>>>>>> that passing the events straight through rulesets from input to
>>>>>> output (and I mean the real destination, not to the files) is simply
>>>>>> more straightforward and easier to manage (after all you don't have
>>>>>> to - for example - deal with file rotation and so on). On the other
>>>>>> hand, you may run into some other problems when your rulesets get too
>>>>>> complicated and "run away".
>>>>>
>>>>> buffered, sequential writes to disk are pretty cheap and perform well
>>>>> even on slow spinning rust, and file rotation (should) happen during
>>>>> off-peak periods, and as previously mentioned gives you a handy backup
>>>>> for those cases where something goes wrong with your SIEM. Yes, you
>>>>> could have disk backed queues in rsyslog, but the performance of such
>>>>> queues is horrid and significantly more I/O load than writing to disk.
>>>>>
>>>>> Not saying that sending directly is wrong, just listing things I've
>>>>> run into that make me default the other way.
>>>>>
>>>> Yep. I wasn't talking about file rotation in terms of performance (since
>>>> it should be relatively cheap anyway, especially if you don't want to
>>>> compress old files). I was referring more to the manageability of the
>>>> whole solution.
>>>>
>>>> Adding another ruleset seems easier than making sure that you have a
>>>> proper directory prepared for the files, possibly reconfiguring your
>>>> SIEM agent, making sure the files are included in the rotation scheme...
>>>> Seems like a headache.
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Basic Rsyslog Troubleshooting [ In reply to ]
I tended to use tcpreplay to play back a captured stream faster, or website
testing tools (apache benchmark or similar) to make lots of short connections.
whatever works for you.

David Lang

On Wed, 11 May 2022, Steven D wrote:

> Date: Wed, 11 May 2022 23:12:17 +0000
> From: Steven D <pheerless@hotmail.com>
> To: David Lang <david@lang.hm>
> Cc: John Chivian <jchivian@chivian.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
>
> I can probably put some thoughts down...
>
> I will admit that I was finally able to prove it wasn't rsyslog/OS by using a syslog stress/testing tool called "loggen"... Used it to replicate a log and exceed the data rates I was seeing from the problematic source... Basically proving reception side was working fine.
>
> It's from syslog-ng, so I apologize for my blasphemy... I couldn't be bothered to build tcpflood.
>
> Lol
>
>
>
> Regards,
>
> Steven.
>
>
>
> -------- Original message --------
> From: David Lang <david@lang.hm>
> Date: 5/11/22 7:03 PM (GMT-05:00)
> To: Steven D <pheerless@hotmail.com>
> Cc: John Chivian <jchivian@chivian.com>, rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm>
> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
>
> rsyslog documentation suffers from being written by people too close to it's
> development (too many assumptions baked in), given the troubleshooting excercise
> that you just went through, would it be reasonable for you to try and writeup a
> guide based on what we talked through with you to help the next person?
>
> David Lang
>
> On Wed, 11 May 2022, Steven D wrote:
>
>> Date: Wed, 11 May 2022 22:56:38 +0000
>> From: Steven D <pheerless@hotmail.com>
>> To: John Chivian <jchivian@chivian.com>,
>> rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: David Lang <david@lang.hm>
>> Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> Wanted to circle back around to this topic for a closure update.
>>
>> We were able to diagnose that one particular data source (a specific firewall vendor) appears to have trouble consistently sending TCP ... So we switched it to UDP and problem is gone. Ultimately no issues with rsyslog or the OS were at fault.
>>
>> The rest of the diagnostic help from here helped us locate a couple other issues we wouldnt have noticed otherwise.
>>
>> Thanks a lot all!
>>
>>
>>
>> Regards,
>>
>> Steven.
>>
>>
>>
>> -------- Original message --------
>> From: John Chivian <jchivian@chivian.com>
>> Date: 4/26/22 12:58 PM (GMT-05:00)
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: David Lang <david@lang.hm>, Steven D <pheerless@hotmail.com>
>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>
>> I second David's thought to switch to imptcp. It is designed for plain TCP performance.
>>
>> I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…”
>>
>> Thanks Gents!
>>
>>
>>> On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>>
>>> Interesting... thanks for the tips. As long as there's no overt issues, I won't press my luck by clobbering the config ????
>>>
>>> I think what we're seeing at this point is some system side inability to handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that rabbithole.
>>> ________________________________
>>> From: David Lang <david@lang.hm>
>>> Sent: Tuesday, April 26, 2022 10:23 AM
>>> To: Steven D <pheerless@hotmail.com>
>>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>
>>> that looks pretty good to me, if it's still running slow, I would look at the
>>> threads in top to see what their utilization is, and try the imptcp vs imtcp and
>>> see if there's a difference
>>>
>>> the way you are specifying a ruleset in an input means that you don't have lots
>>> of rule processing happening.
>>>
>>> one thing you could have is contention for the lock on the main queue. By
>>> default, all messages go to the main queue. If you are tieing an input to a
>>> ruleset, you can give that ruleset a queue and the messages will never touch the
>>> main queue (in effect, it's like you were running a separate rsyslog instance
>>> for that input and ruleset)
>>>
>>> David Lang
>>>
>>>
>>> On Tue, 26 Apr 2022, Steven D wrote:
>>>
>>>> Date: Tue, 26 Apr 2022 14:12:36 +0000
>>>> From: Steven D <pheerless@hotmail.com>
>>>> To: David Lang <david@lang.hm>
>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in.
>>>> ________________________________
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Tuesday, April 26, 2022 9:57 AM
>>>> To: Steven D <pheerless@hotmail.com>
>>>> Cc: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>
>>>> hmm, looks like it didn't make it through the mail server
>>>>
>>>> https://cromwell-intl.com/open-source/performance-tuning/tcp.html
>>>>
>>>> (also note there are similar UDP buffer tuning parameters, if those buffers fill
>>>> up udp logs will be dropped silently)
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 26 Apr 2022, Steven D wrote:
>>>>
>>>>> Date: Tue, 26 Apr 2022 11:00:16 +0000
>>>>> From: Steven D <pheerless@hotmail.com>
>>>>> To: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>>
>>>>> Here's most recent few rotations of pstats data, any additional input would be appreciated.
>>>>>
>>>>> With the keepalives set, TCP connections don't seem to be growing...however it still seems that using TCP (at least for the high throughput firewalls) causes ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that youz could suggestion that make help?
>>>>>
>>>>> Regards,
>>>>> Steven
>>>>> ________________________________
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Steven D via rsyslog <rsyslog@lists.adiscon.com>
>>>>> Sent: Monday, April 25, 2022 9:52 AM
>>>>> To: David Lang <david@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Cc: Steven D <pheerless@hotmail.com>
>>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>>
>>>>> Kind of depends on SIEM agent... With our set up, having the files written to disk allows us to target/optimize the front-end parsing applied by the SIEM agent a little better. Make is so there's less work to be done elsewhere in the solution, but it could be done other ways.
>>>>>
>>>>> Glad there are other SIEM jockeys on here... haha.
>>>>> ________________________________
>>>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>>> Sent: Monday, April 25, 2022 9:08 AM
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>; Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
>>>>>
>>>>>
>>>>> On 25.04.2022 15:04, David Lang wrote:
>>>>>>>>> Sure. I work with them, I know ;-)
>>>>>>>>>
>>>>>>>>> It's just that for some, you can do the same but using rsyslog to
>>>>>>>>> process the message (even filter some events out or trim them or do
>>>>>>>>> many other fancy stuff) an send them directly to SIEM (by means of
>>>>>>>>> native SIEM API, not by syslog)
>>>>>>>>>
>>>>>>>>> instead of killing the server with IOPS. That's all.
>>>>>>>>
>>>>>>>> fair point, but I'll point out that since rsyslog is buffering the
>>>>>>>> writes to disk, the IOPS load is not as high as you would think.
>>>>>>>> It's sequential writes and reads, and in practice should be just
>>>>>>>> sequential writes as the reads should be able to be served from ram
>>>>>>>> (the files should be read shortly after they are written, so should
>>>>>>>> still be cached)
>>>>>>>>
>>>>>>>> It would be interesting to look into the performance (CPU and I/O)
>>>>>>>> of writing to disk and having the files read in batches vs posting
>>>>>>>> each message to the SIEM. I think there are enough variables
>>>>>>>> involved that it's not an obvious win either way.
>>>>>>>>
>>>>>>> Sure, there is also another layer of OS-level caching/buffering. But
>>>>>>> in the end it all has to get written to the disk. ;-) I just think
>>>>>>> that passing the events straight through rulesets from input to
>>>>>>> output (and I mean the real destination, not to the files) is simply
>>>>>>> more straightforward and easier to manage (after all you don't have
>>>>>>> to - for example - deal with file rotation and so on). On the other
>>>>>>> hand, you may run into some other problems when your rulesets get too
>>>>>>> complicated and "run away".
>>>>>>
>>>>>> buffered, sequential writes to disk are pretty cheap and perform well
>>>>>> even on slow spinning rust, and file rotation (should) happen during
>>>>>> off-peak periods, and as previously mentioned gives you a handy backup
>>>>>> for those cases where something goes wrong with your SIEM. Yes, you
>>>>>> could have disk backed queues in rsyslog, but the performance of such
>>>>>> queues is horrid and significantly more I/O load than writing to disk.
>>>>>>
>>>>>> Not saying that sending directly is wrong, just listing things I've
>>>>>> run into that make me default the other way.
>>>>>>
>>>>> Yep. I wasn't talking about file rotation in terms of performance (since
>>>>> it should be relatively cheap anyway, especially if you don't want to
>>>>> compress old files). I was referring more to the manageability of the
>>>>> whole solution.
>>>>>
>>>>> Adding another ruleset seems easier than making sure that you have a
>>>>> proper directory prepared for the files, possibly reconfiguring your
>>>>> SIEM agent, making sure the files are included in the rotation scheme...
>>>>> Seems like a headache.
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

1 2  View All