Mailing List Archive: omudpspoof - any metrics to monitor?

omudpspoof - any metrics to monitor?

Oct 28, 2019, 3:36 PM

Post #1 of 3 (270 views)

Hello all,

I know using omudpspoof plugin is not a good idea at all, but it's only
applicable to my use case. I understood there is some penalty in
performance because of socket operations in background. Is there any way
to monitor that? Any kind of metrics I can read and understand what is
server suffering of? When I use omfwd CPU utilization is high, but all
events are forwarded almost instantly. But rsyslog starts caching when
omudpspoof comes into play. So apparently rsyslog mainQ is aware of
omudpspoof performance and it tries to slow down the flow as much as it
can. While this can work for peaks, it can't work as long term solution.
So far I tried different ways (incl iostats, sar, netstat, ...) to
isolate some metric which can tell me when udpspoof plugin meets its
limits, but without success. Any ideas are more than welcome!

Regards,

Petr

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: omudpspoof - any metrics to monitor? [ In reply to ]

rsyslog at lists

Oct 28, 2019, 8:18 PM

Post #2 of 3 (270 views)

Permalink

start by enabling impstats, that's needed to see what's happening and where the
bottlenexks are.

omudpspoof is slow becuse it needs to make lots of system calls, there's no
direct way to measure this cost, but if you have it as an output with it's own
action queue, you can see the performance of the resulting thread (hit 'H' in
top to per-thread stats)

omudpspoof was a hack I put together to solve the problem. I'ts gotten some
cleanup since I submitted it, but it could be improved significantly by getting
a cache similar to hat dynafiles uses, but it's still going to be significantly
slower than normal output. Someone would need to sponsor such work or contribute
it as this is a low-priority enhancement for the core team.

It can probably use additional metrics as well.

David Lang

On Mon, 28 Oct 2019, Petr Vyhnal via rsyslog wrote:

> Date: Mon, 28 Oct 2019 23:36:32 +0100
> From: Petr Vyhnal via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Petr Vyhnal <rudiik@simerion.net>
> Subject: [rsyslog] omudpspoof - any metrics to monitor?
>
> Hello all,
>
>
> I know using omudpspoof plugin is not a good idea at all, but it's only
> applicable to my use case. I understood there is some penalty in performance
> because of socket operations in background. Is there any way to monitor that?
> Any kind of metrics I can read and understand what is server suffering of?
> When I use omfwd CPU utilization is high, but all events are forwarded almost
> instantly. But rsyslog starts caching when omudpspoof comes into play. So
> apparently rsyslog mainQ is aware of omudpspoof performance and it tries to
> slow down the flow as much as it can. While this can work for peaks, it can't
> work as long term solution. So far I tried different ways (incl iostats, sar,
> netstat, ...) to isolate some metric which can tell me when udpspoof plugin
> meets its limits, but without success. Any ideas are more than welcome!
>
>
> Regards,
>
> Petr
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: omudpspoof - any metrics to monitor? [ In reply to ]

rsyslog at lists

Nov 1, 2019, 2:52 AM

Post #3 of 3 (266 views)

Permalink

Thanks David for sharing details. Actually impstats was my first idea as
well. I will play with it a bit in my lab to monitor performance penalty
in reality.

Thank you!

Petr

On 29.10.2019 4:18, David Lang wrote:
> start by enabling impstats, that's needed to see what's happening and
> where the bottlenexks are.
>
> omudpspoof is slow becuse it needs to make lots of system calls,
> there's no direct way to measure this cost, but if you have it as an
> output with it's own action queue, you can see the performance of the
> resulting thread (hit 'H' in top to per-thread stats)
>
> omudpspoof was a hack I put together to solve the problem. I'ts gotten
> some cleanup since I submitted it, but it could be improved
> significantly by getting a cache similar to hat dynafiles uses, but
> it's still going to be significantly slower than normal output.
> Someone would need to sponsor such work or contribute it as this is a
> low-priority enhancement for the core team.
>
> It can probably use additional metrics as well.
>
> David Lang
>
>
> On Mon, 28 Oct 2019, Petr Vyhnal via rsyslog wrote:
>
>> Date: Mon, 28 Oct 2019 23:36:32 +0100
>> From: Petr Vyhnal via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Petr Vyhnal <rudiik@simerion.net>
>> Subject: [rsyslog] omudpspoof - any metrics to monitor?
>>
>> Hello all,
>>
>>
>> I know using omudpspoof plugin is not a good idea at all, but it's
>> only applicable to my use case. I understood there is some penalty in
>> performance because of socket operations in background. Is there any
>> way to monitor that? Any kind of metrics I can read and understand
>> what is server suffering of? When I use omfwd CPU utilization is
>> high, but all events are forwarded almost instantly. But rsyslog
>> starts caching when omudpspoof comes into play. So apparently rsyslog
>> mainQ is aware of omudpspoof performance and it tries to slow down
>> the flow as much as it can. While this can work for peaks, it can't
>> work as long term solution. So far I tried different ways (incl
>> iostats, sar, netstat, ...) to isolate some metric which can tell me
>> when udpspoof plugin meets its limits, but without success. Any ideas
>> are more than welcome!
>>
>>
>> Regards,
>>
>> Petr
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.