Mailing List Archive

imtcp performance
Would like to know your experience with imtcp and/or imptcp.

With +1100 established TCP connection we get ~100% CPU usage on imtcp
thread causing the TCP stack/connections being stalled/not possible to
establish.

TOP screen:
Threads: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie
%Cpu0 : 21.6 us, 76.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.0 si,
0.0 st
%Cpu1 : 7.5 us, 5.7 sy, 0.0 ni, 83.0 id, 0.0 wa, 0.0 hi, 3.8 si,
0.0 st
KiB Mem: 8197852 total, 7946872 used, 250980 free, 33520 buffers
KiB Swap: 1048572 total, 164648 used, 883924 free. 7067060 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13760 loglin 20 0 826836 24972 4676 R 90.6 0.3 64:33.11 in:imtcp
13761 loglin 20 0 826836 24972 4676 R 5.9 0.3 8:09.49 rs:main
Q:Reg
857 root 20 0 25764 3024 2492 S 2.0 0.0 0:02.72 top
12871 loglocal 20 0 757816 19452 712 S 2.0 0.2 150:04.44 in:imtcp
13762 loglin 20 0 826836 24972 4676 S 2.0 0.3 1:59.07 in:imtcp

Is that expected behavior? What is the suitable number of connections and
throughput imtcp can handle?

--
Peter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: imtcp performance [ In reply to ]
With +1100 established TCP connection we get ~100% CPU usage on imtcp
thread causing the TCP stack/connections being stalled/not possible to
establish.


My syslog server is receiving at a constant 20 megabits, surging to 70 megabits, from around 400 hosts. My CPU load is .28 and my processor usage is 4% right now and maxes out at 20% on a 4 proc 12 gig memory VM. I'm using imptcp though. I also spent a few weeks tinkering with settings to get it this far. At one point I was hitting 100% CPU usage and refusing messages. Found out it was mostly too many threads causing my issue, but I also split things up into multiple queues to enable parallel processing.

Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | Fax: | Mobile: 7349151444 | E-mail: Scot.Kreienkamp@la-z-boy.com
www.la-z-boy.com |facebook.com/lazboy |twitter.com/lazboy |youtube.com/lazboy
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: imtcp performance [ In reply to ]
Regarding throughput, we've had great success using imptcp. (note the "p"
in there).

Scott's 20mb/s to 70mb/s mean something like 250GB/day to 750GB/day ... and
his machine is not sweating at *all*.
We've configured a single (beefy but not crazy) EC2 instance and done more
than 15TB a day with less than 60% CPU and that's including a lot of heavy
processing of the data.
The network/imptcp cost is negligible.

-D.

On Wed, Sep 18, 2019 at 7:02 AM Scot Kreienkamp via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> With +1100 established TCP connection we get ~100% CPU usage on imtcp
> thread causing the TCP stack/connections being stalled/not possible to
> establish.
>
>
> My syslog server is receiving at a constant 20 megabits, surging to 70
> megabits, from around 400 hosts. My CPU load is .28 and my processor usage
> is 4% right now and maxes out at 20% on a 4 proc 12 gig memory VM. I'm
> using imptcp though. I also spent a few weeks tinkering with settings to
> get it this far. At one point I was hitting 100% CPU usage and refusing
> messages. Found out it was mostly too many threads causing my issue, but I
> also split things up into multiple queues to enable parallel processing.
>
> Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | Fax:
> | Mobile: 7349151444 | E-mail: Scot.Kreienkamp@la-z-boy.com
> www.la-z-boy.com |facebook.com/lazboy |twitter.com/lazboy |
> youtube.com/lazboy
> This message is intended only for the individual or entity to which it is
> addressed. It may contain privileged, confidential information which is
> exempt from disclosure under applicable laws. If you are not the intended
> recipient, you are strictly prohibited from disseminating or distributing
> this information (other than to the intended recipient) or copying this
> information. If you have received this communication in error, please
> notify us immediately by e-mail or by telephone at the above number. Thank
> you.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: imtcp performance [ In reply to ]
If the 1100 simultaneous imtcp connections use encryption of any type
this is probably your issue.  There are lots of rsyslog implementations
with that many or more listeners or clients, but I would guess most use
the imptcp module that does not have the encryption overhead.

Regards,


On 9/18/19 8:34 AM, Peter Viskup via rsyslog wrote:
> Would like to know your experience with imtcp and/or imptcp.
>
> With +1100 established TCP connection we get ~100% CPU usage on imtcp
> thread causing the TCP stack/connections being stalled/not possible to
> establish.
>
> TOP screen:
> Threads: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie
> %Cpu0 : 21.6 us, 76.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.0 si,
> 0.0 st
> %Cpu1 : 7.5 us, 5.7 sy, 0.0 ni, 83.0 id, 0.0 wa, 0.0 hi, 3.8 si,
> 0.0 st
> KiB Mem: 8197852 total, 7946872 used, 250980 free, 33520 buffers
> KiB Swap: 1048572 total, 164648 used, 883924 free. 7067060 cached Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 13760 loglin 20 0 826836 24972 4676 R 90.6 0.3 64:33.11 in:imtcp
> 13761 loglin 20 0 826836 24972 4676 R 5.9 0.3 8:09.49 rs:main
> Q:Reg
> 857 root 20 0 25764 3024 2492 S 2.0 0.0 0:02.72 top
> 12871 loglocal 20 0 757816 19452 712 S 2.0 0.2 150:04.44 in:imtcp
> 13762 loglin 20 0 826836 24972 4676 S 2.0 0.3 1:59.07 in:imtcp
>
> Is that expected behavior? What is the suitable number of connections and
> throughput imtcp can handle?
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.