Mailing List Archive

Hi David,

thanks for your message and interest. This is very welcome as I would
really like to get hold of more insight into rsyslog's performance in
real-world and extreme scenarios. So far, I have unfortunately been
unable to do that, because I had no funding for the required hardware
and time to conduct the testing. I would *really* appreciate if you
could help with that and I would be very willing to tweak the code for
optimum performance (actually, I am always very concerned about high
performance and I was a bit sad about the fact that I could not ensure
it).

Having said that, I'd first of all would like to have a look at your
rsyslog.conf, so that we are on common ground. Also, I think, this can
become quite lengthy and also of interest for others. May it be useful
to discuss this on the web forum, so that we can easier keep track of
things. If you think this is useful, I recommend this one here:

http://kb.monitorware.com/general-f34.html

some more comments inline:

On Mon, 2008-09-15 at 07:23 +0200, david at lang.hm wrote:
> I've got a couple projects I'm working on that rsyslog is looking like
> a
> good answer to. I'm going to send questions about them in seperate
> messages.
>
> the first project is a traditional syslog server. I've been testing
> various syslog implementations to find out what sort of performance
> they
> can sustain.
>
> I've gotten the (almost) standard sysklogd to sustail almost 30,000
> messages/sec (udp) before it starts loosing significant numbers of
> logs
>
> rsyslog is able to handle bursts of around 150,000 messages/sec, but
> it
> doesn't seem to write them out very fast, and over time seems to be
> limited to ~22,000 messages/sec
>
> the hardware I am running this test on is insanely overpowered (four
> dual-core opterons, 16G of ram, battery-backed cache on a raid card
> with
> high-speed SAS drives). in production I will be working with slower
> hardware, but I'm trying to find the limits of the software before I
> start
> introducing lower hardware limits.
>
> what can I look at doing to speed up the writing of data from the
> queues?
> does rsyslog write the messages one at a time, or does it have an
> option
> for writing them in batches (pulling a bunch off of the queue and
> sending
> them all at once)?

The design is that each individual message is pushed to the output. The
output than writes the message. There has been discussion about a lazy
write, but nothing in this regard has yet been implemented.

Note that this is a good time to request new things. I am finished with
the last major thing (TLS) support and in front of the next one
(enhanced scripting), but scripting, I think, could be moved to a later
time in favour of something more important (like performance ;)).
>
> some sort of batch mode would be critical for writes to a database
> where
> you can frequently get 1000x speedups if you do the inserts as a
> single
> transaction as opposed to individual transactions.

I am not (any longer) so much a database guy. Can I really do a batch
insert with them? Can you point me to an API?

Looking forward to your reply!
Rainer

On Mon, 15 Sep 2008, Rainer Gerhards wrote:

> Hi David,
>
> thanks for your message and interest. This is very welcome as I would
> really like to get hold of more insight into rsyslog's performance in
> real-world and extreme scenarios. So far, I have unfortunately been
> unable to do that, because I had no funding for the required hardware
> and time to conduct the testing. I would *really* appreciate if you
> could help with that and I would be very willing to tweak the code for
> optimum performance (actually, I am always very concerned about high
> performance and I was a bit sad about the fact that I could not ensure
> it).
>
> Having said that, I'd first of all would like to have a look at your
> rsyslog.conf, so that we are on common ground. Also, I think, this can
> become quite lengthy and also of interest for others. May it be useful
> to discuss this on the web forum, so that we can easier keep track of
> things. If you think this is useful, I recommend this one here:
>
> http://kb.monitorware.com/general-f34.html

done, I created a topic in the developers corner

David Lang

> some more comments inline:
>
> On Mon, 2008-09-15 at 07:23 +0200, david at lang.hm wrote:
>> I've got a couple projects I'm working on that rsyslog is looking like
>> a
>> good answer to. I'm going to send questions about them in seperate
>> messages.
>>
>> the first project is a traditional syslog server. I've been testing
>> various syslog implementations to find out what sort of performance
>> they
>> can sustain.
>>
>> I've gotten the (almost) standard sysklogd to sustail almost 30,000
>> messages/sec (udp) before it starts loosing significant numbers of
>> logs
>>
>> rsyslog is able to handle bursts of around 150,000 messages/sec, but
>> it
>> doesn't seem to write them out very fast, and over time seems to be
>> limited to ~22,000 messages/sec
>>
>> the hardware I am running this test on is insanely overpowered (four
>> dual-core opterons, 16G of ram, battery-backed cache on a raid card
>> with
>> high-speed SAS drives). in production I will be working with slower
>> hardware, but I'm trying to find the limits of the software before I
>> start
>> introducing lower hardware limits.
>>
>> what can I look at doing to speed up the writing of data from the
>> queues?
>> does rsyslog write the messages one at a time, or does it have an
>> option
>> for writing them in batches (pulling a bunch off of the queue and
>> sending
>> them all at once)?
>
> The design is that each individual message is pushed to the output. The
> output than writes the message. There has been discussion about a lazy
> write, but nothing in this regard has yet been implemented.
>
> Note that this is a good time to request new things. I am finished with
> the last major thing (TLS) support and in front of the next one
> (enhanced scripting), but scripting, I think, could be moved to a later
> time in favour of something more important (like performance ;)).
>>
>> some sort of batch mode would be critical for writes to a database
>> where
>> you can frequently get 1000x speedups if you do the inserts as a
>> single
>> transaction as opposed to individual transactions.
>
> I am not (any longer) so much a database guy. Can I really do a batch
> insert with them? Can you point me to an API?
>
> Looking forward to your reply!
> Rainer
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>