On Thu, 24 Jul 2008, Rainer Gerhards wrote:
> (I am not commenting on v2 vs. v3 as this is already done)
>
> First of all, we need to keep in mind that UDP is inherently lossy. Even
> when a frame is seen received by the local stack, it does not mean that
> it will eventually be forwarded to the application. If message bursts
> come in very quickly and the OS scheduler does not schedule the app fast
> enough to receive this messages (or the app is too slow in itself! ;))
> new frames may overwrite frames inside the stack's receive buffers.
>
> So it is always a good idea to avoid UDP if that's possible.
>
> HOWEVER, I, too, find it somewhat unusual that around 90% of all
> incoming frames are discarded before the rsyslog receiver could process
> them. One explanation I have is that you have bursts (or volume in
> general) that outperforms the configured actions. Having seen the config
> file, and seeing it does not include any database writer, it is hard to
> imagine this should happen, assuming reasonable hardware sizing is used.
> A cause could be excessive synchronous writes. Many rules do not put a
> dash in front of the file name and without it (in v2), every write is
> immediately synced. This is very costly. But still, I have never seen
> that this alone outperforms a system.
>
> To dig deeper into what is happening, a debug log would be most useful,
> together with the information which frames have been seen in tcpdump but
> NOT in one of the log files. You can enable debug mode via -dn command
> line switch and is recommended to run rsyslog interactively while doing
> so. Then, you can simply capture its output via stdout redirection.
> Please note that debug mode generates considerable output, and requires
> considerable additional processing time. In any case, though, it should
> show us where the bottleneck is. Please note that I need a consistent
> excerpt from the debug log that shows how things began and how it worked
> during the fault conditions. Usually, this means I need everything ;)
> Debug logs may also reveal sensitive information, even passwords, so you
> should be careful in what you do. I am used to log files around the size
> of 1GB. With reasonable compression, the transfer is usually not a
> problem (but I suggest you place them on a server for me to download).
> Download links and/or smaller logs you can email me privately at
> rgerhards at gmail.com (please NOT at my primary, adiscon, email address).
>
> I hope this helps and I am looking forward for the additional
> information.
So after a long hiatus and a new config the problems went away but only
temporarily. I think because of a rebooted box. They have returned. I'm
going to mail the logs your way. I can send you more. Its very easy to
reproduce.
-Mike
============== Snip ===================
>
> On Wed, 2008-07-23 at 14:21 -0500, Mike McGrath wrote:
> > I've got a RHEL5.2 host with rsyslog-2.0.0-11 installed as a central
> > logging server. When running tcpdump I'm seeing all the udp packets
> > coming in but many of them are not getting logged. And we're talking
> > like 10% or so getting logged (maybe less) and the rest are just lost.
> > I've attached my config file.
> >
> > (side note, if I'm doing something stupid in the config please correct me)
> >
> > -Mike
> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> (I am not commenting on v2 vs. v3 as this is already done)
>
> First of all, we need to keep in mind that UDP is inherently lossy. Even
> when a frame is seen received by the local stack, it does not mean that
> it will eventually be forwarded to the application. If message bursts
> come in very quickly and the OS scheduler does not schedule the app fast
> enough to receive this messages (or the app is too slow in itself! ;))
> new frames may overwrite frames inside the stack's receive buffers.
>
> So it is always a good idea to avoid UDP if that's possible.
>
> HOWEVER, I, too, find it somewhat unusual that around 90% of all
> incoming frames are discarded before the rsyslog receiver could process
> them. One explanation I have is that you have bursts (or volume in
> general) that outperforms the configured actions. Having seen the config
> file, and seeing it does not include any database writer, it is hard to
> imagine this should happen, assuming reasonable hardware sizing is used.
> A cause could be excessive synchronous writes. Many rules do not put a
> dash in front of the file name and without it (in v2), every write is
> immediately synced. This is very costly. But still, I have never seen
> that this alone outperforms a system.
>
> To dig deeper into what is happening, a debug log would be most useful,
> together with the information which frames have been seen in tcpdump but
> NOT in one of the log files. You can enable debug mode via -dn command
> line switch and is recommended to run rsyslog interactively while doing
> so. Then, you can simply capture its output via stdout redirection.
> Please note that debug mode generates considerable output, and requires
> considerable additional processing time. In any case, though, it should
> show us where the bottleneck is. Please note that I need a consistent
> excerpt from the debug log that shows how things began and how it worked
> during the fault conditions. Usually, this means I need everything ;)
> Debug logs may also reveal sensitive information, even passwords, so you
> should be careful in what you do. I am used to log files around the size
> of 1GB. With reasonable compression, the transfer is usually not a
> problem (but I suggest you place them on a server for me to download).
> Download links and/or smaller logs you can email me privately at
> rgerhards at gmail.com (please NOT at my primary, adiscon, email address).
>
> I hope this helps and I am looking forward for the additional
> information.
So after a long hiatus and a new config the problems went away but only
temporarily. I think because of a rebooted box. They have returned. I'm
going to mail the logs your way. I can send you more. Its very easy to
reproduce.
-Mike
============== Snip ===================
>
> On Wed, 2008-07-23 at 14:21 -0500, Mike McGrath wrote:
> > I've got a RHEL5.2 host with rsyslog-2.0.0-11 installed as a central
> > logging server. When running tcpdump I'm seeing all the udp packets
> > coming in but many of them are not getting logged. And we're talking
> > like 10% or so getting logged (maybe less) and the rest are just lost.
> > I've attached my config file.
> >
> > (side note, if I'm doing something stupid in the config please correct me)
> >
> > -Mike
> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>