Mailing List Archive

Feedback request: in-memory buffering messages
Hi all,

I just got an idea for a feature inspired by the ramlog project. I have
blogged about it:

http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-buffer.h
tml

If you have some minutes to spare, please review and provide feedback.

Thanks,
Rainer
Feedback request: in-memory buffering messages [ In reply to ]
Hi Rainer,

> Hi all,
>
> I just got an idea for a feature inspired by the ramlog project. I have
> blogged about it:
>
> http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-buffer.h
> tml
>
> If you have some minutes to spare, please review and provide feedback.

I've read and honestly have no opinion on it :)

The last time I tried delayed writes with rsyslog the machines I set them up
on basically died (load averages above 30, so they were complete unresponsive).

I log rsyslog to a MySQL DB (for maillogs only) and it took me a while to
realise it was the rsyslog changes I made that killed the mail servers. They
even hung on boot ie. would not complete a boot process. When I got into
single user mode and didn't start up any services, I was able to change the
rsyslog.conf file back to my older one, and everything was fine again.

This was some versions back (but still the 3.x stable series) but because
these were production servers, I didn't bother analysing what went wrong or
why it happened, I was just glad I had my mail servers back again and working.

I might give this a second attempt at some stage this year, because I like the
feature for queuing entries if the DB is down, but until then, I really don't
have much to say about the delayed writes stuff :)

Michael.

> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
------- End of Original Message -------
Feedback request: in-memory buffering messages [ In reply to ]
Hi Michael,

I think you were hit by this bug:

http://bugzilla.adiscon.com/show_bug.cgi?id=86

interestingly, nobody who experienced it bothered to report, until
recently. Thus it was for quite a while inside the core engine ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Mansour
> Sent: Thursday, July 17, 2008 1:09 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback request: in-memory buffering messages
>
> Hi Rainer,
>
> > Hi all,
> >
> > I just got an idea for a feature inspired by the ramlog project. I
> have
> > blogged about it:
> >
> > http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-
> buffer.h
> > tml
> >
> > If you have some minutes to spare, please review and provide
> feedback.
>
> I've read and honestly have no opinion on it :)
>
> The last time I tried delayed writes with rsyslog the machines I set
> them up
> on basically died (load averages above 30, so they were complete
> unresponsive).
>
> I log rsyslog to a MySQL DB (for maillogs only) and it took me a while
> to
> realise it was the rsyslog changes I made that killed the mail
servers.
> They
> even hung on boot ie. would not complete a boot process. When I got
> into
> single user mode and didn't start up any services, I was able to
change
> the
> rsyslog.conf file back to my older one, and everything was fine again.
>
> This was some versions back (but still the 3.x stable series) but
> because
> these were production servers, I didn't bother analysing what went
> wrong or
> why it happened, I was just glad I had my mail servers back again and
> working.
>
> I might give this a second attempt at some stage this year, because I
> like the
> feature for queuing entries if the DB is down, but until then, I
really
> don't
> have much to say about the delayed writes stuff :)
>
> Michael.
>
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> ------- End of Original Message -------
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
Feedback request: in-memory buffering messages [ In reply to ]
Hi Rainer-san,

+1

How to use logging facilities varies according to circumstances.
For example, RHEL is used for from edge to mission critical.
Our customers want robust logging functions, like writing to MySQL or
transporting via TCP/RELP.

But some users want 'greener' machines.
It sounds interesting to combine the feature to write log messages to
RAM and the powertop project.
http://www.lesswatts.org/projects/powertop/

Best Rio.

On 2008/07/17, at 19:43, Rainer Gerhards wrote:

> Hi all,
>
> I just got an idea for a feature inspired by the ramlog project. I
> have
> blogged about it:
>
> http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-buffer.h
> tml
>
> If you have some minutes to spare, please review and provide feedback.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

########################################################################
Ryo Fujita <rfujita at redhat.com>
Senior Solution Architect, RHCE
Red Hat K.K.
TEL +81-3-5798-8500
FAX +81-3-5798-8599
Ebisu Neonato 8F
4-1-18 Ebisu, Shibuya-ku,
Tokyo Japan 1500013
########################################################################
Feedback request: in-memory buffering messages [ In reply to ]
This is certainly something I would use. I've got a number of mission
critical devices running off of flash drives (OpenBSD firewalls), and
the less they have to write to disk the better. This would also be
handy on a high-use database machine, where excessive disk writes
could have a significant impact on service delivery.

I doubt it would see very widely used, but it would make rsyslog
appealing to another niche.

-HKS

On Thu, Jul 17, 2008 at 6:43 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Hi all,
>
> I just got an idea for a feature inspired by the ramlog project. I have
> blogged about it:
>
> http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-buffer.h
> tml
>
> If you have some minutes to spare, please review and provide feedback.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
Feedback request: in-memory buffering messages [ In reply to ]
> I just got an idea for a feature inspired by the ramlog project. I have
> blogged about it:
>
> http://blog.gerhards.net/2008/07/writing-syslog-messages-to-ram-buffer.h
> tml

Prior art exists and I think it's well worth the effort, particularly
for busy servers on reliable power. If I read your post right, you're
considering something rather similar to the syslog-ng options
'flush_lines' and 'flush_timeout'?
Feedback request: in-memory buffering messages [ In reply to ]
> Prior art exists and I think it's well worth the effort, particularly
> for busy servers on reliable power. If I read your post right, you're
> considering something rather similar to the syslog-ng options
> 'flush_lines' and 'flush_timeout'?

I actually (really ;)) do not know syslog-ng and keep myself somewhat
away from it to prevent accidental steal of "whatever" from there. But
now that you raise the point: could you quickly describe how it works.
From your mail, it sounds like what I am thinking about...

Rainer
Feedback request: in-memory buffering messages [ In reply to ]
> I actually (really ;)) do not know syslog-ng and keep myself somewhat
> away from it to prevent accidental steal of "whatever" from there. But
> now that you raise the point: could you quickly describe how it works.
> From your mail, it sounds like what I am thinking about...

I 100% agree with and applaud that separation. As you've stated
before, they make a good product and don't warrant any interference.

Not touching the docs myself (since I know too much of it by heart
anyway), flush_lines allows the admin to configure a particular number
of log entries to buffer before forcing a flush to disk, whereas
flush_timeout configures the maximum interval between disk flushes.
Unlike ramlog, it seems to do this with internal allocations and
doesn't rely on a ramdisk. I usually set it rather conservatively (20
and 600 respectively), but I can definitely see being more aggressive
on a dedicated log collector with critical power or a laptop in
ultra-low power mode.

This has been a feature of the public version of syslog-ng for as long
as I can remember (or four years, whichever is sooner ;). Combined
with disk queues I can see a very nice tiered approach to handling
extremely high volumes of log data in a rather reliable manner.
Feedback request: in-memory buffering messages [ In reply to ]
On Thu, 2008-07-17 at 09:08 -0600, RB wrote:
> > I actually (really ;)) do not know syslog-ng and keep myself somewhat
> > away from it to prevent accidental steal of "whatever" from there. But
> > now that you raise the point: could you quickly describe how it works.
> > From your mail, it sounds like what I am thinking about...
>
> I 100% agree with and applaud that separation. As you've stated
> before, they make a good product and don't warrant any interference.

Definitely.

>
> Not touching the docs myself (since I know too much of it by heart
> anyway), flush_lines allows the admin to configure a particular number
> of log entries to buffer before forcing a flush to disk, whereas
> flush_timeout configures the maximum interval between disk flushes.
> Unlike ramlog, it seems to do this with internal allocations and
> doesn't rely on a ramdisk. I usually set it rather conservatively (20
> and 600 respectively), but I can definitely see being more aggressive
> on a dedicated log collector with critical power or a laptop in
> ultra-low power mode.

This begins to make more and more sense. Just one thing to make sure we
are talking along the same lines. The ramdisk approach provides a way to
save the IO while still being able to look at the log lines as fast as
they come in.

If, in rsyslog, I create a memory buffer, I can persist lines to disk
only after it is "time to write them to disk" (whatever that triggers).
So you will not be able to observe the messages while they stick inside
rsyslog's queue. I guess for many cases this is not really relevant. And
of course, it is something that must be configurable on an action basis,
so different files may use different settings.

Thinking more about a potential algorithm, I tend to think it would
probably useful to write log files in chunks of n bytes instead of n
lines. I am thinking along the lines of matching up the output buffer
with the disk sector size (or any multiple of it) so that partial writes
of the same sector are avoided. Of course, that involves some stating of
the file to obtain the size of the first buffer to be written (filesize
mod sector [allocation unit] size). I also requires me to find out the
allocation unit size for a given file system. It obviously involves
writing incomplete lines. It requires additional code. So the best
solution may actually be to permit partial writes (as initially thought)
but recommend large buffers. The overall effect could justify the
performance impact from doing non-optimal writes. But I am in too much
detail.

The actual questions are:

a) is it OK that log data is visible only after the (write) delay?
b) does it sound useful to buffer based on allocation unit sizes?

Thanks,
Rainer
>
> This has been a feature of the public version of syslog-ng for as long
> as I can remember (or four years, whichever is sooner ;). Combined
> with disk queues I can see a very nice tiered approach to handling
> extremely high volumes of log data in a rather reliable manner.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
Feedback request: in-memory buffering messages [ In reply to ]
Hi all,

[ big snip ]

>
> The actual questions are:
>
> a) is it OK that log data is visible only after the (write) delay?

Hmmm.. no.
This would eliminate the "magic" of watching system events in real-time
as in the commonly used 'tail -n0 -f /var/log/messages' which is often
used to troubleshoot problems...

Unless you provide some tools to implement such functionality by
watching the log events in memory... huh...


My 2 cents...

Martin




> b) does it sound useful to buffer based on allocation unit sizes?
>
> Thanks,
> Rainer
> >
> > This has been a feature of the public version of syslog-ng for as
long
> > as I can remember (or four years, whichever is sooner ;). Combined
> > with disk queues I can see a very nice tiered approach to handling
> > extremely high volumes of log data in a rather reliable manner.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
Feedback request: in-memory buffering messages [ In reply to ]
<snip>

> The actual questions are:
>
> a) is it OK that log data is visible only after the (write) delay?
> b) does it sound useful to buffer based on allocation unit sizes?


a) It depends on the situation. If you're using this to conserve power
or flash-disk writes, then it's going to be a pain. If you're using
this to tune performance in a high log-volume environment, then this
is an acceptable tradeoff. Perhaps a secondary script that would send
a signal to rsyslogd and print the lines to STDOUT would be an
acceptable compromise?

b) It's more granular than many (most?) users will need, including
myself, but I'm sure some people will find this very useful.

-HKS
Feedback request: in-memory buffering messages [ In reply to ]
> a) is it OK that log data is visible only after the (write) delay?
For my purposes it is; I know some of the other responses have been
that it's not, but perhaps setting up a signal (maybe USR2) for
triggering the flush mechanism would help those situations. Of
course, people wanting "real-time" data would probably either leave
off the configurations (knowing they're doing so at the expense of
performance) or change it at runtime and HUP.

> b) does it sound useful to buffer based on allocation unit sizes?
I had actually considered suggesting this as a third option
(differentiating from other implementations), but abandoned it since I
was unsure of the value and felt the implementation may be overly
complex.

If you implement byte/allocation-aligned flushes, my preference would
be to have it in addition to but mutually (configuration) exclusive
with record-aligned ones. Partial-sector writes are less of a concern
to me in most cases than getting the whole message to disk, but I can
again see the usefulness of allocation alignment in a very high-volume
environment or one where disk performance is constrained. Regardless,
it's a fascinating development and one that will come in very handy.