Mailing List Archive

Advice on embedded rsyslog use
I'd like to get some advice on how best to add some functionality to
rsyslog.

I'm planning to use rsyslog in an embedded environment (OpenWRT), so
resources are not abundant. It's a flash filesystem too, and not
high-endurance flash, so I'd like to minimize the volume of persistent
writes.

As always, there's a tension between capturing enough detail to diagnose a
problem from the logs alone, and the size of the logs. The bulk of the
lower priority level logging never gets looked at, but the subset
immediately before and after some significant event (presumably logged at a
level of 'error' or above) is pure gold - the proverbial smoke from the gun.

What I'm looking to do is to divert the volume of low-priority log messages
to a circular buffer in ram, only passing through the highest-priority ones
to the destination. When one of the high-priority messages is seen, also
dump what's in the circular buffer to the log destination too - it'll be
out-of-order, but the timestamps so allow us to sort that out. Then
continue to pass through lower-priority messages to the destination for
some time period/count of messages, then revert back to shunting the
lower-priority messages to the circular buffer.

I'm thinking that filtering can be used to trigger an action that sends
those messages to the circular buffer. Other filters can be used to 'dump'
the contents of the circular buffer into an input module, that 're-injects'
those past messages back into rsyslog, though a different ruleset (which
may be empty). For the 'post-trigger event' period/message count, the
action just forwards the message directly to the input module, bypassing
the circular buffer (but not the second rule set).

A nice-to-have would be to be able to configure more than one circular
buffer, each with its own instance of the input module/ruleset. I'm
anticipating situations where we're hot on the trail of some
hard-to-reproduce bug, and have narrowed down which log messages are of
interest, so want to capture as many of those as possible.

I'm new to rsyslog (though not embedded Linux) and have never written a
rsyslog module, so sage advice from those that have would be most valuable.

Many thanks,

- Paul
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Advice on embedded rsyslog use [ In reply to ]
I don't think this is a good fit for rsyslog. Rsyslog handles every message
independently, and may handle them in parallel. This makes it a really bad fit
for maintaining a buffer and then doing something with it.

I am not disagreeing with the value of your request at all, just saying that
rsyslog may not be the best starting point (it has so much functionality that
you don't want or need)

I would look at creating a minimal syslog daemon that maintains the buffer you
are talking about and the rules for triggering a buffer dump, and then when the
dump happens, feed the logs into a separate instance of rsyslog (or other syslog
daemon) so that you don't have to write all the fancy functionality that they
provide (encryption, network protocols and failure handling, etc)

I would consider making your daemon be in-memory only for simplicity, don't even
worry about writing to files, just figure out the input, buffering, and
triggering problems.

David Lang

On Wed, 24 Nov 2021,
Paul Chambers via rsyslog wrote:

> Date: Wed, 24 Nov 2021 18:00:31 -0800
> From: Paul Chambers via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog <rsyslog@lists.adiscon.com>
> Cc: Paul Chambers <bod@bod.org>
> Subject: [rsyslog] Advice on embedded rsyslog use
>
> I'd like to get some advice on how best to add some functionality to
> rsyslog.
>
> I'm planning to use rsyslog in an embedded environment (OpenWRT), so
> resources are not abundant. It's a flash filesystem too, and not
> high-endurance flash, so I'd like to minimize the volume of persistent
> writes.
>
> As always, there's a tension between capturing enough detail to diagnose a
> problem from the logs alone, and the size of the logs. The bulk of the
> lower priority level logging never gets looked at, but the subset
> immediately before and after some significant event (presumably logged at a
> level of 'error' or above) is pure gold - the proverbial smoke from the gun.
>
> What I'm looking to do is to divert the volume of low-priority log messages
> to a circular buffer in ram, only passing through the highest-priority ones
> to the destination. When one of the high-priority messages is seen, also
> dump what's in the circular buffer to the log destination too - it'll be
> out-of-order, but the timestamps so allow us to sort that out. Then
> continue to pass through lower-priority messages to the destination for
> some time period/count of messages, then revert back to shunting the
> lower-priority messages to the circular buffer.
>
> I'm thinking that filtering can be used to trigger an action that sends
> those messages to the circular buffer. Other filters can be used to 'dump'
> the contents of the circular buffer into an input module, that 're-injects'
> those past messages back into rsyslog, though a different ruleset (which
> may be empty). For the 'post-trigger event' period/message count, the
> action just forwards the message directly to the input module, bypassing
> the circular buffer (but not the second rule set).
>
> A nice-to-have would be to be able to configure more than one circular
> buffer, each with its own instance of the input module/ruleset. I'm
> anticipating situations where we're hot on the trail of some
> hard-to-reproduce bug, and have narrowed down which log messages are of
> interest, so want to capture as many of those as possible.
>
> I'm new to rsyslog (though not embedded Linux) and have never written a
> rsyslog module, so sage advice from those that have would be most valuable.
>
> Many thanks,
>
> - Paul
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Advice on embedded rsyslog use [ In reply to ]
Thanks, David. Yes, it's definitely overkill for what I need. Trouble is,
I'm finding very little between the stripped-out 'logd' that OpenWRT
provides by default and the various more enterprise-oriented syslog
daemons. I've looked.at half a dozen already.

Oh well, probably better to extend logd than to fight the nature of
software designed for a completely different use case.

- Paul

On Thu, Nov 25, 2021 at 2:31 AM David Lang <david@lang.hm> wrote:

> I don't think this is a good fit for rsyslog. Rsyslog handles every
> message
> independently, and may handle them in parallel. This makes it a really bad
> fit
> for maintaining a buffer and then doing something with it.
>
> I am not disagreeing with the value of your request at all, just saying
> that
> rsyslog may not be the best starting point (it has so much functionality
> that
> you don't want or need)
>
> I would look at creating a minimal syslog daemon that maintains the buffer
> you
> are talking about and the rules for triggering a buffer dump, and then
> when the
> dump happens, feed the logs into a separate instance of rsyslog (or other
> syslog
> daemon) so that you don't have to write all the fancy functionality that
> they
> provide (encryption, network protocols and failure handling, etc)
>
> I would consider making your daemon be in-memory only for simplicity,
> don't even
> worry about writing to files, just figure out the input, buffering, and
> triggering problems.
>
> David Lang
>
> On Wed, 24 Nov 2021,
> Paul Chambers via rsyslog wrote:
>
> > Date: Wed, 24 Nov 2021 18:00:31 -0800
> > From: Paul Chambers via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog <rsyslog@lists.adiscon.com>
> > Cc: Paul Chambers <bod@bod.org>
> > Subject: [rsyslog] Advice on embedded rsyslog use
> >
> > I'd like to get some advice on how best to add some functionality to
> > rsyslog.
> >
> > I'm planning to use rsyslog in an embedded environment (OpenWRT), so
> > resources are not abundant. It's a flash filesystem too, and not
> > high-endurance flash, so I'd like to minimize the volume of persistent
> > writes.
> >
> > As always, there's a tension between capturing enough detail to diagnose
> a
> > problem from the logs alone, and the size of the logs. The bulk of the
> > lower priority level logging never gets looked at, but the subset
> > immediately before and after some significant event (presumably logged
> at a
> > level of 'error' or above) is pure gold - the proverbial smoke from the
> gun.
> >
> > What I'm looking to do is to divert the volume of low-priority log
> messages
> > to a circular buffer in ram, only passing through the highest-priority
> ones
> > to the destination. When one of the high-priority messages is seen, also
> > dump what's in the circular buffer to the log destination too - it'll be
> > out-of-order, but the timestamps so allow us to sort that out. Then
> > continue to pass through lower-priority messages to the destination for
> > some time period/count of messages, then revert back to shunting the
> > lower-priority messages to the circular buffer.
> >
> > I'm thinking that filtering can be used to trigger an action that sends
> > those messages to the circular buffer. Other filters can be used to
> 'dump'
> > the contents of the circular buffer into an input module, that
> 're-injects'
> > those past messages back into rsyslog, though a different ruleset (which
> > may be empty). For the 'post-trigger event' period/message count, the
> > action just forwards the message directly to the input module, bypassing
> > the circular buffer (but not the second rule set).
> >
> > A nice-to-have would be to be able to configure more than one circular
> > buffer, each with its own instance of the input module/ruleset. I'm
> > anticipating situations where we're hot on the trail of some
> > hard-to-reproduce bug, and have narrowed down which log messages are of
> > interest, so want to capture as many of those as possible.
> >
> > I'm new to rsyslog (though not embedded Linux) and have never written a
> > rsyslog module, so sage advice from those that have would be most
> valuable.
> >
> > Many thanks,
> >
> > - Paul
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.