On Fri, 2008-04-11 at 16:10 -0600, RB wrote:
> Read once, digested for a few hours, read again. *whew* A monolithic
> message about a monolithic problem... ;) Also please understand I'm
> new here and am not completely familiar with your codebase, so may
> well miss the mark.
>
> Do I understand you correctly that part of your concern with looser
> coupling is the higher potential for lossiness?
Yes, there are two issues with that: increased lossiness (due to less
knowledge inside rsyslog core of what happens & the same problem plain
tcp has) and performance. A pipe interface is much slower than the
native in-process plugin interface - this can turn out to become a
problem in high performance big enterprise deployments.
HOWEVER, all of this can be solved if the module can be compiled to
utilize either one of both interfaces. I have also thought more about
this and it looks like the absolutely perfect solution to all needs (the
bottom line is: the user decides what to use and has all options at
hand).
>
> I don't think I can afford a line-by-line analysis at the moment, but
> got a persistent thought both times through: if you're considering
> pursuing a loosely-coupled model, why not consider going all the way
> and treat rsyslogd as a dispatch/routing agent among the various
> modules?
Actually, this is the approach rsyslog takes. Though this page is a bit
old (and generic), it still covers the base ideas:
http://www.rsyslog.com/doc-generic_design.html
This document was originally written for an IETF discussion, so it is
not rsyslog-specific (but given the same authorship... ;)).
> That's by no means a simple solution, but I wonder why
> certain pieces of functionality must be tightly coupled, either to
> each other (network/crypt) or to the core itself (network). I *think*
> it becomes even more interesting when you start thinking about ways to
> 'stack' modules.
The most recent rsyslog loader supports stacking. We now have a call
where an object can request it to find other objects, which can lead to
an object load request if the object is not yet in core. Of course, if a
new object is loaded, it can request the loader to find other objects it
needs... Run a recent build of rsyslogd with -d -n and you'll see these
calls in action ;) However, not all of the rsyslog modules are full
blown objects as of now. Converting them is part of the ongoing
modularization effort.
But this also touches the core of my long message: the loader and all
helpers currently live in rsyslogd core ONLY. They are not usable by any
entity that is not either part of rsyslog or being *called by rsyslog*
(plugins). So it is currently impossible to use any of rsyslog's
technology without rsyslogd. Among others, this prevents loosely
coupling modules (because the module would need access to the runtime
objects without rsyslog). It also prevents cases where someone wants to
use parts that depend on rsyslog technology, but without using rsyslogd
itself. While, of course, I do not find that smart, it may be just the
right thing for the user in question. I intend to no longer limit the
user in his options, freeing the code base even more.
This is both a technical issue as well as a licensing one. The solution,
the more I think about it, is really obvious: create a runtime system
and place it under LGPL. That'll solve all issues and make rsyslog
modular not only on a detail level but also from the high-level
perspective.
Rainer
> Read once, digested for a few hours, read again. *whew* A monolithic
> message about a monolithic problem... ;) Also please understand I'm
> new here and am not completely familiar with your codebase, so may
> well miss the mark.
>
> Do I understand you correctly that part of your concern with looser
> coupling is the higher potential for lossiness?
Yes, there are two issues with that: increased lossiness (due to less
knowledge inside rsyslog core of what happens & the same problem plain
tcp has) and performance. A pipe interface is much slower than the
native in-process plugin interface - this can turn out to become a
problem in high performance big enterprise deployments.
HOWEVER, all of this can be solved if the module can be compiled to
utilize either one of both interfaces. I have also thought more about
this and it looks like the absolutely perfect solution to all needs (the
bottom line is: the user decides what to use and has all options at
hand).
>
> I don't think I can afford a line-by-line analysis at the moment, but
> got a persistent thought both times through: if you're considering
> pursuing a loosely-coupled model, why not consider going all the way
> and treat rsyslogd as a dispatch/routing agent among the various
> modules?
Actually, this is the approach rsyslog takes. Though this page is a bit
old (and generic), it still covers the base ideas:
http://www.rsyslog.com/doc-generic_design.html
This document was originally written for an IETF discussion, so it is
not rsyslog-specific (but given the same authorship... ;)).
> That's by no means a simple solution, but I wonder why
> certain pieces of functionality must be tightly coupled, either to
> each other (network/crypt) or to the core itself (network). I *think*
> it becomes even more interesting when you start thinking about ways to
> 'stack' modules.
The most recent rsyslog loader supports stacking. We now have a call
where an object can request it to find other objects, which can lead to
an object load request if the object is not yet in core. Of course, if a
new object is loaded, it can request the loader to find other objects it
needs... Run a recent build of rsyslogd with -d -n and you'll see these
calls in action ;) However, not all of the rsyslog modules are full
blown objects as of now. Converting them is part of the ongoing
modularization effort.
But this also touches the core of my long message: the loader and all
helpers currently live in rsyslogd core ONLY. They are not usable by any
entity that is not either part of rsyslog or being *called by rsyslog*
(plugins). So it is currently impossible to use any of rsyslog's
technology without rsyslogd. Among others, this prevents loosely
coupling modules (because the module would need access to the runtime
objects without rsyslog). It also prevents cases where someone wants to
use parts that depend on rsyslog technology, but without using rsyslogd
itself. While, of course, I do not find that smart, it may be just the
right thing for the user in question. I intend to no longer limit the
user in his options, freeing the code base even more.
This is both a technical issue as well as a licensing one. The solution,
the more I think about it, is really obvious: create a runtime system
and place it under LGPL. That'll solve all issues and make rsyslog
modular not only on a detail level but also from the high-level
perspective.
Rainer