Mailing List Archive

Hi Luis Fernando,

glad you asked :) The documentation is ... well... not much ;) The best
thing is probably to start with the existing MySQL module. HOWEVER, I
myself would be very interested in a native, high-performing Oracle
driver. I neither have the expertise to do it not the test environment.

If you like, we could collaborate on this effort. I'd create a skeleton
output module for you, guide you through using it and you provide the
Oracle bits to it (that should be fairly easy). Of course, that means
your module would need to be contributed back to the project.

What do you think about this?

Rainer


On Wed, 2009-02-11 at 14:59 +0100, Luis Fernando Muñoz Mejías wrote:
> Hello, world!
>
> I'm trying to store my logs centrally on an Oracle database. I need it
> to be Oracle.
>
> Due to some restrictions around there, I can't use the recommended
> version of libdbi, but an older one that is not working with rsyslog. I
> can't rebuild that RPM, and I have no influence on the upgrade process
> or schedule.
>
> Finally, I also need something with *excellent* performance, and the
> documentation states omlibdbi doesn't perform as well as other
> DB-related modules, so I decided to try my own small set of input and
> output modules.
>
> So, my question is: is there any documentation or guide for writing
> modules, something that gives some insight of what all those macros
> (BEGINcreateInstance, CODESTARTcreateInstance, etc, etc, etc) I see
> mean? Do I need to grep the code for that?
>
> I've googled for it and reviewed the Git repository, but found nothing.
>
> Thanks in advance.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Wed, 11 Feb 2009, Luis Fernando Mu?oz Mej?as wrote:

> Finally, I also need something with *excellent* performance, and the
> documentation states omlibdbi doesn't perform as well as other
> DB-related modules, so I decided to try my own small set of input and
> output modules.

one major limitation you will run into is that rsyslog proceses log
entries one at a time. this means that each log entry inserted into the
oracle database will be a seperate transaction.

changing this is a fairly major task that will require someone to sponser
it (I've asked about this in the past, but my company has not needed the
performance enough to do this yet)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Rainer,

My apologies for the late reply. I was subscribed to the digest format
and didn't receive your replies.

> glad you asked :) The documentation is ... well... not much ;)

Oops!

> The best thing is probably to start with the existing MySQL
> module. HOWEVER, I myself would be very interested in a native,
> high-performing Oracle driver. I neither have the expertise to do it
> not the test environment.

I don't have the expertise either, but do have the test
environment... we can try. ;)

> If you like, we could collaborate on this effort. I'd create a
> skeleton output module for you, guide you through using it and you
> provide the Oracle bits to it (that should be fairly easy). Of course,
> that means your module would need to be contributed back to the
> project.

That sounds really great. Before you start coding or preparing anything,
let me check how well our DBs perform, because it's not yet clear if
they'll be able to cope with the high insertion rate we expect. If we
don't go for the Oracle database this work doesn't make sense. I bet
we'll want the Oracle, anyways.

For this evaluation, I already have a timestamp formatter that fits into
Oracle, something that can be used with the property replacer, like
%timereported:::date-oracle%. It still needs some real-world testing,
but provided it works, is it interesting for the project? If so, should
I submit the patch via bugzilla? Is there any paperwork (copyright
assingmnents a la FSF or whatever) that should be fullfilled?

Thanks.
--
Luis Fernando Muñoz Mejías
Luis.Fernando.Munoz.Mejias@cern.ch

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Sorry, this time my reply is sluggish ;)

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz Mejías
> Sent: Monday, February 16, 2009 11:41 AM
> To: rsyslog@lists.adiscon.com
> Subject: Re: [rsyslog] Documentation on writing rsyslog modules?
>
> Rainer,
>
> My apologies for the late reply. I was subscribed to the digest format
> and didn't receive your replies.
>
> > glad you asked :) The documentation is ... well... not much ;)
>
> Oops!
>
> > The best thing is probably to start with the existing MySQL
> > module. HOWEVER, I myself would be very interested in a native,
> > high-performing Oracle driver. I neither have the expertise to do it
> > not the test environment.
>
> I don't have the expertise either, but do have the test
> environment... we can try. ;)

A test environment is first step to expertise ;)

>
> > If you like, we could collaborate on this effort. I'd create a
> > skeleton output module for you, guide you through using it and you
> > provide the Oracle bits to it (that should be fairly easy). Of
> course,
> > that means your module would need to be contributed back to the
> > project.
>
> That sounds really great. Before you start coding or preparing
> anything,
> let me check how well our DBs perform, because it's not yet clear if
> they'll be able to cope with the high insertion rate we expect. If we
> don't go for the Oracle database this work doesn't make sense. I bet
> we'll want the Oracle, anyways.

Sounds fair.

>
> For this evaluation, I already have a timestamp formatter that fits
> into
> Oracle, something that can be used with the property replacer, like
> %timereported:::date-oracle%.

Sounds good. This is one of the bad things about current code base, though. The formatter should long come from the custom plugin, but I didn't manage to do the script engine so far (the core of custom functions). Not a big deal, but something that annoys *me* ;)

> It still needs some real-world testing,
> but provided it works, is it interesting for the project?

Definitely

> If so, should
> I submit the patch via bugzilla?

Any way is fine. You can also just email me. Proper credits, of course are guaranteed.

> Is there any paperwork (copyright
> assingmnents a la FSF or whatever) that should be fullfilled?

Nope, we think the project is in a strong enough position even if the submitter holds the copyright. Actually, this was one project goal: keep contributions easy. I really don't like all the blabla that e.g. you need to do when contributing to syslog-ng. Just be aware that some parts of rsyslog come under GPL while some come under LGPL (the runtime files). We assume that whatever you contribute comes under the license of the core module. For new runtime files, this means we expect LGPL, else we have a problem license-wise.

Really looking forward to your results,
Rainer
>
> Thanks.
> --
> Luis Fernando Muñoz Mejías
> Luis.Fernando.Munoz.Mejias@cern.ch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Rainer,

Good and bad news...

> > That sounds really great. Before you start coding or preparing
> > anything, let me check how well our DBs perform, because it's not
> > yet clear if they'll be able to cope with the high insertion rate we
> > expect. If we don't go for the Oracle database this work doesn't
> > make sense. I bet we'll want the Oracle, anyways.
>
> Sounds fair.

Good news: I did my tests and, for many tasks I need to do, Oracle is
our way to go. So, I'm willing to write the module, with your
guidance/advise.

As I said, I need **excellent** performance. I definitely need batch
operations, the ability to prepare the statements given as arguments on
the configuration file, and not to commit entries one by one, but after
a number of entries are ready or (better) after some not so small
time. According to the advise I got from experts around here, I'll have
to use Oracle Call Interface for this module, I don't know if there are
any licensing issues.

It seems I'll have to review how rsyslog's queing modules work...

> > For this evaluation, I already have a timestamp formatter that fits
> > into Oracle, something that can be used with the property replacer,
> > like %timereported:::date-oracle%.
>
The bad news is that this timestamp formatter works perfectly on
interactive sessions (sqlplus) but not on non-interactive ones, f.i, in
Python scripts. You need to call Oracle's to_timestamp(string, format),
and by bloating your code with this ugly function the rfc-3339 formatter
is good enough. So I won't submit this one.

Cheers.
--
Luis Fernando Muñoz Mejías
Luis.Fernando.Munoz.Mejias@cern.ch

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Fri, 2009-02-27 at 18:48 +0100, Luis Fernando Muñoz Mejías wrote:
> Rainer,
>
> Good and bad news...
>
> > > That sounds really great. Before you start coding or preparing
> > > anything, let me check how well our DBs perform, because it's not
> > > yet clear if they'll be able to cope with the high insertion rate we
> > > expect. If we don't go for the Oracle database this work doesn't
> > > make sense. I bet we'll want the Oracle, anyways.
> >
> > Sounds fair.
>
> Good news: I did my tests and, for many tasks I need to do, Oracle is
> our way to go. So, I'm willing to write the module, with your
> guidance/advise.
>
so far this sounds good ;)

> As I said, I need **excellent** performance. I definitely need batch
> operations, the ability to prepare the statements given as arguments on
> the configuration file, and not to commit entries one by one, but after
> a number of entries are ready or (better) after some not so small
> time. According to the advise I got from experts around here, I'll have
> to use Oracle Call Interface for this module, I don't know if there are
> any licensing issues.

I can't comment on the licensing issue, I simply don't know what Oracle
demands. On thing to do it is let the output module handle the
"combination work" together. The output module is called one per
message, however, it does not mean the output must directly write them
to the database. It may buffer them until the batch is large enough. But
this currently needs to be implemented on the output module basis.
Obviously, that will not make coding simpler.

If we find a sponsor for the necessary non-trivial extension of the core
engine, the output module's task may become much easier. If things go
well, such a sponsor may show up...

>
> It seems I'll have to review how rsyslog's queing modules work...

I would suggest not to move into them - but, of course, if you like
to... Lol, this is the non-trival task I talked about, there are
numerous subtleties and, of course, they are weakly documented (but the
inline doc is quite good).
>
> > > For this evaluation, I already have a timestamp formatter that fits
> > > into Oracle, something that can be used with the property replacer,
> > > like %timereported:::date-oracle%.
> >
> The bad news is that this timestamp formatter works perfectly on
> interactive sessions (sqlplus) but not on non-interactive ones, f.i, in
> Python scripts. You need to call Oracle's to_timestamp(string, format),
> and by bloating your code with this ugly function the rfc-3339 formatter
> is good enough. So I won't submit this one.
>
Sounds fair ;)

Do you have a time frame for your project? (and maybe a rough overview
of the "big picture" - I am always soooo curios ;))

Rainer
> Cheers.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Just one quick note, more following:

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz Mejías
> Sent: Tuesday, March 03, 2009 3:29 PM
> To: rsyslog@lists.adiscon.com
> Subject: Re: [rsyslog] Documentation on writing rsyslog modules?
>
> Hi there.
>
> > > As I said, I need **excellent** performance. I definitely need
> batch
> > > operations, the ability to prepare the statements given as
> arguments
> > > on the configuration file, and not to commit entries one by one,
> but
> > > after a number of entries are ready or (better) after some not so
> > > small time. According to the advise I got from experts around here,
> > > I'll have to use Oracle Call Interface for this module, I don't
> know
> > > if there are any licensing issues.
> >
> > I can't comment on the licensing issue, I simply don't know what
> > Oracle demands.
>
> I'm not sure how GPL-compatible it is to link to already existing
> proprietary code. Anyways, first I code, then we test, then we (you,
> actually) decide the legal aspects.

Actually, not me ;) I leave this risk to the user. If someone pays the legal counselor, I'll add his POV to the project doc.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Hi there.

> > As I said, I need **excellent** performance. I definitely need batch
> > operations, the ability to prepare the statements given as arguments
> > on the configuration file, and not to commit entries one by one, but
> > after a number of entries are ready or (better) after some not so
> > small time. According to the advise I got from experts around here,
> > I'll have to use Oracle Call Interface for this module, I don't know
> > if there are any licensing issues.
>
> I can't comment on the licensing issue, I simply don't know what
> Oracle demands.

I'm not sure how GPL-compatible it is to link to already existing
proprietary code. Anyways, first I code, then we test, then we (you,
actually) decide the legal aspects.

> On thing to do it is let the output module handle the "combination
> work" together. The output module is called one per message, however,
> it does not mean the output must directly write them to the
> database. It may buffer them until the batch is large enough. But this
> currently needs to be implemented on the output module basis.
> Obviously, that will not make coding simpler.

That's what I expected, indeed.


> > It seems I'll have to review how rsyslog's queing modules work...
>
> I would suggest not to move into them - but, of course, if you like
> to... Lol, this is the non-trival task I talked about, there are
> numerous subtleties and, of course, they are weakly documented (but
> the inline doc is quite good).

OK. I'll just have a buffer of entries to be committed.

> Do you have a time frame for your project? (and maybe a rough overview
> of the "big picture" - I am always soooo curios ;))

Not a full timescale. Let's say that as soon as you can provide me with
the documentation/skeleton module most (say 70%) of my work will be
developing this output module. Then, when I understand what a bad
nightmare OCI is I'll be able to give a full timescale. After looking at
ompgsql, it looks like writing output modules is easy if you know what
you're doing. ;)

Then, I'll be able to provide support for this module (fixing bugs and
so on) for a couple of years, so it won't be shoot and forget.

Cheers.
--
Luis Fernando Muñoz Mejías
Luis.Fernando.Munoz.Mejias@cern.ch

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Sorry for the delay, it is currently quite busy here at my end :(

I have now created a very rough skeleton template output module. You need to
pull from git. It is contained in the master branch.

So far, it does not perform useful work. I was a bit hesitant to add much
more description, because I think this can either be brief and not matching
what you need - or very elaborate (bbok-like), for what I currently do not
have enough time.

I suggest that you have a look at the template module, and then we simply try
to get this going. It would be good if you could ask questions or tell me
what needs to be placed inside the module. Or I can create yet another
skeleton, based on ommysql, that has a bit more logic so that you can fill in
the initial Oracle functionality. That will not offer superior performance,
but I think it would be a good starting point to pursue the rest of this
project.

Please let me know what you think.

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, March 03, 2009 3:26 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Documentation on writing rsyslog modules?
>
> Just one quick note, more following:
>
> > -----Original Message-----
> > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz Mejías
> > Sent: Tuesday, March 03, 2009 3:29 PM
> > To: rsyslog@lists.adiscon.com
> > Subject: Re: [rsyslog] Documentation on writing rsyslog modules?
> >
> > Hi there.
> >
> > > > As I said, I need **excellent** performance. I definitely need
> > batch
> > > > operations, the ability to prepare the statements given as
> > arguments
> > > > on the configuration file, and not to commit entries one by one,
> > but
> > > > after a number of entries are ready or (better) after some not so
> > > > small time. According to the advise I got from experts around
> here,
> > > > I'll have to use Oracle Call Interface for this module, I don't
> > know
> > > > if there are any licensing issues.
> > >
> > > I can't comment on the licensing issue, I simply don't know what
> > > Oracle demands.
> >
> > I'm not sure how GPL-compatible it is to link to already existing
> > proprietary code. Anyways, first I code, then we test, then we (you,
> > actually) decide the legal aspects.
>
> Actually, not me ;) I leave this risk to the user. If someone pays the
> legal counselor, I'll add his POV to the project doc.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

El Lunes, 16 de Marzo de 2009 17:53, Rainer Gerhards escribió:
> Please let me know what you think.

I just came back from a week of holidays, I'm reviewing the skeleton,
which looks pretty comprehensive. Thanks!
--
Luis Fernando Muñoz Mejías
Luis.Fernando.Munoz.Mejias@cern.ch

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

When you are ready, I'd actually suggest that I create an "omoracle" git
branch for you and do place a copy of ommysql into it. This, together with
the comments from omtemplate, would probably one way to get a (non-optimal)
quick start.

I would suggest that we build a very basic oracle driver first and after we
see it works well, then look into the performance optimization.

Let me know what you think (and when you have time). I could actually create
what I have proposed with little delay once you give a go.

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz Mejías
> Sent: Wednesday, March 18, 2009 10:54 AM
> To: rsyslog@lists.adiscon.com
> Subject: Re: [rsyslog] Documentation on writing rsyslog modules?
>
> El Lunes, 16 de Marzo de 2009 17:53, Rainer Gerhards escribió:
> > Please let me know what you think.
>
> I just came back from a week of holidays, I'm reviewing the skeleton,
> which looks pretty comprehensive. Thanks!
> --
> Luis Fernando Muñoz Mejías
> Luis.Fernando.Munoz.Mejias@cern.ch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com