> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz MejÃas
> Sent: Wednesday, April 01, 2009 6:02 PM
> To: rsyslog-users
> Subject: [rsyslog] RFC: On rsyslog output modules and support for
> batchoperations
>
> Hello, world.
>
> I discussed this in private with Rainer, and he suggested me to bring
> the discussion here.
>
> I'm already developing an output module for feeding an Oracle database
> with rsyslog input. Rainer already committed some patches to the
> "oracle" branch, in git. Let me remember that this is highly
> experimental, and I'm sending a big semantic change today. But, in
> principle, the module does what you'd expect from it: it connects to a
> DB, receives a SQL statement via doAction, prepares that statement,
> runs
> it, commits.
If I didn't screw up, everything should be committed now.
> It works, but it's way too slow for my needs. As I said when I started
> this project, I need to be very fast, to prepare the statement at
> connection time, run it many times, and definitely want batch
> operations. Say, I want to insert 1000 entries with a single call to
> the
> Oracle interface, then commit.
>
> With what I know now of rsyslog, I can do it more or less like this:
>
> $OmoracleStatementTemplate,"insert into foo(field1, field2, field3)
> values(:val1, :val2, :val3)"
>
> which is the statement to prepare by Oracle. This way, I can prepare
> the
> statement at createInstance() time. Then, I can specify the batch size
> with something like
>
> $OmoracleBatchSize 1000
>
> With this, also at createInstance() time I can specify that doAction is
> called only if there are 1000 entries pending for this selector, like
> this:
>
> CODE_STD_STRING_REQUESTparseSelectorAct(batch_size);
>
> The bad part is that rsyslog will deliver to the output module a single
> string per entry. So, I'd have to split each entry into its fields as
> part of the doAction() code. I'd need some funny separator for each
> field, to avoid problems. So far, it can be done. But the configuration
> would look like this:
>
> $OmoracleDB logdb
> $OmoracleDBUser dbuser
> $OmoracleDBPassword dbpassword
> $OmoracleStatement "insert into foo(col1, col2) values (:fied1,
> :field2)"
> $OmoracleBatchSize 1000
> $OmoracleFieldSeparator ****
>
> *.* :omoracle:;"%field1%****%field2%"
>
> and make doAction split the fields appropriately.
There are a couple of subtleties, but I think it can work. In essence, you
need a template that feeds into the values via ($template!) and also a config
string for the prepared statement. It's actually not even that hard to do. It
may be useful (and of course doable) to enable the property replace to escape
special characters, so that, for example, we could use CSV and replace commas
by two of them.
>
> I bet it works. But it's probably too ugly for users. Cleaner ways may
> need deeper changes into rsyslog's API so that the module gets direct
> access to each field. That's probably a lot of work and I can't wait
> for
> that.
I need to check if there are actually larger changes required. The main
reason for this interface initially was security (do not pass to the module
the full object). Assuming that I have the object available at the time of
the plugin call, I could use a different entry point to pass that data in. If
so, that would not be too much effort. Security concerns could be (somewhat)
addressed by a config statement which enables such object access for the next
action, so one could specifically grant that privilege.
What is the overall opinion on this list? Should we look further into that
direction?
Rainer
>
> So, my questions (at last!): Are there any other alternatives? Is this
> "ugly" way of working good for other users? Should I keep it for
> internal use?
>
> Thanks a lot.
> --
> Luis Fernando Muñoz MejÃas
> Luis.Fernando.Munoz.Mejias@cern.ch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Luis Fernando Muñoz MejÃas
> Sent: Wednesday, April 01, 2009 6:02 PM
> To: rsyslog-users
> Subject: [rsyslog] RFC: On rsyslog output modules and support for
> batchoperations
>
> Hello, world.
>
> I discussed this in private with Rainer, and he suggested me to bring
> the discussion here.
>
> I'm already developing an output module for feeding an Oracle database
> with rsyslog input. Rainer already committed some patches to the
> "oracle" branch, in git. Let me remember that this is highly
> experimental, and I'm sending a big semantic change today. But, in
> principle, the module does what you'd expect from it: it connects to a
> DB, receives a SQL statement via doAction, prepares that statement,
> runs
> it, commits.
If I didn't screw up, everything should be committed now.
> It works, but it's way too slow for my needs. As I said when I started
> this project, I need to be very fast, to prepare the statement at
> connection time, run it many times, and definitely want batch
> operations. Say, I want to insert 1000 entries with a single call to
> the
> Oracle interface, then commit.
>
> With what I know now of rsyslog, I can do it more or less like this:
>
> $OmoracleStatementTemplate,"insert into foo(field1, field2, field3)
> values(:val1, :val2, :val3)"
>
> which is the statement to prepare by Oracle. This way, I can prepare
> the
> statement at createInstance() time. Then, I can specify the batch size
> with something like
>
> $OmoracleBatchSize 1000
>
> With this, also at createInstance() time I can specify that doAction is
> called only if there are 1000 entries pending for this selector, like
> this:
>
> CODE_STD_STRING_REQUESTparseSelectorAct(batch_size);
>
> The bad part is that rsyslog will deliver to the output module a single
> string per entry. So, I'd have to split each entry into its fields as
> part of the doAction() code. I'd need some funny separator for each
> field, to avoid problems. So far, it can be done. But the configuration
> would look like this:
>
> $OmoracleDB logdb
> $OmoracleDBUser dbuser
> $OmoracleDBPassword dbpassword
> $OmoracleStatement "insert into foo(col1, col2) values (:fied1,
> :field2)"
> $OmoracleBatchSize 1000
> $OmoracleFieldSeparator ****
>
> *.* :omoracle:;"%field1%****%field2%"
>
> and make doAction split the fields appropriately.
There are a couple of subtleties, but I think it can work. In essence, you
need a template that feeds into the values via ($template!) and also a config
string for the prepared statement. It's actually not even that hard to do. It
may be useful (and of course doable) to enable the property replace to escape
special characters, so that, for example, we could use CSV and replace commas
by two of them.
>
> I bet it works. But it's probably too ugly for users. Cleaner ways may
> need deeper changes into rsyslog's API so that the module gets direct
> access to each field. That's probably a lot of work and I can't wait
> for
> that.
I need to check if there are actually larger changes required. The main
reason for this interface initially was security (do not pass to the module
the full object). Assuming that I have the object available at the time of
the plugin call, I could use a different entry point to pass that data in. If
so, that would not be too much effort. Security concerns could be (somewhat)
addressed by a config statement which enables such object access for the next
action, so one could specifically grant that privilege.
What is the overall opinion on this list? Should we look further into that
direction?
Rainer
>
> So, my questions (at last!): Are there any other alternatives? Is this
> "ugly" way of working good for other users? Should I keep it for
> internal use?
>
> Thanks a lot.
> --
> Luis Fernando Muñoz MejÃas
> Luis.Fernando.Munoz.Mejias@cern.ch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com