Hello, world.
I discussed this in private with Rainer, and he suggested me to bring
the discussion here.
I'm already developing an output module for feeding an Oracle database
with rsyslog input. Rainer already committed some patches to the
"oracle" branch, in git. Let me remember that this is highly
experimental, and I'm sending a big semantic change today. But, in
principle, the module does what you'd expect from it: it connects to a
DB, receives a SQL statement via doAction, prepares that statement, runs
it, commits.
It works, but it's way too slow for my needs. As I said when I started
this project, I need to be very fast, to prepare the statement at
connection time, run it many times, and definitely want batch
operations. Say, I want to insert 1000 entries with a single call to the
Oracle interface, then commit.
With what I know now of rsyslog, I can do it more or less like this:
$OmoracleStatementTemplate,"insert into foo(field1, field2, field3)
values(:val1, :val2, :val3)"
which is the statement to prepare by Oracle. This way, I can prepare the
statement at createInstance() time. Then, I can specify the batch size
with something like
$OmoracleBatchSize 1000
With this, also at createInstance() time I can specify that doAction is
called only if there are 1000 entries pending for this selector, like
this:
CODE_STD_STRING_REQUESTparseSelectorAct(batch_size);
The bad part is that rsyslog will deliver to the output module a single
string per entry. So, I'd have to split each entry into its fields as
part of the doAction() code. I'd need some funny separator for each
field, to avoid problems. So far, it can be done. But the configuration
would look like this:
$OmoracleDB logdb
$OmoracleDBUser dbuser
$OmoracleDBPassword dbpassword
$OmoracleStatement "insert into foo(col1, col2) values (:fied1,
:field2)"
$OmoracleBatchSize 1000
$OmoracleFieldSeparator ****
*.* :omoracle:;"%field1%****%field2%"
and make doAction split the fields appropriately.
I bet it works. But it's probably too ugly for users. Cleaner ways may
need deeper changes into rsyslog's API so that the module gets direct
access to each field. That's probably a lot of work and I can't wait for
that.
So, my questions (at last!): Are there any other alternatives? Is this
"ugly" way of working good for other users? Should I keep it for
internal use?
Thanks a lot.
--
Luis Fernando Muñoz MejÃas
Luis.Fernando.Munoz.Mejias@cern.ch
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
I discussed this in private with Rainer, and he suggested me to bring
the discussion here.
I'm already developing an output module for feeding an Oracle database
with rsyslog input. Rainer already committed some patches to the
"oracle" branch, in git. Let me remember that this is highly
experimental, and I'm sending a big semantic change today. But, in
principle, the module does what you'd expect from it: it connects to a
DB, receives a SQL statement via doAction, prepares that statement, runs
it, commits.
It works, but it's way too slow for my needs. As I said when I started
this project, I need to be very fast, to prepare the statement at
connection time, run it many times, and definitely want batch
operations. Say, I want to insert 1000 entries with a single call to the
Oracle interface, then commit.
With what I know now of rsyslog, I can do it more or less like this:
$OmoracleStatementTemplate,"insert into foo(field1, field2, field3)
values(:val1, :val2, :val3)"
which is the statement to prepare by Oracle. This way, I can prepare the
statement at createInstance() time. Then, I can specify the batch size
with something like
$OmoracleBatchSize 1000
With this, also at createInstance() time I can specify that doAction is
called only if there are 1000 entries pending for this selector, like
this:
CODE_STD_STRING_REQUESTparseSelectorAct(batch_size);
The bad part is that rsyslog will deliver to the output module a single
string per entry. So, I'd have to split each entry into its fields as
part of the doAction() code. I'd need some funny separator for each
field, to avoid problems. So far, it can be done. But the configuration
would look like this:
$OmoracleDB logdb
$OmoracleDBUser dbuser
$OmoracleDBPassword dbpassword
$OmoracleStatement "insert into foo(col1, col2) values (:fied1,
:field2)"
$OmoracleBatchSize 1000
$OmoracleFieldSeparator ****
*.* :omoracle:;"%field1%****%field2%"
and make doAction split the fields appropriately.
I bet it works. But it's probably too ugly for users. Cleaner ways may
need deeper changes into rsyslog's API so that the module gets direct
access to each field. That's probably a lot of work and I can't wait for
that.
So, my questions (at last!): Are there any other alternatives? Is this
"ugly" way of working good for other users? Should I keep it for
internal use?
Thanks a lot.
--
Luis Fernando Muñoz MejÃas
Luis.Fernando.Munoz.Mejias@cern.ch
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com