I am writing an omprog module, and I have a couple of questions about
omprog with transactions:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions
I am using rsyslog 8.32.0-1ubuntu4 under Ubuntu 18.04.
1. I have been unable to get batching to work.
I don't see a batch size option for omprog itself - whereas omhttp
<https://www.rsyslog.com/doc/master/configuration/modules/omhttp.html#batch>
has documented batch.maxbytes and batch.maxsize parameters, with
defaults of 100 and 10MB respectively.
At
https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html#batching-of-messages
it just says:
"Note that batch sizes are variable within the range of 1 to a user
configured maximum limit."
I found queue parameters
<https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html>,
so I thought perhaps I could add
queue.type="LinkedList"
queue.minDequeueBatchSize="50"
to my action(...) settings - but if I submit two messages immediately
after each other, I still receive each one in a separate transaction.
Docs say that by default it should have a queue for 1000 messages and
wait for up to 1 second to dequeue.
Here's my full statement:
local0.* action(type="omprog"
template="loki"
binary="/usr/local/sbin/omprog-loki.py"
action.resumeInterval="5"
confirmMessages="on"
useTransactions="on"
output="/tmp/loki.err"
queue.type="LinkedList"
queue.minDequeueBatchSize="50"
forceSingleInstance="on")
& stop
If I log the communication to my program I see:
'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.741573+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
aaa\n'
'COMMIT TRANSACTION\n'
'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.759152+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
bbb\n'
'COMMIT TRANSACTION\n'
2. It appears that it's possible to enable useTransactions without
enabling confirmMessages. If you do that, do you still need to confirm
the COMMIT TRANSACTION message or not?
It doesn't explicitly say so at
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions
If no such response is required, then there seems to be little benefit
in enabling useTransactions without confirmMessages. The external
program might as well make batches on whatever boundaries it feels
like. However I can see value in confirming just the commit message,
without the individual messages - it would make the message transfer
less chatty and so more efficient.
Perhaps I'm misunderstanding something here.
Regards,
Brian Candler.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
omprog with transactions:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions
I am using rsyslog 8.32.0-1ubuntu4 under Ubuntu 18.04.
1. I have been unable to get batching to work.
I don't see a batch size option for omprog itself - whereas omhttp
<https://www.rsyslog.com/doc/master/configuration/modules/omhttp.html#batch>
has documented batch.maxbytes and batch.maxsize parameters, with
defaults of 100 and 10MB respectively.
At
https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html#batching-of-messages
it just says:
"Note that batch sizes are variable within the range of 1 to a user
configured maximum limit."
I found queue parameters
<https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html>,
so I thought perhaps I could add
queue.type="LinkedList"
queue.minDequeueBatchSize="50"
to my action(...) settings - but if I submit two messages immediately
after each other, I still receive each one in a separate transaction.
Docs say that by default it should have a queue for 1000 messages and
wait for up to 1 second to dequeue.
Here's my full statement:
local0.* action(type="omprog"
template="loki"
binary="/usr/local/sbin/omprog-loki.py"
action.resumeInterval="5"
confirmMessages="on"
useTransactions="on"
output="/tmp/loki.err"
queue.type="LinkedList"
queue.minDequeueBatchSize="50"
forceSingleInstance="on")
& stop
If I log the communication to my program I see:
'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.741573+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
aaa\n'
'COMMIT TRANSACTION\n'
'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.759152+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
bbb\n'
'COMMIT TRANSACTION\n'
2. It appears that it's possible to enable useTransactions without
enabling confirmMessages. If you do that, do you still need to confirm
the COMMIT TRANSACTION message or not?
It doesn't explicitly say so at
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions
If no such response is required, then there seems to be little benefit
in enabling useTransactions without confirmMessages. The external
program might as well make batches on whatever boundaries it feels
like. However I can see value in confirming just the commit message,
without the individual messages - it would make the message transfer
less chatty and so more efficient.
Perhaps I'm misunderstanding something here.
Regards,
Brian Candler.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.