Mailing List Archive

omprog and transactions
I am writing an omprog module, and I have a couple of questions about
omprog with transactions:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions

I am using rsyslog 8.32.0-1ubuntu4 under Ubuntu 18.04.

1.  I have been unable to get batching to work.

I don't see a batch size option for omprog itself - whereas omhttp
<https://www.rsyslog.com/doc/master/configuration/modules/omhttp.html#batch>
has documented batch.maxbytes and batch.maxsize parameters, with
defaults of 100 and 10MB respectively.

At
https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html#batching-of-messages
it just says:

"Note that batch sizes are variable within the range of 1 to a user
configured maximum limit."

I found queue parameters
<https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html>,
so I thought perhaps I could add

  queue.type="LinkedList"
  queue.minDequeueBatchSize="50"

to my action(...) settings - but if I submit two messages immediately
after each other, I still receive each one in a separate transaction. 
Docs say that by default it should have a queue for 1000 messages and
wait for up to 1 second to dequeue.

Here's my full statement:

local0.*  action(type="omprog"
  template="loki"
  binary="/usr/local/sbin/omprog-loki.py"
  action.resumeInterval="5"
  confirmMessages="on"
  useTransactions="on"
  output="/tmp/loki.err"
  queue.type="LinkedList"
  queue.minDequeueBatchSize="50"
  forceSingleInstance="on")
& stop

If I log the communication to my program I see:

'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.741573+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
aaa\n'
'COMMIT TRANSACTION\n'
'BEGIN TRANSACTION\n'
'2019-09-16T09:55:37.759152+00:00
{job="rsyslog",src_ip="127.0.0.1",hostname="loki",facility="local0",severity="info"}
bbb\n'
'COMMIT TRANSACTION\n'


2. It appears that it's possible to enable useTransactions without
enabling confirmMessages.  If you do that, do you still need to confirm
the COMMIT TRANSACTION message or not?

It doesn't explicitly say so at
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions

If no such response is required, then there seems to be little benefit
in enabling useTransactions without confirmMessages.  The external
program might as well make batches on whatever boundaries it feels
like.  However I can see value in confirming just the commit message,
without the individual messages - it would make the message transfer
less chatty and so more efficient.

Perhaps I'm misunderstanding something here.

Regards,

Brian Candler.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
On 16/09/2019 11:25, Brian Candler wrote:
>
> I found queue parameters
> <https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html>,
> so I thought perhaps I could add
>
>   queue.type="LinkedList"
>   queue.minDequeueBatchSize="50"
>
> to my action(...) settings - but if I submit two messages immediately
> after each other, I still receive each one in a separate transaction.
>
To answer my own question: queue.minDequeueBatchSize was only added in
Jan 2019, after Ubuntu 18.04 was released.  It made it into v8.1901.0.

rsyslogd itself was telling me this, as I found in its own logs.

Sep 16 16:05:05 loki rsyslogd: error during parsing file
/etc/rsyslog.d/40-loki.conf, on or before line 37: parameter
'queue.minDequeueBatchSize' not known -- typo in config file? [v8.32.0
try http://www.rsyslog.com/e/2207 ]


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
On Mon, 16 Sep 2019, Brian Candler via rsyslog wrote:

> 1.  I have been unable to get batching to work.

batching only happens under load, the worker thread goes to fetch messages from
the queue and says 'give me the first messages, up to batch_size, if there is
only one message in the queue at that time, it only gets one message. It doesn't
delay processing that message to let others accumulate. If it can't do this fast
enough, eventually when it asks for more messgaes, there will be more than one
waiting.

this minimizes latency for every message, at the cost of cpu on the system.

I don't think that you would see the effect with omprog as the program will see
all the messages in stdin serially, so the batching is just going to be an
efficiency thing in it's interaction with the queue it's reading from


> Docs say that by default it should have a queue for 1000 messages and
> wait for up to 1 second to dequeue.

by default it will not wait
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
be careful with mindequeuebatchsize, it adds delays which may be trivial at high
log volumes, but could bite you at low traffic volumes

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
Note: the last major omprog refcotring was done by
https://github.com/jsiwrk - I suggest to create a github issues for
the questions and mention @jsiwrk - he has agreed to take care of the
module.

Rainer

El mar., 17 sept. 2019 a las 1:50, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> On Mon, 16 Sep 2019, Brian Candler via rsyslog wrote:
>
> > 1. I have been unable to get batching to work.
>
> batching only happens under load, the worker thread goes to fetch messages from
> the queue and says 'give me the first messages, up to batch_size, if there is
> only one message in the queue at that time, it only gets one message. It doesn't
> delay processing that message to let others accumulate. If it can't do this fast
> enough, eventually when it asks for more messgaes, there will be more than one
> waiting.
>
> this minimizes latency for every message, at the cost of cpu on the system.
>
> I don't think that you would see the effect with omprog as the program will see
> all the messages in stdin serially, so the batching is just going to be an
> efficiency thing in it's interaction with the queue it's reading from
>
>
> > Docs say that by default it should have a queue for 1000 messages and
> > wait for up to 1 second to dequeue.
>
> by default it will not wait
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
On 17/09/2019 00:51, David Lang wrote:
> be careful with mindequeuebatchsize, it adds delays which may be
> trivial at high log volumes, but could bite you at low traffic volumes

I'm happy with (say) 500ms-1s of delay under low volumes.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omprog and transactions [ In reply to ]
> 2. It appears that it's possible to enable useTransactions without
> enabling confirmMessages.  If you do that, do you still need to confirm
> the COMMIT TRANSACTION message or not?

No you don't. For more details, please see:
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

However, note that the use of transactions in omprog is experimental and has
some known issues, as noted in the documentation
(https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html#usetransactions).


> If no such response is required, then there seems to be little benefit
> in enabling useTransactions without confirmMessages.  The external
> program might as well make batches on whatever boundaries it feels
> like.

The usefulness of this use case is admittedly quite subtle: it would be
difficult for the program to decide the batch boundaries by its own, since
it does not know if there will be a subsequent message in a short time, or
not. (To do this correctly, the program would need to have a background
thread that can decide to process an accumulated batch. However this would
greatly complicate the program. Such programs are expected to be
single-threaded and react only when receiving more input from stdin.)


> However I can see value in confirming just the commit message,
> without the individual messages - it would make the message transfer
> less chatty and so more efficient.

I think you are right, it would be a good idea to have this working mode.
Currently the interface just mimics how native rsyslog plugins internally
work: when transactions are used, each individual message must still be
confirmed as OK to the rsyslog core. But the omprog module could handle this
confirmation automatically, without requiring a response from the external
program.



-----
--
Joan Sala
--
Sent from: http://rsyslog-users.1305293.n2.nabble.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.