Mailing List Archive

Problems flushing with omprog
Hello,
I've been having issues trying to print the confirmation message for
rsyslog acknowledgement.
on line:
onMessage(line)
status = "OK"
print(status, flush=True)

My program is printing BrokenPipeError: [Errno 32] Broken pipe for a lot of
the messages it receives, which leads me to think that the message in
question has not been processed. In fact, there are plenty of messages not
fully processed.

I'm using the following configuration in rsyslog:
if ($programname == "sysd11") then {
if (re_match($msg, "status=([a-zA-Z0-9]+)")) then {
action(type="omprog"
name="parser_processed"
binary="/usr/local/scripts/py/logparser.py"
template="POSTFIX_FormatTo"
confirmMessages="on"
queue.type="LinkedList"
queue.size="25000"
queue.workerThreads="15"
queue.workerThreadMinimumMessages="40"
output="/var/log/system/processed.log"
)
}

Could anyone help me on that?
I'm using impstats to review if messages are discarded but there's no
message discarded, which should lead into all messages been processed, but
that's not the case, definitely.

Appreciated,
Eudald
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Problems flushing with omprog [ In reply to ]
The BrokenPipeError message could be related to how you write to stderr
from your Python program. If you are writing without flushing immediately
(as I recall from a previous question from you), then maybe Python tries to
write the stderr buffer when the process exits (if there is a high load,
rsyslog will from time to time start and stop instances of the process),
but at this moment omprog has already closed the pipe, hence the
BrokenPipeError... Take a look at this, maybe it helps:
https://stackoverflow.com/questions/26692284/how-to-prevent-brokenpipeerror-when-doing-a-flush-in-python

You could also try with a simpler program, to try to reduce the scope of
the problem (if the simpler program works, then the problem is probably in
your Python code). In the rsyslog test suites there are some simple bash
scripts that are being used for testing omprog in CI (with and without the
confirmMessages mode, with and without the output flag, etc.). If you don't
find them, tell me.

If none of these hints help, please send your program, maybe we can spot
the problem (I'm not fluent in Python, but I can try...).

Regards

On Fri, Feb 7, 2020, 09:03 Eudald ValcĂ rcel via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hello,
> I've been having issues trying to print the confirmation message for
> rsyslog acknowledgement.
> on line:
> onMessage(line)
> status = "OK"
> print(status, flush=True)
>
> My program is printing BrokenPipeError: [Errno 32] Broken pipe for a lot of
> the messages it receives, which leads me to think that the message in
> question has not been processed. In fact, there are plenty of messages not
> fully processed.
>
> I'm using the following configuration in rsyslog:
> if ($programname == "sysd11") then {
> if (re_match($msg, "status=([a-zA-Z0-9]+)")) then {
> action(type="omprog"
> name="parser_processed"
> binary="/usr/local/scripts/py/logparser.py"
> template="POSTFIX_FormatTo"
> confirmMessages="on"
> queue.type="LinkedList"
> queue.size="25000"
> queue.workerThreads="15"
> queue.workerThreadMinimumMessages="40"
> output="/var/log/system/processed.log"
> )
> }
>
> Could anyone help me on that?
> I'm using impstats to review if messages are discarded but there's no
> message discarded, which should lead into all messages been processed, but
> that's not the case, definitely.
>
> Appreciated,
> Eudald
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.