Mailing List Archive: Using a program to modify incoming syslog messages and then forward them to a remote machine

Hello, I have a situation in which some log messages are coming (from a network machineA) in a linux machineB (that has rsyslog 8 demon operating) and machineB forwards these messages to another network machineC.  What I want to do is modify these messages in a way that some usernames are transformed to a generic string . like "UserName" so they are not visible when they reach the target 3rd machineC.
I am hoping that what will help me is the mmexternal module , like you mention herehttps://www.rsyslog.com/doc/v8-stable/configuration/modules/mmexternal.htmlIn the example at the end of the above link, you mention that in the /etc/rsyslog.conf (or alternatively in a file /etc/rsyslog.d/my_modification.conf ), we should have :module(load="mmexternal")action(type="mmexternal" binary="/path/to/my_transformation_script.py")
I am trying that content, but I get errors that the /my_transformation_script.py cannot be called , even when I change its ownership to syslog:syslog, and have it executable.  I also tried for 2nd line this (not sure what I could put instead of *.* , because I do not know what facility and severity should I put there):
if($hostname == 'machineA') then { *.* action(type="mmexternal" binary="/path/to/my_transformation_script.py")}
but still my program: my_transformation_script.py cannot be called (executed)
And on the other hand in the 4th slide of this link https://www.slideshare.net/rainergerhards1/writing-rsyslog-pI see that you mention: module(load="omprog")but nothing about : module(load="mmexternal")So what should I use? "omprog" or "mmexternal" ? And in some of the examples presented onhttps://github.com/rsyslog/rsyslog/tree/master/plugins/externalthere are no ways on how to make your rsyslog.conf contents.Only here it gives an example , and it does not mention omprog , only mmexternal :https://github.com/rsyslog/rsyslog/tree/master/plugins/external/messagemod/anon_cc_nbrsSo what is the correct thing to do ?
Furthermore I had a look at the text ofhttps://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.mdbut I do not understand how am I supposed to implement these mentioned exchanged messages. Who will reply that "OK" , to whom? Should my: my_transformation_script.py have some output somewhere (where? sys.stdout ? std.stderr ? ) that gives these "OK" strings to something? The 3 examples mentioned inhttps://github.com/rsyslog/rsyslog/tree/master/plugins/external/skeletons/pythondo not bother (as far as I can understand) with reporting back any "OK" to somebody.So is this "OK" replying something I can ignore?
I have a python function that does the transformation that I want, using regular expressions, but having this function work in a program (the one that I have named: my_transformation_script.py ) that is actually called by rsyslog, and then having these transformed messages forwarded to machineC, seems impossible. So to summarize , should I use mmexternal , or omprog?And in any case, how should my /etc/rsyslog.d/my_modification.conf look like? Any dangerous point about who owns the file my_transformation_script.py and is there any suggested directory to place it?
Georgios Doumas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

whatever mail client you are using is rapping things badly, making it a little
hard to read.

omprog is when you are doing output that you need to run through a custom
program

mmexternal is when you want to have an external program modify variables

the built-in properties cannot be modified, so if you want to change your
output, you will need to create a new template that uses the variables that you
define instead of the built-in ones.

I don't know how common the logs are that you are wanting to modify, but be
aware that regex matches tend to be preformance limiters.

In your place, I would use mmnormalize to parse the messages and tag the ones
that you want to modify, and then use a lookup table to lookup the username and
return a placeholder value for ones that match (and a nomatch value that you can
test for to then set the value to the original username)

then use this variable in a template to craft your new output message

if you want to use mmexternal to modify a variable, you can do that instead of
mmnormalize/table_lookup(), but you would then need to restart rsyslog if your
list of users to filter changes.

log messages with the template RSYSLOG_DebugFormat to see what all the variables
are at the point where you log the message. Do this before and after your call
to mmexternal to see what it has changed. Also have your external script write
what it receives to a file so you can compare it with the debugformat output.

the slide deck that you mention probably predates mmexternal

David Lang

On Thu, 13 Feb 2020, george doumas via rsyslog wrote:

> Date: Thu, 13 Feb 2020 09:51:09 +0000 (UTC)
> From: george doumas via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: george doumas <geodoomgr@yahoo.gr>
> Subject: [rsyslog] Using a program to modify incoming syslog messages and then
> forward them to a remote machine
>
> Hello, I have a situation in which some log messages are coming (from a network machineA) in a linux machineB (that has rsyslog 8 demon operating) and machineB forwards these messages to another network machineC.  What I want to do is modify these messages in a way that some usernames are transformed to a generic string . like "UserName" so they are not visible when they reach the target 3rd machineC.
> I am hoping that what will help me is the mmexternal module , like you mention herehttps://www.rsyslog.com/doc/v8-stable/configuration/modules/mmexternal.htmlIn the example at the end of the above link, you mention that in the /etc/rsyslog.conf (or alternatively in a file /etc/rsyslog.d/my_modification.conf ), we should have :module(load="mmexternal")action(type="mmexternal" binary="/path/to/my_transformation_script.py")
> I am trying that content, but I get errors that the /my_transformation_script.py cannot be called , even when I change its ownership to syslog:syslog, and have it executable.  I also tried for 2nd line this (not sure what I could put instead of *.* , because I do not know what facility and severity should I put there):
> if($hostname == 'machineA') then { *.* action(type="mmexternal" binary="/path/to/my_transformation_script.py")}
> but still my program: my_transformation_script.py cannot be called (executed)
> And on the other hand in the 4th slide of this link https://www.slideshare.net/rainergerhards1/writing-rsyslog-pI see that you mention: module(load="omprog")but nothing about : module(load="mmexternal")So what should I use? "omprog" or "mmexternal" ? And in some of the examples presented onhttps://github.com/rsyslog/rsyslog/tree/master/plugins/externalthere are no ways on how to make your rsyslog.conf contents.Only here it gives an example , and it does not mention omprog , only mmexternal :https://github.com/rsyslog/rsyslog/tree/master/plugins/external/messagemod/anon_cc_nbrsSo what is the correct thing to do ?
> Furthermore I had a look at the text ofhttps://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.mdbut I do not understand how am I supposed to implement these mentioned exchanged messages. Who will reply that "OK" , to whom? Should my: my_transformation_script.py have some output somewhere (where? sys.stdout ? std.stderr ? ) that gives these "OK" strings to something? The 3 examples mentioned inhttps://github.com/rsyslog/rsyslog/tree/master/plugins/external/skeletons/pythondo not bother (as far as I can understand) with reporting back any "OK" to somebody.So is this "OK" replying something I can ignore?
> I have a python function that does the transformation that I want, using regular expressions, but having this function work in a program (the one that I have named: my_transformation_script.py ) that is actually called by rsyslog, and then having these transformed messages forwarded to machineC, seems impossible. So to summarize , should I use mmexternal , or omprog?And in any case, how should my /etc/rsyslog.d/my_modification.conf look like? Any dangerous point about who owns the file my_transformation_script.py and is there any suggested directory to place it?
> Georgios Doumas
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.