Mailing List Archive

Fwd: Json template and forward slash escape
Hi,

I've noticed something bothering when using templates to create a json :
rsyslog keeps escaping forward slashes.
Eg.:
{"attr": "/value/with/forward/slash"}
becomes
{"attr": "\/value\/with\/forward\/slash"}

Witnessed on Debian, with RSyslog version 8.1901.0 and 8.2110.0. I guess
it's always been like this.

Is it possible to prevent this behavior ? Perhaps some hidden option
that's not documented ?

Simple repro :
#### Rsyslog config:
template(name="test-json" type="list") {
constant(value="{")
property(outname="host" name="hostname" format="jsonf")
constant(value=",")
property(name="$!all-json" position.from="2")
}

module(load="mmjsonparse")
ruleset(name="test_template") {
action(type="mmjsonparse" cookie="")
if $parsesuccess == "OK" then {
action(type="omfile" file="/var/log/rsyslog/test_parsed.log")
action(type="omfile" file="/var/log/rsyslog/test_template.log"
template="test-json")
}
else {
action(type="omfile" file="/var/log/rsyslog/test_error.log")
}
stop
}

module(load="imfile")
input(type="imfile"
File="/var/log/rsyslog/test.log"
Tag="ignore"
Ruleset="test_template"
)

#### And then:
$ echo '{"attr": "/value/with/forward/slash"}' >
/var/log/rsyslog/test.log

$ cat /var/log/rsyslog/test_parsed.log
2021-11-18T11:45:00.415066+01:00 host123 ignore {"attr":
"/value/with/forward/slash"}

$ cat /var/log/rsyslog/test_template.log
{"host":"host123", "metadata": { "filename":
"\/var\/log\/rsyslog\/test.log", "fileoffset": "0" }, "attr":
"\/value\/with\/forward\/slash" }
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Json template and forward slash escape [ In reply to ]
If there is a way to disable that behavior I have not been able to find it, and frankly, due to the mandate to “not break what has come before”, I would be surprised if the behavior is changed. The maintainers will know for sure. We get around it by replacing \\/ with / during SIEM searches.

Regards,


> On Nov 18, 2021, at 05:13, nope via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi,
>
> I've noticed something bothering when using templates to create a json : rsyslog keeps escaping forward slashes.
> Eg.:
> {"attr": "/value/with/forward/slash"}
> becomes
> {"attr": "\/value\/with\/forward\/slash"}
>
> Witnessed on Debian, with RSyslog version 8.1901.0 and 8.2110.0. I guess it's always been like this.
>
> Is it possible to prevent this behavior ? Perhaps some hidden option that's not documented ?
>
> Simple repro :
> #### Rsyslog config:
> template(name="test-json" type="list") {
> constant(value="{")
> property(outname="host" name="hostname" format="jsonf")
> constant(value=",")
> property(name="$!all-json" position.from="2")
> }
>
> module(load="mmjsonparse")
> ruleset(name="test_template") {
> action(type="mmjsonparse" cookie="")
> if $parsesuccess == "OK" then {
> action(type="omfile" file="/var/log/rsyslog/test_parsed.log")
> action(type="omfile" file="/var/log/rsyslog/test_template.log" template="test-json")
> }
> else {
> action(type="omfile" file="/var/log/rsyslog/test_error.log")
> }
> stop
> }
>
> module(load="imfile")
> input(type="imfile"
> File="/var/log/rsyslog/test.log"
> Tag="ignore"
> Ruleset="test_template"
> )
>
> #### And then:
> $ echo '{"attr": "/value/with/forward/slash"}' > /var/log/rsyslog/test.log
>
> $ cat /var/log/rsyslog/test_parsed.log
> 2021-11-18T11:45:00.415066+01:00 host123 ignore {"attr": "/value/with/forward/slash"}
>
> $ cat /var/log/rsyslog/test_template.log
> {"host":"host123", "metadata": { "filename": "\/var\/log\/rsyslog\/test.log", "fileoffset": "0" }, "attr": "\/value\/with\/forward\/slash" }
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Json template and forward slash escape [ In reply to ]
Thanks for your feedback.
If there's no option to avoid getting escaped forward slashed, this
could be a feature request then.

The final destination of our logs is an Elasticsearch server.
My workaround for the escaping problem is to output the raw messages to
a python script (using omprog module).
The python script parse the message, creates a json object, adds some
info to it (like host, a custom id, the timestamp, ...), and finally
dumps that json to a file.
Those files are then gathered by a bash script (run by a cronjob) to
make bulk requests to insert the data on Elasticsearch.
A bit convoluted, and not very efficient, isn't it?

An option to disable the escaping would be very helpful.


Le 2021-11-18 13:05, John Chivian a écrit :
> If there is a way to disable that behavior I have not been able to find
> it,
> and frankly, due to the mandate to “not break what has come before”, I
> would
> be surprised if the behavior is changed. The maintainers will know for
> sure. We get around it by replacing \\/ with / during SIEM searches.
>
> Regards,
>
>
>> On Nov 18, 2021, at 05:13, nope via rsyslog
>> <rsyslog@lists.adiscon.com>
>> wrote:
>>
>> Hi,
>>
>> I've noticed something bothering when using templates to create a json
>> :
>> rsyslog keeps escaping forward slashes.
>> Eg.:
>> {"attr": "/value/with/forward/slash"}
>> becomes
>> {"attr": "\/value\/with\/forward\/slash"}
>>
>> Witnessed on Debian, with RSyslog version 8.1901.0 and 8.2110.0. I
>> guess
>> it's always been like this.
>>
>> Is it possible to prevent this behavior ? Perhaps some hidden option
>> that's not documented ?
>>
>> Simple repro :
>> #### Rsyslog config:
>> template(name="test-json" type="list") {
>> constant(value="{")
>> property(outname="host" name="hostname" format="jsonf")
>> constant(value=",")
>> property(name="$!all-json" position.from="2")
>> }
>>
>> module(load="mmjsonparse")
>> ruleset(name="test_template") {
>> action(type="mmjsonparse" cookie="")
>> if $parsesuccess == "OK" then {
>> action(type="omfile" file="/var/log/rsyslog/test_parsed.log")
>> action(type="omfile" file="/var/log/rsyslog/test_template.log"
>> template="test-json")
>> }
>> else {
>> action(type="omfile" file="/var/log/rsyslog/test_error.log")
>> }
>> stop
>> }
>>
>> module(load="imfile")
>> input(type="imfile"
>> File="/var/log/rsyslog/test.log"
>> Tag="ignore"
>> Ruleset="test_template"
>> )
>>
>> #### And then:
>> $ echo '{"attr": "/value/with/forward/slash"}' >
>> /var/log/rsyslog/test.log
>>
>> $ cat /var/log/rsyslog/test_parsed.log
>> 2021-11-18T11:45:00.415066+01:00 host123 ignore {"attr":
>> "/value/with/forward/slash"}
>>
>> $ cat /var/log/rsyslog/test_template.log
>> {"host":"host123", "metadata": { "filename":
>> "\/var\/log\/rsyslog\/test.log", "fileoffset": "0" }, "attr":
>> "\/value\/with\/forward\/slash" }
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.