Mailing List Archive

I do not fully understand the question (maybe language issue on my
side), but there is a syntax error:

In a string template, properties must be enclosed in percent sign. so:

... string="%msg%

HTH
Rainer

El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hello!
> Field msg contains complete json with data. I would like to instert it
> to mongodb as is. But now rsyslog inserts it as a value of key "msg".
> So now is:
> msg: '{"foo":"bar"}
> a i'd like to insert: only:
> '{"foo","bar"}'
> I tried with template:
> template(name="ui-json" type="string" string="%msg")
> but it doesn't do what I need. Is it possible to configure it using
> template or this is imposible due to ommnongodb limitation?
>
> Marcin Miros?aw
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Maybe when I show examples from mongo it will be more clear.

> db.log2.find()
[.
{ _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg: '{"foo":"bar"}' },
{ _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
]

First record shows how msg is inserted to mongo by rsyslog. Second
record is what I would like to get.

rsyslog do:
db.log2.insert({msg:'{"foo":"bar"}'})

but I'd like to have:
db.log2.insert({"foo":"bar"})

Marcin


W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
> I do not fully understand the question (maybe language issue on my
> side), but there is a syntax error:
>
> In a string template, properties must be enclosed in percent sign. so:
>
> ... string="%msg%
>
> HTH
> Rainer
>
> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> Hello!
>> Field msg contains complete json with data. I would like to instert it
>> to mongodb as is. But now rsyslog inserts it as a value of key "msg".
>> So now is:
>> msg: '{"foo":"bar"}
>> a i'd like to insert: only:
>> '{"foo","bar"}'
>> I tried with template:
>> template(name="ui-json" type="string" string="%msg")
>> but it doesn't do what I need. Is it possible to configure it using
>> template or this is imposible due to ommnongodb limitation?
>>
>> Marcin Miros?aw
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Don't know about this particular output module but in general what you
want is for rsyslog to parse the message and insert it as json object.

So you need to use parse_json() function on the input string and then
use proper template which will render the json to appropirate string. I
use similar approach (without the parsing part) to create output json
for Splunk's HEC input - the idea is roughly the same.


On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
> Maybe when I show examples from mongo it will be more clear.
>
> > db.log2.find()
> [.
>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg: '{"foo":"bar"}' },
>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
> ]
>
> First record shows how msg is inserted to mongo by rsyslog. Second
> record is what I would like to get.
>
> rsyslog do:
> db.log2.insert({msg:'{"foo":"bar"}'})
>
> but I'd like to have:
> db.log2.insert({"foo":"bar"})
>
> Marcin
>
>
> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>> I do not fully understand the question (maybe language issue on my
>> side), but there is a syntax error:
>>
>> In a string template, properties must be enclosed in percent sign. so:
>>
>> ... string="%msg%
>>
>> HTH
>> Rainer
>>
>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>>>
>>> Hello!
>>> Field msg contains complete json with data. I would like to instert it
>>> to mongodb as is. But now rsyslog inserts it as a value of key "msg".
>>> So now is:
>>> msg: '{"foo":"bar"}
>>> a i'd like to insert: only:
>>> '{"foo","bar"}'
>>> I tried with template:
>>> template(name="ui-json" type="string" string="%msg")
>>> but it doesn't do what I need. Is it possible to configure it using
>>> template or this is imposible due to ommnongodb limitation?
>>>
>>> Marcin Miros?aw
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
> Don't know about this particular output module but in general what you
> want is for rsyslog to parse the message and insert it as json object.

Meseems that now rsyslog put %msg% as json object. (
ex: msg: '{"foo":"bar"}' }
)


> So you need to use parse_json() function on the input string and then
> use proper template which will render the json to appropirate string. I
> use similar approach (without the parsing part) to create output json
> for Splunk's HEC input - the idea is roughly the same.
>
>
> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>> Maybe when I show examples from mongo it will be more clear.
>>
>> > db.log2.find()
>> [.
>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg: '{"foo":"bar"}' },
>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>> ]
>>
>> First record shows how msg is inserted to mongo by rsyslog. Second
>> record is what I would like to get.
>>
>> rsyslog do:
>> db.log2.insert({msg:'{"foo":"bar"}'})
>>
>> but I'd like to have:
>> db.log2.insert({"foo":"bar"})
>>
>> Marcin
>>
>>
>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>> I do not fully understand the question (maybe language issue on my
>>> side), but there is a syntax error:
>>>
>>> In a string template, properties must be enclosed in percent sign. so:
>>>
>>> ... string="%msg%
>>>
>>> HTH
>>> Rainer
>>>
>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>
>>>> Hello!
>>>> Field msg contains complete json with data. I would like to instert it
>>>> to mongodb as is. But now rsyslog inserts it as a value of key "msg".
>>>> So now is:
>>>> msg: '{"foo":"bar"}
>>>> a i'd like to insert: only:
>>>> '{"foo","bar"}'
>>>> I tried with template:
>>>> template(name="ui-json" type="string" string="%msg")
>>>> but it doesn't do what I need. Is it possible to configure it using
>>>> template or this is imposible due to ommnongodb limitation?
>>>>
>>>> Marcin Miros?aw
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if you DON'T LIKE THAT.
>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

You didn't read the docs, did you? ;-)

https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html

"Note rsyslog contains a canned default template to write to the
MongoDB. It will be used automatically if no other template is specified
to be used. This template is:

template(name="BSON" type="string" string="\\"sys\\" : \\"%hostname%\\",
\\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
\\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
\\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
\\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
\\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
\\"level\\" : \\"%syslogpriority-text%\\"")

This creates the BSON document needed for MongoDB if no template is
specified. The default schema is aligned to CEE and project lumberjack.
As such, the field names are standard lumberjack field names, and *not*
rsyslog property names
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."

PS: Sorry for the direct reply, I sometimes hit the "reply" button
instead of "reply to list".

On 3.10.2022 22:02, Marcin Miros?aw wrote:
> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>> Don't know about this particular output module but in general what
>> you want is for rsyslog to parse the message and insert it as json
>> object.
>
> Meseems that now rsyslog put %msg% as json object. (
> ex: msg: '{"foo":"bar"}' }
> )
>
>
>> So you need to use parse_json() function on the input string and then
>> use proper template which will render the json to appropirate string.
>> I use similar approach (without the parsing part) to create output
>> json for Splunk's HEC input - the idea is roughly the same.
>>
>>
>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>> Maybe when I show examples from mongo it will be more clear.
>>>
>>> > db.log2.find()
>>> [.
>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg: '{"foo":"bar"}' },
>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>> ]
>>>
>>> First record shows how msg is inserted to mongo by rsyslog. Second
>>> record is what I would like to get.
>>>
>>> rsyslog do:
>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>
>>> but I'd like to have:
>>> db.log2.insert({"foo":"bar"})
>>>
>>> Marcin
>>>
>>>
>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>> I do not fully understand the question (maybe language issue on my
>>>> side), but there is a syntax error:
>>>>
>>>> In a string template, properties must be enclosed in percent sign. so:
>>>>
>>>> ... string="%msg%
>>>>
>>>> HTH
>>>> Rainer
>>>>
>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>
>>>>> Hello!
>>>>> Field msg contains complete json with data. I would like to
>>>>> instert it
>>>>> to mongodb as is. But now rsyslog inserts it as a value of key "msg".
>>>>> So now is:
>>>>> msg: '{"foo":"bar"}
>>>>> a i'd like to insert: only:
>>>>> '{"foo","bar"}'
>>>>> I tried with template:
>>>>> template(name="ui-json" type="string" string="%msg")
>>>>> but it doesn't do what I need. Is it possible to configure it using
>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>
>>>>> Marcin Miros?aw
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

:-)

OK, I understand that you did

template(name="i-json" type="string" string="%msg%")

action(type="ommongodb" server="..." [...] template="i-json")

right?

This way you should indeed be pushing the %msg% part of the incoming
syslog message.

Try adding

action(type="omfile" file="/tmp/debug.log" template="RSYSLOG_DebugFormat")

immediately before your ommongodb action to see what exactly your
properties look like. That usually helps finding what's happening inside
your processing pipeline.

On 4.10.2022 09:03, Marcin Miros?aw wrote:
>
> I did in the part about connection to mongodb ;)
>
> "It will be used automatically if no other template is specified to be
> used"
>
> But I specified template: template(name="ui-json" type="string"
> string="%msg%")
>
> so if I understand qutoed docs correctly, all about default, canned
> template doesn't apply to my case, am I right?
>
>
> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>
>> You didn't read the docs, did you? ;-)
>>
>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>
>> "Note rsyslog contains a canned default template to write to the
>> MongoDB. It will be used automatically if no other template is
>> specified to be used. This template is:
>>
>> template(name="BSON" type="string" string="\\"sys\\" : \\"%hostname%\\",
>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>
>> This creates the BSON document needed for MongoDB if no template is
>> specified. The default schema is aligned to CEE and project
>> lumberjack. As such, the field names are standard lumberjack field
>> names, and *not* rsyslog property names
>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>
>>
>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>> Don't know about this particular output module but in general what
>>>> you want is for rsyslog to parse the message and insert it as json
>>>> object.
>>>
>>> Meseems that now rsyslog put %msg% as json object. (
>>> ex: msg: '{"foo":"bar"}' }
>>> )
>>>
>>>
>>>> So you need to use parse_json() function on the input string and
>>>> then use proper template which will render the json to appropirate
>>>> string. I use similar approach (without the parsing part) to create
>>>> output json for Splunk's HEC input - the idea is roughly the same.
>>>>
>>>>
>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>
>>>>> > db.log2.find()
>>>>> [.
>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>> '{"foo":"bar"}' },
>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>> ]
>>>>>
>>>>> First record shows how msg is inserted to mongo by rsyslog. Second
>>>>> record is what I would like to get.
>>>>>
>>>>> rsyslog do:
>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>
>>>>> but I'd like to have:
>>>>> db.log2.insert({"foo":"bar"})
>>>>>
>>>>> Marcin
>>>>>
>>>>>
>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>> I do not fully understand the question (maybe language issue on my
>>>>>> side), but there is a syntax error:
>>>>>>
>>>>>> In a string template, properties must be enclosed in percent
>>>>>> sign. so:
>>>>>>
>>>>>> ... string="%msg%
>>>>>>
>>>>>> HTH
>>>>>> Rainer
>>>>>>
>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>> (<rsyslog@lists.adiscon.com> <mailto:rsyslog@lists.adiscon.com>)
>>>>>> escribió:
>>>>>>>
>>>>>>> Hello!
>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>> instert it
>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of key
>>>>>>> "msg".
>>>>>>> So now is:
>>>>>>> msg: '{"foo":"bar"}
>>>>>>> a i'd like to insert: only:
>>>>>>> '{"foo","bar"}'
>>>>>>> I tried with template:
>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>> but it doesn't do what I need. Is it possible to configure it using
>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>
>>>>>>> Marcin Miros?aw
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> <https://twitter.com/rgerhards>
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> http://www.rsyslog.com/professional-services/
>>>>> <http://www.rsyslog.com/professional-services/>
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> <https://twitter.com/rgerhards>
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> http://www.rsyslog.com/professional-services/
>>>> <http://www.rsyslog.com/professional-services/>
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> <https://twitter.com/rgerhards>
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

:)
I had to censore log...


Debug line with all properties:
FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux', PRI:
174,
syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-', MSGID:
'-',
TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality tzKnown="1"
isSynced="1" syncAccuracy="415383"]',
msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00 linux
a1 - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="415383"]
{"foo":"bar
","mode":750,"date":"2022-10-04T15:38:53"}'
$!:
$.:
$/:




W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
> :-)
>
> OK, I understand that you did
>
> template(name="i-json" type="string" string="%msg%")
>
> action(type="ommongodb" server="..." [...] template="i-json")
>
> right?
>
> This way you should indeed be pushing the %msg% part of the incoming
> syslog message.
>
> Try adding
>
> action(type="omfile" file="/tmp/debug.log"
> template="RSYSLOG_DebugFormat")
>
> immediately before your ommongodb action to see what exactly your
> properties look like. That usually helps finding what's happening
> inside your processing pipeline.
>
> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>
>> I did in the part about connection to mongodb ;)
>>
>> "It will be used automatically if no other template is specified to be
>> used"
>>
>> But I specified template: template(name="ui-json" type="string"
>> string="%msg%")
>>
>> so if I understand qutoed docs correctly, all about default, canned
>> template doesn't apply to my case, am I right?
>>
>>
>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>
>>> You didn't read the docs, did you? ;-)
>>>
>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>
>>> "Note rsyslog contains a canned default template to write to the
>>> MongoDB. It will be used automatically if no other template is
>>> specified to be used. This template is:
>>>
>>> template(name="BSON" type="string" string="\\"sys\\" :
>>> \\"%hostname%\\",
>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>
>>> This creates the BSON document needed for MongoDB if no template is
>>> specified. The default schema is aligned to CEE and project
>>> lumberjack. As such, the field names are standard lumberjack field
>>> names, and *not* rsyslog property names
>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>
>>>
>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>> Don't know about this particular output module but in general what
>>>>> you want is for rsyslog to parse the message and insert it as json
>>>>> object.
>>>>
>>>> Meseems that now rsyslog put %msg% as json object. (
>>>> ex: msg: '{"foo":"bar"}' }
>>>> )
>>>>
>>>>
>>>>> So you need to use parse_json() function on the input string and
>>>>> then use proper template which will render the json to appropirate
>>>>> string. I use similar approach (without the parsing part) to create
>>>>> output json for Splunk's HEC input - the idea is roughly the same.
>>>>>
>>>>>
>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>
>>>>>> > db.log2.find()
>>>>>> [.
>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>> '{"foo":"bar"}' },
>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>> ]
>>>>>>
>>>>>> First record shows how msg is inserted to mongo by rsyslog. Second
>>>>>> record is what I would like to get.
>>>>>>
>>>>>> rsyslog do:
>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>
>>>>>> but I'd like to have:
>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>
>>>>>> Marcin
>>>>>>
>>>>>>
>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>> I do not fully understand the question (maybe language issue on
>>>>>>> my
>>>>>>> side), but there is a syntax error:
>>>>>>>
>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>> sign. so:
>>>>>>>
>>>>>>> ... string="%msg%
>>>>>>>
>>>>>>> HTH
>>>>>>> Rainer
>>>>>>>
>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>> (<rsyslog@lists.adiscon.com> <mailto:rsyslog@lists.adiscon.com>)
>>>>>>> escribió:
>>>>>>>>
>>>>>>>> Hello!
>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>> instert it
>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of key
>>>>>>>> "msg".
>>>>>>>> So now is:
>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>> a i'd like to insert: only:
>>>>>>>> '{"foo","bar"}'
>>>>>>>> I tried with template:
>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>> but it doesn't do what I need. Is it possible to configure it
>>>>>>>> using
>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>
>>>>>>>> Marcin Miros?aw
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> <https://twitter.com/rgerhards>
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST if you DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> http://www.rsyslog.com/professional-services/
>>>>> <http://www.rsyslog.com/professional-services/>
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> <https://twitter.com/rgerhards>
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Looks relatively normal.

You can of course try writing to a file with your "i-json" template to
make sure that's what you want to be sent to mongodb.

But then, if your ommongodb action does contain the template="i-json"
parameter, it simply should work.

The only other thing you can do to make sure what's going on over the
wire is of course the tcpdump (unless your traffic is encrypted).

Ot simply looks that it should work - it should not need any more
"processing" as it is.

On 4.10.2022 15:45, Marcin Miros?aw wrote:
> :)
> I had to censore log...
>
>
> Debug line with all properties:
> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
> PRI: 174,
> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-', MSGID:
> '-',
> TIMESTAMP: 'Oct  4 15:38:53', STRUCTURED-DATA: '[timeQuality
> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
> syncAccuracy="415383"] {"foo":"bar
> ","mode":750,"date":"2022-10-04T15:38:53"}'
> $!:
> $.:
> $/:
>
>
>
>
> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>> :-)
>>
>> OK, I understand that you did
>>
>> template(name="i-json" type="string" string="%msg%")
>>
>> action(type="ommongodb" server="..." [...] template="i-json")
>>
>> right?
>>
>> This way you should indeed be pushing the %msg% part of the incoming
>> syslog message.
>>
>> Try adding
>>
>> action(type="omfile" file="/tmp/debug.log"
>> template="RSYSLOG_DebugFormat")
>>
>> immediately before your ommongodb action to see what exactly your
>> properties look like. That usually helps finding what's happening
>> inside your processing pipeline.
>>
>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>>
>>> I did in the part about connection to mongodb ;)
>>>
>>> "It will be used automatically if no other template is specified to
>>> be used"
>>>
>>> But I specified template: template(name="ui-json" type="string"
>>> string="%msg%")
>>>
>>> so if I understand qutoed docs correctly, all about default, canned
>>> template doesn't apply to my case, am I right?
>>>
>>>
>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>>
>>>> You didn't read the docs, did you? ;-)
>>>>
>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>
>>>>
>>>> "Note rsyslog contains a canned default template to write to the
>>>> MongoDB. It will be used automatically if no other template is
>>>> specified to be used. This template is:
>>>>
>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>> \\"%hostname%\\",
>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>
>>>> This creates the BSON document needed for MongoDB if no template is
>>>> specified. The default schema is aligned to CEE and project
>>>> lumberjack. As such, the field names are standard lumberjack field
>>>> names, and *not* rsyslog property names
>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>
>>>>
>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>> Don't know about this particular output module but in general
>>>>>> what you want is for rsyslog to parse the message and insert it
>>>>>> as json object.
>>>>>
>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>> ex: msg: '{"foo":"bar"}' }
>>>>> )
>>>>>
>>>>>
>>>>>> So you need to use parse_json() function on the input string and
>>>>>> then use proper template which will render the json to
>>>>>> appropirate string. I use similar approach (without the parsing
>>>>>> part) to create output json for Splunk's HEC input - the idea is
>>>>>> roughly the same.
>>>>>>
>>>>>>
>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>
>>>>>>> > db.log2.find()
>>>>>>> [.
>>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>> '{"foo":"bar"}' },
>>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>> ]
>>>>>>>
>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>> Second record is what I would like to get.
>>>>>>>
>>>>>>> rsyslog do:
>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>
>>>>>>> but I'd like to have:
>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>
>>>>>>> Marcin
>>>>>>>
>>>>>>>
>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>> I do not fully understand the question (maybe language issue on my
>>>>>>>> side), but there is a syntax error:
>>>>>>>>
>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>> sign. so:
>>>>>>>>
>>>>>>>> ... string="%msg%
>>>>>>>>
>>>>>>>> HTH
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>> instert it
>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of key
>>>>>>>>> "msg".
>>>>>>>>> So now is:
>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>> a i'd like to insert: only:
>>>>>>>>> '{"foo","bar"}'
>>>>>>>>> I tried with template:
>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>> but it doesn't do what I need. Is it possible to configure it
>>>>>>>>> using
>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>
>>>>>>>>> Marcin Miros?aw
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> <https://twitter.com/rgerhards>
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> <https://twitter.com/rgerhards>
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST if you DON'T LIKE THAT.
>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi!
If I use omfile then logs contains messages in my desired format but not
with ommongodb. tcpdump shows:

..msg.....{"foo":"bar"}..

Marcin


W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
> Looks relatively normal.
>
> You can of course try writing to a file with your "i-json" template to
> make sure that's what you want to be sent to mongodb.
>
> But then, if your ommongodb action does contain the template="i-json"
> parameter, it simply should work.
>
> The only other thing you can do to make sure what's going on over the
> wire is of course the tcpdump (unless your traffic is encrypted).
>
> Ot simply looks that it should work - it should not need any more
> "processing" as it is.
>
> On 4.10.2022 15:45, Marcin Miros?aw wrote:
>> :)
>> I had to censore log...
>>
>>
>> Debug line with all properties:
>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>> PRI: 174,
>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-', MSGID:
>> '-',
>> TIMESTAMP: 'Oct  4 15:38:53', STRUCTURED-DATA: '[timeQuality
>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>> escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>> syncAccuracy="415383"] {"foo":"bar
>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>> $!:
>> $.:
>> $/:
>>
>>
>>
>>
>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>>> :-)
>>>
>>> OK, I understand that you did
>>>
>>> template(name="i-json" type="string" string="%msg%")
>>>
>>> action(type="ommongodb" server="..." [...] template="i-json")
>>>
>>> right?
>>>
>>> This way you should indeed be pushing the %msg% part of the incoming
>>> syslog message.
>>>
>>> Try adding
>>>
>>> action(type="omfile" file="/tmp/debug.log"
>>> template="RSYSLOG_DebugFormat")
>>>
>>> immediately before your ommongodb action to see what exactly your
>>> properties look like. That usually helps finding what's happening
>>> inside your processing pipeline.
>>>
>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>>>
>>>> I did in the part about connection to mongodb ;)
>>>>
>>>> "It will be used automatically if no other template is specified to
>>>> be used"
>>>>
>>>> But I specified template: template(name="ui-json" type="string"
>>>> string="%msg%")
>>>>
>>>> so if I understand qutoed docs correctly, all about default, canned
>>>> template doesn't apply to my case, am I right?
>>>>
>>>>
>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>>>
>>>>> You didn't read the docs, did you? ;-)
>>>>>
>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>> "Note rsyslog contains a canned default template to write to the
>>>>> MongoDB. It will be used automatically if no other template is
>>>>> specified to be used. This template is:
>>>>>
>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>>> \\"%hostname%\\",
>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>>
>>>>> This creates the BSON document needed for MongoDB if no template is
>>>>> specified. The default schema is aligned to CEE and project
>>>>> lumberjack. As such, the field names are standard lumberjack field
>>>>> names, and *not* rsyslog property names
>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>>
>>>>>
>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>>> Don't know about this particular output module but in general
>>>>>>> what you want is for rsyslog to parse the message and insert it
>>>>>>> as json object.
>>>>>>
>>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>>> ex: msg: '{"foo":"bar"}' }
>>>>>> )
>>>>>>
>>>>>>
>>>>>>> So you need to use parse_json() function on the input string and
>>>>>>> then use proper template which will render the json to
>>>>>>> appropirate string. I use similar approach (without the parsing
>>>>>>> part) to create output json for Splunk's HEC input - the idea is
>>>>>>> roughly the same.
>>>>>>>
>>>>>>>
>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>>
>>>>>>>> > db.log2.find()
>>>>>>>> [.
>>>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>>> '{"foo":"bar"}' },
>>>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>>> ]
>>>>>>>>
>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>>> Second record is what I would like to get.
>>>>>>>>
>>>>>>>> rsyslog do:
>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>>
>>>>>>>> but I'd like to have:
>>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>>
>>>>>>>> Marcin
>>>>>>>>
>>>>>>>>
>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>>> I do not fully understand the question (maybe language issue on
>>>>>>>>> my
>>>>>>>>> side), but there is a syntax error:
>>>>>>>>>
>>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>>> sign. so:
>>>>>>>>>
>>>>>>>>> ... string="%msg%
>>>>>>>>>
>>>>>>>>> HTH
>>>>>>>>> Rainer
>>>>>>>>>
>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>
>>>>>>>>>> Hello!
>>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>>> instert it
>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of key
>>>>>>>>>> "msg".
>>>>>>>>>> So now is:
>>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>>> a i'd like to insert: only:
>>>>>>>>>> '{"foo","bar"}'
>>>>>>>>>> I tried with template:
>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>>> but it doesn't do what I need. Is it possible to configure it
>>>>>>>>>> using
>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>>
>>>>>>>>>> Marcin Miros?aw
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> <https://twitter.com/rgerhards>
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>>> POST if you DON'T LIKE THAT.
>>>>
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST
>>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Out of sheer curiosity I looked into the ommongodb.c and found this:

557     if(pData->tplName == NULL) {
558         doc = getDefaultBSON(*(smsg_t**)pMsgData);
559     } else {
560         doc = BSONFromJSONObject(*(struct json_object **)pMsgData);
561     }

I'm not that good in rsyslog internals but for me it seems as if the
module does not use the specified template at all. If the template is
not specified, it creates a BSON with pre-defined files. If the template
is specified, it just dumps a bson object from the message data.


On 5.10.2022 11:28, Marcin Miros?aw via rsyslog wrote:
> Hi!
> If I use omfile then logs contains messages in my desired format but
> not with ommongodb. tcpdump shows:
>
> ..msg.....{"foo":"bar"}..
>
> Marcin
>
>
> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
>> Looks relatively normal.
>>
>> You can of course try writing to a file with your "i-json" template to
>> make sure that's what you want to be sent to mongodb.
>>
>> But then, if your ommongodb action does contain the template="i-json"
>> parameter, it simply should work.
>>
>> The only other thing you can do to make sure what's going on over the
>> wire is of course the tcpdump (unless your traffic is encrypted).
>>
>> Ot simply looks that it should work - it should not need any more
>> "processing" as it is.
>>
>> On 4.10.2022 15:45, Marcin Miros?aw wrote:
>>> :)
>>> I had to censore log...
>>>
>>>
>>> Debug line with all properties:
>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>>> PRI: 174,
>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
>>> MSGID: '-',
>>> TIMESTAMP: 'Oct  4 15:38:53', STRUCTURED-DATA: '[timeQuality
>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>> escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>>> syncAccuracy="415383"] {"foo":"bar
>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>>> $!:
>>> $.:
>>> $/:
>>>
>>>
>>>
>>>
>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>>>> :-)
>>>>
>>>> OK, I understand that you did
>>>>
>>>> template(name="i-json" type="string" string="%msg%")
>>>>
>>>> action(type="ommongodb" server="..." [...] template="i-json")
>>>>
>>>> right?
>>>>
>>>> This way you should indeed be pushing the %msg% part of the incoming
>>>> syslog message.
>>>>
>>>> Try adding
>>>>
>>>> action(type="omfile" file="/tmp/debug.log"
>>>> template="RSYSLOG_DebugFormat")
>>>>
>>>> immediately before your ommongodb action to see what exactly your
>>>> properties look like. That usually helps finding what's happening
>>>> inside your processing pipeline.
>>>>
>>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>>>>
>>>>> I did in the part about connection to mongodb ;)
>>>>>
>>>>> "It will be used automatically if no other template is specified
>>>>> to be used"
>>>>>
>>>>> But I specified template: template(name="ui-json" type="string"
>>>>> string="%msg%")
>>>>>
>>>>> so if I understand qutoed docs correctly, all about default,
>>>>> canned template doesn't apply to my case, am I right?
>>>>>
>>>>>
>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>>>>
>>>>>> You didn't read the docs, did you? ;-)
>>>>>>
>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>>> "Note rsyslog contains a canned default template to write to the
>>>>>> MongoDB. It will be used automatically if no other template is
>>>>>> specified to be used. This template is:
>>>>>>
>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>>>> \\"%hostname%\\",
>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>>>
>>>>>> This creates the BSON document needed for MongoDB if no template
>>>>>> is specified. The default schema is aligned to CEE and project
>>>>>> lumberjack. As such, the field names are standard lumberjack
>>>>>> field names, and *not* rsyslog property names
>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>>>
>>>>>>
>>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>>>> Don't know about this particular output module but in general
>>>>>>>> what you want is for rsyslog to parse the message and insert it
>>>>>>>> as json object.
>>>>>>>
>>>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>>>> ex: msg: '{"foo":"bar"}' }
>>>>>>> )
>>>>>>>
>>>>>>>
>>>>>>>> So you need to use parse_json() function on the input string
>>>>>>>> and then use proper template which will render the json to
>>>>>>>> appropirate string. I use similar approach (without the parsing
>>>>>>>> part) to create output json for Splunk's HEC input - the idea
>>>>>>>> is roughly the same.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>>>
>>>>>>>>> > db.log2.find()
>>>>>>>>> [.
>>>>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>>>> '{"foo":"bar"}' },
>>>>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>>>> ]
>>>>>>>>>
>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>>>> Second record is what I would like to get.
>>>>>>>>>
>>>>>>>>> rsyslog do:
>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>>>
>>>>>>>>> but I'd like to have:
>>>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>>>
>>>>>>>>> Marcin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>>>> I do not fully understand the question (maybe language issue
>>>>>>>>>> on my
>>>>>>>>>> side), but there is a syntax error:
>>>>>>>>>>
>>>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>>>> sign. so:
>>>>>>>>>>
>>>>>>>>>> ... string="%msg%
>>>>>>>>>>
>>>>>>>>>> HTH
>>>>>>>>>> Rainer
>>>>>>>>>>
>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>> Hello!
>>>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>>>> instert it
>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
>>>>>>>>>>> key "msg".
>>>>>>>>>>> So now is:
>>>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>>>> a i'd like to insert: only:
>>>>>>>>>>> '{"foo","bar"}'
>>>>>>>>>>> I tried with template:
>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
>>>>>>>>>>> it using
>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>>>
>>>>>>>>>>> Marcin Miros?aw
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>>>>>> NOT POST if you DON'T LIKE THAT.
>>>>>
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

There is also some inconsistency within the documentation page (it lists
OMSR_TPL_AS_MSG as the default template whereas few lines later it shows
a definition of a template named BSON which is supposed to be default).

So I suspect that something wasn't quite finished here ;-)

On 5.10.2022 12:03, Mariusz Kruk wrote:
> Out of sheer curiosity I looked into the ommongodb.c and found this:
>
> 557     if(pData->tplName == NULL) {
> 558         doc = getDefaultBSON(*(smsg_t**)pMsgData);
> 559     } else {
> 560         doc = BSONFromJSONObject(*(struct json_object **)pMsgData);
> 561     }
>
> I'm not that good in rsyslog internals but for me it seems as if the
> module does not use the specified template at all. If the template is
> not specified, it creates a BSON with pre-defined files. If the
> template is specified, it just dumps a bson object from the message data.
>
>
> On 5.10.2022 11:28, Marcin Miros?aw via rsyslog wrote:
>> Hi!
>> If I use omfile then logs contains messages in my desired format but
>> not with ommongodb. tcpdump shows:
>>
>> ..msg.....{"foo":"bar"}..
>>
>> Marcin
>>
>>
>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
>>> Looks relatively normal.
>>>
>>> You can of course try writing to a file with your "i-json" template to
>>> make sure that's what you want to be sent to mongodb.
>>>
>>> But then, if your ommongodb action does contain the template="i-json"
>>> parameter, it simply should work.
>>>
>>> The only other thing you can do to make sure what's going on over the
>>> wire is of course the tcpdump (unless your traffic is encrypted).
>>>
>>> Ot simply looks that it should work - it should not need any more
>>> "processing" as it is.
>>>
>>> On 4.10.2022 15:45, Marcin Miros?aw wrote:
>>>> :)
>>>> I had to censore log...
>>>>
>>>>
>>>> Debug line with all properties:
>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>>>> PRI: 174,
>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
>>>> MSGID: '-',
>>>> TIMESTAMP: 'Oct  4 15:38:53', STRUCTURED-DATA: '[timeQuality
>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>> escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>>>> syncAccuracy="415383"] {"foo":"bar
>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>>>> $!:
>>>> $.:
>>>> $/:
>>>>
>>>>
>>>>
>>>>
>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>>>>> :-)
>>>>>
>>>>> OK, I understand that you did
>>>>>
>>>>> template(name="i-json" type="string" string="%msg%")
>>>>>
>>>>> action(type="ommongodb" server="..." [...] template="i-json")
>>>>>
>>>>> right?
>>>>>
>>>>> This way you should indeed be pushing the %msg% part of the incoming
>>>>> syslog message.
>>>>>
>>>>> Try adding
>>>>>
>>>>> action(type="omfile" file="/tmp/debug.log"
>>>>> template="RSYSLOG_DebugFormat")
>>>>>
>>>>> immediately before your ommongodb action to see what exactly your
>>>>> properties look like. That usually helps finding what's happening
>>>>> inside your processing pipeline.
>>>>>
>>>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>>>>>
>>>>>> I did in the part about connection to mongodb ;)
>>>>>>
>>>>>> "It will be used automatically if no other template is specified
>>>>>> to be used"
>>>>>>
>>>>>> But I specified template: template(name="ui-json" type="string"
>>>>>> string="%msg%")
>>>>>>
>>>>>> so if I understand qutoed docs correctly, all about default,
>>>>>> canned template doesn't apply to my case, am I right?
>>>>>>
>>>>>>
>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>>>>>
>>>>>>> You didn't read the docs, did you? ;-)
>>>>>>>
>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>>>> "Note rsyslog contains a canned default template to write to the
>>>>>>> MongoDB. It will be used automatically if no other template is
>>>>>>> specified to be used. This template is:
>>>>>>>
>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>>>>> \\"%hostname%\\",
>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>>>>
>>>>>>> This creates the BSON document needed for MongoDB if no template
>>>>>>> is specified. The default schema is aligned to CEE and project
>>>>>>> lumberjack. As such, the field names are standard lumberjack
>>>>>>> field names, and *not* rsyslog property names
>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>>>>
>>>>>>>
>>>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>>>>> Don't know about this particular output module but in general
>>>>>>>>> what you want is for rsyslog to parse the message and insert
>>>>>>>>> it as json object.
>>>>>>>>
>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>>>>> ex: msg: '{"foo":"bar"}' }
>>>>>>>> )
>>>>>>>>
>>>>>>>>
>>>>>>>>> So you need to use parse_json() function on the input string
>>>>>>>>> and then use proper template which will render the json to
>>>>>>>>> appropirate string. I use similar approach (without the
>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
>>>>>>>>> the idea is roughly the same.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>>>>
>>>>>>>>>> > db.log2.find()
>>>>>>>>>> [.
>>>>>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>>>>> '{"foo":"bar"}' },
>>>>>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>>>>> ]
>>>>>>>>>>
>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>>>>> Second record is what I would like to get.
>>>>>>>>>>
>>>>>>>>>> rsyslog do:
>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>>>>
>>>>>>>>>> but I'd like to have:
>>>>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>>>>
>>>>>>>>>> Marcin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>>>>> I do not fully understand the question (maybe language issue
>>>>>>>>>>> on my
>>>>>>>>>>> side), but there is a syntax error:
>>>>>>>>>>>
>>>>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>>>>> sign. so:
>>>>>>>>>>>
>>>>>>>>>>> ... string="%msg%
>>>>>>>>>>>
>>>>>>>>>>> HTH
>>>>>>>>>>> Rainer
>>>>>>>>>>>
>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello!
>>>>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>>>>> instert it
>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
>>>>>>>>>>>> key "msg".
>>>>>>>>>>>> So now is:
>>>>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>>>>> a i'd like to insert: only:
>>>>>>>>>>>> '{"foo","bar"}'
>>>>>>>>>>>> I tried with template:
>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
>>>>>>>>>>>> it using
>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>>>>
>>>>>>>>>>>> Marcin Miros?aw
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>> https://twitter.com/rgerhards <https://twitter.com/rgerhards>
>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST
>>>>> if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

It would be easier to fix if it would be my misconfiguration :(
Rainer, would you look into ommongodb module, please?
Marcin

W dniu 2022-10-05 12:20, Mariusz Kruk via rsyslog napisa?(a):
> There is also some inconsistency within the documentation page (it
> lists OMSR_TPL_AS_MSG as the default template whereas few lines later
> it shows a definition of a template named BSON which is supposed to be
> default).
>
> So I suspect that something wasn't quite finished here ;-)
>
> On 5.10.2022 12:03, Mariusz Kruk wrote:
>> Out of sheer curiosity I looked into the ommongodb.c and found this:
>>
>> 557     if(pData->tplName == NULL) {
>> 558         doc = getDefaultBSON(*(smsg_t**)pMsgData);
>> 559     } else {
>> 560         doc = BSONFromJSONObject(*(struct json_object
>> **)pMsgData);
>> 561     }
>>
>> I'm not that good in rsyslog internals but for me it seems as if the
>> module does not use the specified template at all. If the template is
>> not specified, it creates a BSON with pre-defined files. If the
>> template is specified, it just dumps a bson object from the message
>> data.
>>
>>
>> On 5.10.2022 11:28, Marcin Miros?aw via rsyslog wrote:
>>> Hi!
>>> If I use omfile then logs contains messages in my desired format but
>>> not with ommongodb. tcpdump shows:
>>>
>>> ..msg.....{"foo":"bar"}..
>>>
>>> Marcin
>>>
>>>
>>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
>>>> Looks relatively normal.
>>>>
>>>> You can of course try writing to a file with your "i-json" template
>>>> to
>>>> make sure that's what you want to be sent to mongodb.
>>>>
>>>> But then, if your ommongodb action does contain the
>>>> template="i-json"
>>>> parameter, it simply should work.
>>>>
>>>> The only other thing you can do to make sure what's going on over
>>>> the
>>>> wire is of course the tcpdump (unless your traffic is encrypted).
>>>>
>>>> Ot simply looks that it should work - it should not need any more
>>>> "processing" as it is.
>>>>
>>>> On 4.10.2022 15:45, Marcin Miros?aw wrote:
>>>>> :)
>>>>> I had to censore log...
>>>>>
>>>>>
>>>>> Debug line with all properties:
>>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>>>>> PRI: 174,
>>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
>>>>> MSGID: '-',
>>>>> TIMESTAMP: 'Oct  4 15:38:53', STRUCTURED-DATA: '[timeQuality
>>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> escaped msg:
>>>>> '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>>>>> syncAccuracy="415383"] {"foo":"bar
>>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> $!:
>>>>> $.:
>>>>> $/:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>>>>>> :-)
>>>>>>
>>>>>> OK, I understand that you did
>>>>>>
>>>>>> template(name="i-json" type="string" string="%msg%")
>>>>>>
>>>>>> action(type="ommongodb" server="..." [...] template="i-json")
>>>>>>
>>>>>> right?
>>>>>>
>>>>>> This way you should indeed be pushing the %msg% part of the
>>>>>> incoming
>>>>>> syslog message.
>>>>>>
>>>>>> Try adding
>>>>>>
>>>>>> action(type="omfile" file="/tmp/debug.log"
>>>>>> template="RSYSLOG_DebugFormat")
>>>>>>
>>>>>> immediately before your ommongodb action to see what exactly your
>>>>>> properties look like. That usually helps finding what's happening
>>>>>> inside your processing pipeline.
>>>>>>
>>>>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>>>>>>>
>>>>>>> I did in the part about connection to mongodb ;)
>>>>>>>
>>>>>>> "It will be used automatically if no other template is specified
>>>>>>> to be used"
>>>>>>>
>>>>>>> But I specified template: template(name="ui-json" type="string"
>>>>>>> string="%msg%")
>>>>>>>
>>>>>>> so if I understand qutoed docs correctly, all about default,
>>>>>>> canned template doesn't apply to my case, am I right?
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>>>>>>>
>>>>>>>> You didn't read the docs, did you? ;-)
>>>>>>>>
>>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>>>>> "Note rsyslog contains a canned default template to write to the
>>>>>>>> MongoDB. It will be used automatically if no other template is
>>>>>>>> specified to be used. This template is:
>>>>>>>>
>>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>>>>>> \\"%hostname%\\",
>>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\"
>>>>>>>> :
>>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>>>>>
>>>>>>>> This creates the BSON document needed for MongoDB if no template
>>>>>>>> is specified. The default schema is aligned to CEE and project
>>>>>>>> lumberjack. As such, the field names are standard lumberjack
>>>>>>>> field names, and *not* rsyslog property names
>>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>>>>>> Don't know about this particular output module but in general
>>>>>>>>>> what you want is for rsyslog to parse the message and insert
>>>>>>>>>> it as json object.
>>>>>>>>>
>>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>>>>>> ex: msg: '{"foo":"bar"}' }
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> So you need to use parse_json() function on the input string
>>>>>>>>>> and then use proper template which will render the json to
>>>>>>>>>> appropirate string. I use similar approach (without the
>>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
>>>>>>>>>> the idea is roughly the same.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>>>>>
>>>>>>>>>>> > db.log2.find()
>>>>>>>>>>> [.
>>>>>>>>>>>   { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>>>>>> '{"foo":"bar"}' },
>>>>>>>>>>>   { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>>>>>> ]
>>>>>>>>>>>
>>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>>>>>> Second record is what I would like to get.
>>>>>>>>>>>
>>>>>>>>>>> rsyslog do:
>>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>>>>>
>>>>>>>>>>> but I'd like to have:
>>>>>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>>>>>
>>>>>>>>>>> Marcin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>>>>>> I do not fully understand the question (maybe language issue
>>>>>>>>>>>> on my
>>>>>>>>>>>> side), but there is a syntax error:
>>>>>>>>>>>>
>>>>>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>>>>>> sign. so:
>>>>>>>>>>>>
>>>>>>>>>>>> ... string="%msg%
>>>>>>>>>>>>
>>>>>>>>>>>> HTH
>>>>>>>>>>>> Rainer
>>>>>>>>>>>>
>>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>>>>>> instert it
>>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
>>>>>>>>>>>>> key "msg".
>>>>>>>>>>>>> So now is:
>>>>>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>>>>>> a i'd like to insert: only:
>>>>>>>>>>>>> '{"foo","bar"}'
>>>>>>>>>>>>> I tried with template:
>>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
>>>>>>>>>>>>> it using
>>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marcin Miros?aw
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>> https://twitter.com/rgerhards
>>>>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
>>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST
>>>>>> if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST
>>>> if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

It's contributed code.

Albeit I have done some improvements, I need to familiarize myself
again with the module. IAW: it will take some time ;-)

Rainer

El mié, 5 oct 2022 a las 16:04, Marcin Miros?aw via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> It would be easier to fix if it would be my misconfiguration :(
> Rainer, would you look into ommongodb module, please?
> Marcin
>
> W dniu 2022-10-05 12:20, Mariusz Kruk via rsyslog napisa?(a):
> > There is also some inconsistency within the documentation page (it
> > lists OMSR_TPL_AS_MSG as the default template whereas few lines later
> > it shows a definition of a template named BSON which is supposed to be
> > default).
> >
> > So I suspect that something wasn't quite finished here ;-)
> >
> > On 5.10.2022 12:03, Mariusz Kruk wrote:
> >> Out of sheer curiosity I looked into the ommongodb.c and found this:
> >>
> >> 557 if(pData->tplName == NULL) {
> >> 558 doc = getDefaultBSON(*(smsg_t**)pMsgData);
> >> 559 } else {
> >> 560 doc = BSONFromJSONObject(*(struct json_object
> >> **)pMsgData);
> >> 561 }
> >>
> >> I'm not that good in rsyslog internals but for me it seems as if the
> >> module does not use the specified template at all. If the template is
> >> not specified, it creates a BSON with pre-defined files. If the
> >> template is specified, it just dumps a bson object from the message
> >> data.
> >>
> >>
> >> On 5.10.2022 11:28, Marcin Miros?aw via rsyslog wrote:
> >>> Hi!
> >>> If I use omfile then logs contains messages in my desired format but
> >>> not with ommongodb. tcpdump shows:
> >>>
> >>> ..msg.....{"foo":"bar"}..
> >>>
> >>> Marcin
> >>>
> >>>
> >>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
> >>>> Looks relatively normal.
> >>>>
> >>>> You can of course try writing to a file with your "i-json" template
> >>>> to
> >>>> make sure that's what you want to be sent to mongodb.
> >>>>
> >>>> But then, if your ommongodb action does contain the
> >>>> template="i-json"
> >>>> parameter, it simply should work.
> >>>>
> >>>> The only other thing you can do to make sure what's going on over
> >>>> the
> >>>> wire is of course the tcpdump (unless your traffic is encrypted).
> >>>>
> >>>> Ot simply looks that it should work - it should not need any more
> >>>> "processing" as it is.
> >>>>
> >>>> On 4.10.2022 15:45, Marcin Miros?aw wrote:
> >>>>> :)
> >>>>> I had to censore log...
> >>>>>
> >>>>>
> >>>>> Debug line with all properties:
> >>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
> >>>>> PRI: 174,
> >>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
> >>>>> MSGID: '-',
> >>>>> TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality
> >>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
> >>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> escaped msg:
> >>>>> '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
> >>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
> >>>>> syncAccuracy="415383"] {"foo":"bar
> >>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> $!:
> >>>>> $.:
> >>>>> $/:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
> >>>>>> :-)
> >>>>>>
> >>>>>> OK, I understand that you did
> >>>>>>
> >>>>>> template(name="i-json" type="string" string="%msg%")
> >>>>>>
> >>>>>> action(type="ommongodb" server="..." [...] template="i-json")
> >>>>>>
> >>>>>> right?
> >>>>>>
> >>>>>> This way you should indeed be pushing the %msg% part of the
> >>>>>> incoming
> >>>>>> syslog message.
> >>>>>>
> >>>>>> Try adding
> >>>>>>
> >>>>>> action(type="omfile" file="/tmp/debug.log"
> >>>>>> template="RSYSLOG_DebugFormat")
> >>>>>>
> >>>>>> immediately before your ommongodb action to see what exactly your
> >>>>>> properties look like. That usually helps finding what's happening
> >>>>>> inside your processing pipeline.
> >>>>>>
> >>>>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
> >>>>>>>
> >>>>>>> I did in the part about connection to mongodb ;)
> >>>>>>>
> >>>>>>> "It will be used automatically if no other template is specified
> >>>>>>> to be used"
> >>>>>>>
> >>>>>>> But I specified template: template(name="ui-json" type="string"
> >>>>>>> string="%msg%")
> >>>>>>>
> >>>>>>> so if I understand qutoed docs correctly, all about default,
> >>>>>>> canned template doesn't apply to my case, am I right?
> >>>>>>>
> >>>>>>>
> >>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
> >>>>>>>
> >>>>>>>> You didn't read the docs, did you? ;-)
> >>>>>>>>
> >>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
> >>>>>>>> "Note rsyslog contains a canned default template to write to the
> >>>>>>>> MongoDB. It will be used automatically if no other template is
> >>>>>>>> specified to be used. This template is:
> >>>>>>>>
> >>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
> >>>>>>>> \\"%hostname%\\",
> >>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
> >>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
> >>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\"
> >>>>>>>> :
> >>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
> >>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
> >>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
> >>>>>>>>
> >>>>>>>> This creates the BSON document needed for MongoDB if no template
> >>>>>>>> is specified. The default schema is aligned to CEE and project
> >>>>>>>> lumberjack. As such, the field names are standard lumberjack
> >>>>>>>> field names, and *not* rsyslog property names
> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
> >>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
> >>>>>>>>>> Don't know about this particular output module but in general
> >>>>>>>>>> what you want is for rsyslog to parse the message and insert
> >>>>>>>>>> it as json object.
> >>>>>>>>>
> >>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
> >>>>>>>>> ex: msg: '{"foo":"bar"}' }
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> So you need to use parse_json() function on the input string
> >>>>>>>>>> and then use proper template which will render the json to
> >>>>>>>>>> appropirate string. I use similar approach (without the
> >>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
> >>>>>>>>>> the idea is roughly the same.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
> >>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
> >>>>>>>>>>>
> >>>>>>>>>>> > db.log2.find()
> >>>>>>>>>>> [.
> >>>>>>>>>>> { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
> >>>>>>>>>>> '{"foo":"bar"}' },
> >>>>>>>>>>> { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
> >>>>>>>>>>> ]
> >>>>>>>>>>>
> >>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
> >>>>>>>>>>> Second record is what I would like to get.
> >>>>>>>>>>>
> >>>>>>>>>>> rsyslog do:
> >>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
> >>>>>>>>>>>
> >>>>>>>>>>> but I'd like to have:
> >>>>>>>>>>> db.log2.insert({"foo":"bar"})
> >>>>>>>>>>>
> >>>>>>>>>>> Marcin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
> >>>>>>>>>>>> I do not fully understand the question (maybe language issue
> >>>>>>>>>>>> on my
> >>>>>>>>>>>> side), but there is a syntax error:
> >>>>>>>>>>>>
> >>>>>>>>>>>> In a string template, properties must be enclosed in percent
> >>>>>>>>>>>> sign. so:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ... string="%msg%
> >>>>>>>>>>>>
> >>>>>>>>>>>> HTH
> >>>>>>>>>>>> Rainer
> >>>>>>>>>>>>
> >>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
> >>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
> >>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello!
> >>>>>>>>>>>>> Field msg contains complete json with data. I would like to
> >>>>>>>>>>>>> instert it
> >>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
> >>>>>>>>>>>>> key "msg".
> >>>>>>>>>>>>> So now is:
> >>>>>>>>>>>>> msg: '{"foo":"bar"}
> >>>>>>>>>>>>> a i'd like to insert: only:
> >>>>>>>>>>>>> '{"foo","bar"}'
> >>>>>>>>>>>>> I tried with template:
> >>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
> >>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
> >>>>>>>>>>>>> it using
> >>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Marcin Miros?aw
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>> https://twitter.com/rgerhards
> >>>>>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> >>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
> >>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
> >>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
> >>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com/professional-services/
> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>>>>> POST
> >>>>>> if you DON'T LIKE THAT.
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>>> POST
> >>>> if you DON'T LIKE THAT.
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>> POST if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi!
Rainer, Mariusz, thank you for your help.
Marcin


W dniu 2022-10-06 16:39, Rainer Gerhards napisa?(a):
> It's contributed code.
>
> Albeit I have done some improvements, I need to familiarize myself
> again with the module. IAW: it will take some time ;-)
>
> Rainer
>
> El mié, 5 oct 2022 a las 16:04, Marcin Miros?aw via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> It would be easier to fix if it would be my misconfiguration :(
>> Rainer, would you look into ommongodb module, please?
>> Marcin
>>
>> W dniu 2022-10-05 12:20, Mariusz Kruk via rsyslog napisa?(a):
>> > There is also some inconsistency within the documentation page (it
>> > lists OMSR_TPL_AS_MSG as the default template whereas few lines later
>> > it shows a definition of a template named BSON which is supposed to be
>> > default).
>> >
>> > So I suspect that something wasn't quite finished here ;-)
>> >
>> > On 5.10.2022 12:03, Mariusz Kruk wrote:
>> >> Out of sheer curiosity I looked into the ommongodb.c and found this:
>> >>
>> >> 557 if(pData->tplName == NULL) {
>> >> 558 doc = getDefaultBSON(*(smsg_t**)pMsgData);
>> >> 559 } else {
>> >> 560 doc = BSONFromJSONObject(*(struct json_object
>> >> **)pMsgData);
>> >> 561 }
>> >>
>> >> I'm not that good in rsyslog internals but for me it seems as if the
>> >> module does not use the specified template at all. If the template is
>> >> not specified, it creates a BSON with pre-defined files. If the
>> >> template is specified, it just dumps a bson object from the message
>> >> data.
>> >>
>> >>
>> >> On 5.10.2022 11:28, Marcin Miros?aw via rsyslog wrote:
>> >>> Hi!
>> >>> If I use omfile then logs contains messages in my desired format but
>> >>> not with ommongodb. tcpdump shows:
>> >>>
>> >>> ..msg.....{"foo":"bar"}..
>> >>>
>> >>> Marcin
>> >>>
>> >>>
>> >>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisa?(a):
>> >>>> Looks relatively normal.
>> >>>>
>> >>>> You can of course try writing to a file with your "i-json" template
>> >>>> to
>> >>>> make sure that's what you want to be sent to mongodb.
>> >>>>
>> >>>> But then, if your ommongodb action does contain the
>> >>>> template="i-json"
>> >>>> parameter, it simply should work.
>> >>>>
>> >>>> The only other thing you can do to make sure what's going on over
>> >>>> the
>> >>>> wire is of course the tcpdump (unless your traffic is encrypted).
>> >>>>
>> >>>> Ot simply looks that it should work - it should not need any more
>> >>>> "processing" as it is.
>> >>>>
>> >>>> On 4.10.2022 15:45, Marcin Miros?aw wrote:
>> >>>>> :)
>> >>>>> I had to censore log...
>> >>>>>
>> >>>>>
>> >>>>> Debug line with all properties:
>> >>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>> >>>>> PRI: 174,
>> >>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
>> >>>>> MSGID: '-',
>> >>>>> TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality
>> >>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>> >>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>> >>>>> escaped msg:
>> >>>>> '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>> >>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>> >>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>> >>>>> syncAccuracy="415383"] {"foo":"bar
>> >>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>> >>>>> $!:
>> >>>>> $.:
>> >>>>> $/:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisa?(a):
>> >>>>>> :-)
>> >>>>>>
>> >>>>>> OK, I understand that you did
>> >>>>>>
>> >>>>>> template(name="i-json" type="string" string="%msg%")
>> >>>>>>
>> >>>>>> action(type="ommongodb" server="..." [...] template="i-json")
>> >>>>>>
>> >>>>>> right?
>> >>>>>>
>> >>>>>> This way you should indeed be pushing the %msg% part of the
>> >>>>>> incoming
>> >>>>>> syslog message.
>> >>>>>>
>> >>>>>> Try adding
>> >>>>>>
>> >>>>>> action(type="omfile" file="/tmp/debug.log"
>> >>>>>> template="RSYSLOG_DebugFormat")
>> >>>>>>
>> >>>>>> immediately before your ommongodb action to see what exactly your
>> >>>>>> properties look like. That usually helps finding what's happening
>> >>>>>> inside your processing pipeline.
>> >>>>>>
>> >>>>>> On 4.10.2022 09:03, Marcin Miros?aw wrote:
>> >>>>>>>
>> >>>>>>> I did in the part about connection to mongodb ;)
>> >>>>>>>
>> >>>>>>> "It will be used automatically if no other template is specified
>> >>>>>>> to be used"
>> >>>>>>>
>> >>>>>>> But I specified template: template(name="ui-json" type="string"
>> >>>>>>> string="%msg%")
>> >>>>>>>
>> >>>>>>> so if I understand qutoed docs correctly, all about default,
>> >>>>>>> canned template doesn't apply to my case, am I right?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisa?(a):
>> >>>>>>>
>> >>>>>>>> You didn't read the docs, did you? ;-)
>> >>>>>>>>
>> >>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>> >>>>>>>> "Note rsyslog contains a canned default template to write to the
>> >>>>>>>> MongoDB. It will be used automatically if no other template is
>> >>>>>>>> specified to be used. This template is:
>> >>>>>>>>
>> >>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>> >>>>>>>> \\"%hostname%\\",
>> >>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>> >>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>> >>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\"
>> >>>>>>>> :
>> >>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>> >>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>> >>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>> >>>>>>>>
>> >>>>>>>> This creates the BSON document needed for MongoDB if no template
>> >>>>>>>> is specified. The default schema is aligned to CEE and project
>> >>>>>>>> lumberjack. As such, the field names are standard lumberjack
>> >>>>>>>> field names, and *not* rsyslog property names
>> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 3.10.2022 22:02, Marcin Miros?aw wrote:
>> >>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>> >>>>>>>>>> Don't know about this particular output module but in general
>> >>>>>>>>>> what you want is for rsyslog to parse the message and insert
>> >>>>>>>>>> it as json object.
>> >>>>>>>>>
>> >>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
>> >>>>>>>>> ex: msg: '{"foo":"bar"}' }
>> >>>>>>>>> )
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> So you need to use parse_json() function on the input string
>> >>>>>>>>>> and then use proper template which will render the json to
>> >>>>>>>>>> appropirate string. I use similar approach (without the
>> >>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
>> >>>>>>>>>> the idea is roughly the same.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 3.10.2022 18:35, Marcin Miros?aw via rsyslog wrote:
>> >>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>> >>>>>>>>>>>
>> >>>>>>>>>>> > db.log2.find()
>> >>>>>>>>>>> [.
>> >>>>>>>>>>> { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>> >>>>>>>>>>> '{"foo":"bar"}' },
>> >>>>>>>>>>> { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>> >>>>>>>>>>> ]
>> >>>>>>>>>>>
>> >>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>> >>>>>>>>>>> Second record is what I would like to get.
>> >>>>>>>>>>>
>> >>>>>>>>>>> rsyslog do:
>> >>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>> >>>>>>>>>>>
>> >>>>>>>>>>> but I'd like to have:
>> >>>>>>>>>>> db.log2.insert({"foo":"bar"})
>> >>>>>>>>>>>
>> >>>>>>>>>>> Marcin
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>> >>>>>>>>>>>> I do not fully understand the question (maybe language issue
>> >>>>>>>>>>>> on my
>> >>>>>>>>>>>> side), but there is a syntax error:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> In a string template, properties must be enclosed in percent
>> >>>>>>>>>>>> sign. so:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> ... string="%msg%
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> HTH
>> >>>>>>>>>>>> Rainer
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Miros?aw via rsyslog
>> >>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
>> >>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hello!
>> >>>>>>>>>>>>> Field msg contains complete json with data. I would like to
>> >>>>>>>>>>>>> instert it
>> >>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
>> >>>>>>>>>>>>> key "msg".
>> >>>>>>>>>>>>> So now is:
>> >>>>>>>>>>>>> msg: '{"foo":"bar"}
>> >>>>>>>>>>>>> a i'd like to insert: only:
>> >>>>>>>>>>>>> '{"foo","bar"}'
>> >>>>>>>>>>>>> I tried with template:
>> >>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>> >>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
>> >>>>>>>>>>>>> it using
>> >>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Marcin Miros?aw
>> >>>>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>> >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>> >>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>> >>>>>>>>>>>>> What's up with rsyslog? Follow
>> >>>>>>>>>>>>> https://twitter.com/rgerhards
>> >>>>>>>>>>>>> <https://twitter.com/rgerhards>
>> >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>> >>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
>> >>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> >>>>>>>>>>>
>> >>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>> >>>>>>>>>>> http://www.rsyslog.com/professional-services/
>> >>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>> >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>>>>>>> <https://twitter.com/rgerhards>
>> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> >>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>> >>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>> >>>>>>>>>> _______________________________________________
>> >>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>> >>>>>>>>>> http://www.rsyslog.com/professional-services/
>> >>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>>>>>> <https://twitter.com/rgerhards>
>> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> >>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>> >>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>> >>>>>>>
>> >>>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> rsyslog mailing list
>> >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> http://www.rsyslog.com/professional-services/
>> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >>>>>> POST
>> >>>>>> if you DON'T LIKE THAT.
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com/professional-services/
>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >>>> POST
>> >>>> if you DON'T LIKE THAT.
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com/professional-services/
>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >>> POST if you DON'T LIKE THAT.
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> > if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.