This email may contain proprietary information of BAE Systems and/or third parties.
Fixed it, I think there was some confusion over the template and quotes and formatting the message field, so the end of the template should be:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"message\":") property(name="msg")
constant(value="}\n")
}
Not:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"message\":\"") property(name="msg" format="jsonf") #<--remove /" & format
constant(value="\"}\n") #<--remove /"
}
'Tis now working perfectly, thank you all.
Sean.
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Lennon, Sean (UK) via rsyslog
Sent: 20 September 2023 13:21
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped quotations and additional quotations
----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.
------------------------------------------------------------------------------------
This email may contain proprietary information of BAE Systems and/or third parties.
Hi Simon,
Yes, I did during early development of the mmexternal code, I tried all the combinations. However, all of this Linux dev is new to me and there is so much to take in, so given that there were a lot of unknowns a few weeks ago I may have missed something, so I will go through it again this afternoon and check out those other options.
Cheers,
Sean.
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Simon Lundström via rsyslog
Sent: 20 September 2023 12:44
To: rsyslog@lists.adiscon.com
Cc: Simon Lundström <simlu@su.se>
Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped quotations and additional quotations
----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.
------------------------------------------------------------------------------------
That's excellent Sean!
Have you tried using any of the other json formats on your template:
https://rsyslog.readthedocs.io/en/latest/configuration/templates.html#property-statement template(name="json-template" type="list" option.jsonf="on") {
property(outname="@timestamp" name="timereported" dataformat="rfc3339" format="jsonf")
property(outname="message" name="msg" format="jsonf") }
So format="json" and so on?
BR,
- Simon
On Wed, 2023-09-20 at 13:22:09 +0200, Lennon, Sean (UK) wrote:
>
>
>
>
> This email may contain proprietary information of BAE Systems and/or third parties.
>
> Rainer/David/Simon,
>
> Thanks for your help, I have done what you have requested/suggested:
>
> The following are redacted versions of the shortest message I have. I've used David's suggestion of using sed to process the messages to remove sensitive data, I've opted to change numbers to 0. I have left some inconsequential fields un-redacted for clarity. These messages were captured using tcpdump as suggested by Simon, therefore they are what rsyslog puts out to logstash.
>
> The first message is what I've settled on for the time being, i.e. logstash will accept it and we can get logstash to process it to remove the quotes around the value for the message field and also remove the escape characters. It's a bit of a fudge and not ideal, but it 'works'.
> {
> "@timestamp":"2023-09-20T11:06:03.571548+01:00",
> "@version":"1",
> "sysloghost":"XXXXXXXXXXXXXXX",
> "severity":"info",
> "facility":"local6",
> "message":"{
> \"XXXX\": [.
> {
> \"XXXXXXXXXXX\":0,
> \"XXXXXXX\":0,
> \"XXXXXXXXXXXXX\": {
> \"XXXXXX\": \"XXXXXXXXXXXXXXXX\",
> \"XXXXXXXXXXXXX\": {
> \"XXXX_XXXXXX\": \"XXXX_XXXXXXX\",
> \"XXXX_XXXXXX_XXXX\": 0
> }
> }
> }
> ]
> }"
> }
>
> The second message is produced after removing format="jsonf" from the message field in the template. Note that the escaped characters have gone but the quotes around the message field value have not. This causes logstash to barf, hence the previous message is the one being used.
> {
> "@timestamp":"2023-09-20T11:16:14.159361+01:00",
> "@version":"1",
> "sysloghost":"XXXXXXXXXXXXXXX",
> "severity":"info",
> "facility":"local6",
> "message":"{
> "XXXX": [.
> {
> "XXXXXXXXXXX":0,
> "XXXXXXX":0,
> "XXXXXXXXXXXXX": {
> "XXXXXX": "XXXXXXXXXXXXXXXX",
> "XXXXXXXXXXXXX": {
> "XXXX_XXXXXX": "XXXX_XXXXXXX",
> "XXXX_XXXXXX_XXXX": 0
> }
> }
> }
> ]
> }"
> }
>
> So, how do I get rid of the quotes around the message value?
>
> Cheers,
>
> Sean.
>
> -----Original Message-----
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Sent: 19 September 2023 08:51
> To: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> escaped quotations and additional quotations
>
> ----------------------------- PHISHING ALERT
> ----------------------------- This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
> For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.
>
> ----------------------------------------------------------------------
> --------------
>
> It's not easy to guess what rsyslog really sees, but I have one final
> shot. As it looks, msg might already contain json. In that case, do
> not use any special json formatting option. These options exist to
> ensure non-json data (or json data as an inner encapsulation layer)
> will be received as-is.
>
> Rainer
>
> El lun, 18 sept 2023 a las 17:04, Lennon, Sean (UK)
> (<sean.lennon2@baesystems.com>) escribió:
> >
> > This email may contain proprietary information of BAE Systems and/or third parties.
> >
> >
> >
> > Sorry, but for ‘reasons’ I can only give you a severely edited version, I have used debug output from mmexternal first and the received message from logstash second:
> >
> >
> >
> > 1. mexternal debug output – I am satisfied with this.
> > { “msg” :
> > {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
> >
> > 2. what logstash receives
> > “message” => “{ \“msg\” : {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
> >
> >
> >
> >
> >
> > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Sent: 18 September 2023 15:47
> > To: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> >
> >
> >
> > PHISHING ALERT
> >
> > This email has been sent from an account outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please access the Global Intranet then select <Functions> / <IT>.
> > If you think this is a phishing email, please report it by using the "Report Phishing" button in Outlook.
> >
> >
> >
> >
> >
> > Output the message with RSYSLOG_DebugFormat template. I need to see which data msg actually has.
> >
> >
> >
> > Rainer
> >
> > Sent from phone, thus brief.
> >
> >
> >
> > Lennon, Sean (UK) <sean.lennon2@baesystems.com> schrieb am Mo., 18. Sept. 2023, 16:41:
> >
> >
> >
> >
> >
> > This email may contain proprietary information of BAE Systems and/or third parties.
> >
> > Thanks for your response Rainer. I don't think it answers my question, I have property fields from the Rsyslog message that are fine, they get formatted correctly, for example 'timereported' or 'syslogseverity-text'. So, the output json for these and others are correct, it's the msg field that is returned from my custom code (using mmexternal) that is the problem.
> >
> > I have created a newer template that is more upto date and looks something similar to this:
> >
> > template(name="json-template" type="list" option.jsonf="on") {
> > property(outname="@timestamp" name="timereported" dataformat="rfc3339" format="jsonf")
> > property(outname="message" name="msg" format="jsonf") }
> >
> > -----Original Message-----
> > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Sent: 18 September 2023 15:26
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> > ----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.
> >
> > --------------------------------------------------------------------
> > ----------------
> >
> > Does this example from the rsyslog testbench help?
> >
> > https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff
> > 11ff8be3990e/tests/json-nonstring.sh#L4
> >
> > Rainer
> >
> > El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > >
> > >
> > >
> > >
> > > This email may contain proprietary information of BAE Systems and/or third parties.
> > >
> > > This is the one I meant.
> > >
> > > -----Original Message-----
> > > From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
> > > Lennon, Sean (UK) via rsyslog
> > > Sent: 29 August 2023 17:39
> > > To: rsyslog@lists.adiscon.com
> > > Cc: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
> > > Subject: [rsyslog] rsyslog mmextenal logstash json output with
> > > escaped quotations and additional quotations
> > >
> > > ----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.
> > >
> > > Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
> > > For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.
> > >
> > > ------------------------------------------------------------------
> > > ----
> > > --------------
> > >
> > > This email may contain proprietary information of BAE Systems and/or third parties.
> > >
> > > Hi all,
> > >
> > > I've encountered an issue with formatting json output to logstash. I'm using mmexternal to reformat data received from a remote system, the data is project specific and needs to be massaged into json for use with logstash. The intention is to create a json message for logstash with the mmexternal output being part of that message. I'm able to receive this json output at logstash but the message field (which contains the mmexternal output) is encapsulated within double quotes and all json fields within have escaped double quotes. This means that logstash is not able to interpret part of the message. If I take the raw output of the mmextenal code and send it to a omfile then it looks perfectly fine.
> > >
> > > I have asked a more detailed question, on Stackoverflow:
> > > https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-log
> > > stas h-json-output-with-escaped-quotations-and-additional-qu
> > >
> > > What am I missing?
> > >
> > > I appreciate your help.
> > >
> > > Sean
> > >
> > > ******************************************************************
> > > ** This email and any attachments are confidential to the intended
> > > recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender.
> > > You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.
> > > ******************************************************************
> > > **
> > >
> > > BAE Systems may process information about you that may be subject
> > > to data protection laws. For more information about how we use
> > > your personal information, how we protect your information, our
> > > legal basis for using your information, your rights and who you
> > > can contact, please refer to our Privacy Notice at
> > > www.baesystems.com/en/privacy
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.