Mailing List Archive: Input module for custom log files and JSON format

Hi all,

I'm looking for an input plugin support some custom log files and I have
not understood how this works. The imfile seems to simply assume that
the whole log line is the message. But the severity, time, facility must
be parsed first. improg seems to be an option and combine it with
sed/awk/jq (for json), but I'm wondering if there are better ways,
especially for the json version.

These are 2 examples I have:

text based:

2023-10-01T23:41:44.144639044+02:00 [INFO][WEB][Main] Report Caller is
set to [false]

JSON based:

{"level":"debug","ts":"2023-09-29T17:49:52.218Z","logger":"Web
Server","caller":"migrator/migrator.go:71","msg":"trace","elapsed":0.001367452,"rows":1,"sql":"SELECT
count(*) FROM information_schema.tables WHERE table_schema =
CURRENT_SCHEMA() AND table_name = 'stocks' AND table_type = 'BASE TABLE'"}

Regards,

Karsten

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

look at mmnormalize for ways to parse the message into various fields under $!
that you can then use in templates. There is a very simple json parsing option
as part of this.

David Lang

On Mon, 2 Oct 2023, Karsten Ohme via rsyslog wrote:

> Hi all,
>
> I'm looking for an input plugin support some custom log files and I have
> not understood how this works. The imfile seems to simply assume that
> the whole log line is the message. But the severity, time, facility must
> be parsed first. improg seems to be an option and combine it with
> sed/awk/jq (for json), but I'm wondering if there are better ways,
> especially for the json version.
>
> These are 2 examples I have:
>
> text based:
>
> 2023-10-01T23:41:44.144639044+02:00 [INFO][WEB][Main] Report Caller is
> set to [false]
>
> JSON based:
>
> {"level":"debug","ts":"2023-09-29T17:49:52.218Z","logger":"Web
> Server","caller":"migrator/migrator.go:71","msg":"trace","elapsed":0.001367452,"rows":1,"sql":"SELECT
> count(*) FROM information_schema.tables WHERE table_schema =
> CURRENT_SCHEMA() AND table_name = 'stocks' AND table_type = 'BASE TABLE'"}
>
>
> Regards,
>
> Karsten
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.