Mailing List Archive: can't extract fields from log files using mmnormalize

here is rsyslog config:

```

module(load="imfile")
module(load="mmnormalize")

input (
    type="imfile"
    File="/var/log/libvirt/qemu/*.log"
    Tag="qemu-kvm"
    Severity="info"
    ruleset="qemu_logs"
)

ruleset(name="qemu_logs") {
    action(type="mmnormalize"
ruleset="/etc/rsyslog.d/qemu-kvm.rulebase" UseRawMsg="on")
    action(
        type="omfile"
        file="/var/log/libvirt/qemu.log"
        template="RSYSLOG_DebugFormat"
    )
}

```

rulebase:

```

version=2

# 2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0:
char device redirected to /dev/pts/3 (label charserial0)
rule=:%_tiemstamp:date-rfc5424% qemu-kvm: %_mesg:rest%

```

sample log input:

```

2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0: char
device redirected to /dev/pts/3 (label charserial0)

```

produced output:

```

Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'kvm-host.company.name', PRI: 134,
syslogtag 'qemu-kvm', programname: 'qemu-kvm', APP-NAME: 'qemu-kvm',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'Oct 6 09:24:29', STRUCTURED-DATA: '-',
msg: '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0:
char device redirected to /dev/pts/3 (label charserial0)'
escaped msg: '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev
pty,id=charserial0: char device redirected to /dev/pts/3 (label
charserial0)'
inputname: imfile rawmsg: '2020-10-06T06:13:21.797593Z qemu-kvm:
-chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label
charserial0)'
$!:{ "metadata": { "filename":
"\/var\/log\/libvirt\/qemu\/vmNUMBER.log", "fileoffset": "31305" } }
$.:
$/:

```

raw log message parsed by lognormalizer.

```

$ echo '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev
pty,id=charserial0: char device redirected to /dev/pts/3 (label
charserial0)' | lognormalizer -r /etc/rsyslog.d/qemu-kvm.rulebase
{ "_mesg": "-chardev pty,id=charserial0: char device redirected to
\/dev\/pts\/3 (label charserial0)", "_tiemstamp":
"2020-10-06T06:13:21.797593Z" }

```

my end goal is to extract several fields (mainly, timestamp) and forward
it all to graylog via its GELF input. however, no matter what I do,
additional fields do not appear anywhere.

how else could I debug this issue?

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

HI
try to define new template (to see variables produced by mmnormalize) and
use it instead of RSYSLOG_DebugFormat:

template (name="json_w_eol" type="string" string="%$!all-json%\n")
....
file="/var/log/libvirt/qemu.log"
template="json_w_eol"
...

m.

On Tue, Oct 6, 2020 at 8:35 AM Wasil W. Siargiejczyk via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> here is rsyslog config:
>
> ```
>
> module(load="imfile")
> module(load="mmnormalize")
>
>
> input (
> type="imfile"
> File="/var/log/libvirt/qemu/*.log"
> Tag="qemu-kvm"
> Severity="info"
> ruleset="qemu_logs"
> )
>
> ruleset(name="qemu_logs") {
> action(type="mmnormalize"
> ruleset="/etc/rsyslog.d/qemu-kvm.rulebase" UseRawMsg="on")
> action(
> type="omfile"
> file="/var/log/libvirt/qemu.log"
> template="RSYSLOG_DebugFormat"
> )
> }
>
> ```
>
>
> rulebase:
>
> ```
>
> version=2
>
> # 2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0:
> char device redirected to /dev/pts/3 (label charserial0)
> rule=:%_tiemstamp:date-rfc5424% qemu-kvm: %_mesg:rest%
>
> ```
>
>
> sample log input:
>
> ```
>
> 2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0: char
> device redirected to /dev/pts/3 (label charserial0)
>
> ```
>
>
> produced output:
>
> ```
>
> Debug line with all properties:
> FROMHOST: '', fromhost-ip: '', HOSTNAME: 'kvm-host.company.name', PRI:
> 134,
> syslogtag 'qemu-kvm', programname: 'qemu-kvm', APP-NAME: 'qemu-kvm',
> PROCID: '-', MSGID: '-',
> TIMESTAMP: 'Oct 6 09:24:29', STRUCTURED-DATA: '-',
> msg: '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev pty,id=charserial0:
> char device redirected to /dev/pts/3 (label charserial0)'
> escaped msg: '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev
> pty,id=charserial0: char device redirected to /dev/pts/3 (label
> charserial0)'
> inputname: imfile rawmsg: '2020-10-06T06:13:21.797593Z qemu-kvm:
> -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label
> charserial0)'
> $!:{ "metadata": { "filename":
> "\/var\/log\/libvirt\/qemu\/vmNUMBER.log", "fileoffset": "31305" } }
> $.:
> $/:
>
> ```
>
>
> raw log message parsed by lognormalizer.
>
> ```
>
> $ echo '2020-10-06T06:13:21.797593Z qemu-kvm: -chardev
> pty,id=charserial0: char device redirected to /dev/pts/3 (label
> charserial0)' | lognormalizer -r /etc/rsyslog.d/qemu-kvm.rulebase
> { "_mesg": "-chardev pty,id=charserial0: char device redirected to
> \/dev\/pts\/3 (label charserial0)", "_tiemstamp":
> "2020-10-06T06:13:21.797593Z" }
>
> ```
>
>
> my end goal is to extract several fields (mainly, timestamp) and forward
> it all to graylog via its GELF input. however, no matter what I do,
> additional fields do not appear anywhere.
>
> how else could I debug this issue?
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.