Mailing List Archive

RHEL / CentOS 8.1 omelasticsearch JSON issues
I installed the omelasticsearch package on a CentOS 8.1 host using the
package manager.

My process generates messages with the @cee: prefix.

When I look at the entries in OpenSearch the JSON is all embedded inside
a "MESSAGE" field:



"MESSAGE": "@cee: {\"hostname\":\" .....




If I tell the CentOS host to forward the messages over TCP to a more
modern rsyslog on Ubuntu then it is working fine.



myProcess
-> syslog()
-> rsyslogd/CentOS 8.1
-> omelasticsearch
-> MESSAGE as text


myProcess
-> syslog()
-> rsyslogd/CentOS 8.1
-> TCP
-> rsyslogd on Ubuntu
-> omelasticsearch
-> all the CEE fields appear in index



_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
full configuration please.

David Lang
On Fri, 9 Jul 2021, Daniel Pocock via rsyslog wrote:

> Date: Fri, 9 Jul 2021 01:48:07 +0200
> From: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Daniel Pocock <daniel@pocock.pro>
> Subject: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues
>
>
> I installed the omelasticsearch package on a CentOS 8.1 host using the
> package manager.
>
> My process generates messages with the @cee: prefix.
>
> When I look at the entries in OpenSearch the JSON is all embedded inside
> a "MESSAGE" field:
>
>
>
> "MESSAGE": "@cee: {\"hostname\":\" .....
>
>
>
>
> If I tell the CentOS host to forward the messages over TCP to a more
> modern rsyslog on Ubuntu then it is working fine.
>
>
>
> myProcess
> -> syslog()
> -> rsyslogd/CentOS 8.1
> -> omelasticsearch
> -> MESSAGE as text
>
>
> myProcess
> -> syslog()
> -> rsyslogd/CentOS 8.1
> -> TCP
> -> rsyslogd on Ubuntu
> -> omelasticsearch
> -> all the CEE fields appear in index
>
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 09/07/2021 02:01, David Lang wrote:
> full configuration please.

I've used an identical omelasticsearch config on both CentOS and Ubuntu,
see below. It works on Ubuntu, not on CentOS

Ubuntu package versions:
rsyslog 8.32.0-1ubuntu4
rsyslog-elasticsearch 8.32.0-1ubuntu4


CentOS 8.1 package versions:
rsyslog-elasticsearch-8.1911.0-7.el8_4.2.x86_64
rsyslog-relp-8.1911.0-7.el8_4.2.x86_64
rsyslog-gssapi-8.1911.0-7.el8_4.2.x86_64
rsyslog-gnutls-8.1911.0-7.el8_4.2.x86_64
rsyslog-8.1911.0-7.el8_4.2.x86_64



OpenSearch RC1 from 1 June:
https://opensearch.org/downloads.html



CentOS 8.1 host is a fresh install.

/etc/rsyslog.conf is default

/etc/rsyslog.d/reConServer-test.conf:

module(load="mmjsonparse")

*.* :mmjsonparse:

template(name="isJSON" type="list") {
property(name="$!all-json")
}

module(load="omelasticsearch")
*.* action(type="omelasticsearch"
template="isJSON"
server="my-server"
serverport="9200"
searchIndex="log"
uid="admin"
pwd="my secret")
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
8.32 is quite a bit older than 8.1911 (2-3 years)

Please log a message with the template RSYSLOG_DebugFormat on both so we can see
what's different

I also don't know what the default rsyslog.conf is on every system, so please
include that as well.

on 8.1911 you can start rsyslog with the command line option -o /path/to/file
and that file will then contain the combined config (including ny included
files)

At this point, I suspect that what is different is where the include is for the
different distros, one including the file before it writes things to the default
files and the other after, but that's a guess without seeing the full configs.

David Lang

On Fri, 9 Jul 2021, Daniel Pocock via rsyslog wrote:

> I've used an identical omelasticsearch config on both CentOS and Ubuntu,
> see below. It works on Ubuntu, not on CentOS
>
> Ubuntu package versions:
> rsyslog 8.32.0-1ubuntu4
> rsyslog-elasticsearch 8.32.0-1ubuntu4
>
>
> CentOS 8.1 package versions:
> rsyslog-elasticsearch-8.1911.0-7.el8_4.2.x86_64
> rsyslog-relp-8.1911.0-7.el8_4.2.x86_64
> rsyslog-gssapi-8.1911.0-7.el8_4.2.x86_64
> rsyslog-gnutls-8.1911.0-7.el8_4.2.x86_64
> rsyslog-8.1911.0-7.el8_4.2.x86_64
>
>
>
> OpenSearch RC1 from 1 June:
> https://opensearch.org/downloads.html
>
>
>
> CentOS 8.1 host is a fresh install.
>
> /etc/rsyslog.conf is default
>
> /etc/rsyslog.d/reConServer-test.conf:
>
> module(load="mmjsonparse")
>
> *.* :mmjsonparse:
>
> template(name="isJSON" type="list") {
> property(name="$!all-json")
> }
>
> module(load="omelasticsearch")
> *.* action(type="omelasticsearch"
> template="isJSON"
> server="my-server"
> serverport="9200"
> searchIndex="log"
> uid="admin"
> pwd="my secret")
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 09/07/2021 02:32, David Lang wrote:
> 8.32 is quite a bit older than 8.1911 (2-3 years)
>
> Please log a message with the template RSYSLOG_DebugFormat on both so we
> can see what's different
>

This came from CentOS 8.1 with RSYSLOG_DebugFormat:

Debug line with all properties:
FROMHOST: 'something', fromhost-ip: '127.0.0.1', HOSTNAME: 'something',
PRI: 30,
syslogtag 'lt-reConServer[12456]:', programname: 'lt-reConServer',
APP-NAME: 'lt-reConServer', PROCID: '12456', MSGID: '-',
TIMESTAMP: 'Jul 9 02:38:08', STRUCTURED-DATA: '-',
msg: '@cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
escaped msg: '@cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
inputname: imjournal rawmsg: '@cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
$!:{ "_BOOT_ID": "94fbf657a095412d80b4c387cbd90230", "_MACHINE_ID":
"e339bc1ec88911eb92d2fb6499360034", "PRIORITY": "6", "SYSLOG_FACILITY":
"3", "_TRANSPORT": "syslog", "_CAP_EFFECTIVE": "0", "_HOSTNAME":
"something.example.org", "_AUDIT_LOGINUID": "1000",
"_SYSTEMD_OWNER_UID": "1000", "_SYSTEMD_SLICE": "user-1000.slice",
"_SYSTEMD_USER_SLICE": "-.slice", "_UID": "1000", "_GID": "1000",
"_SELINUX_CONTEXT":
"unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023",
"SYSLOG_IDENTIFIER": "lt-reConServer", "_COMM": "lt-reConServer",
"_EXE":
"\/home\/daniel\/ws\/resiprocate\/resip-github\/apps\/reConServer\/.libs\/lt-reConServer",
"_CMDLINE":
"\/home\/daniel\/ws\/resiprocate\/resip-github\/apps\/reConServer\/.libs\/lt-reConServer
apps\/reConServer\/reConServer.config.test-local", "_AUDIT_SESSION":
"3", "_SYSTEMD_CGROUP":
"\/user.slice\/user-1000.slice\/session-3.scope", "_SYSTEMD_SESSION":
"3", "_SYSTEMD_UNIT": "session-3.scope", "_SYSTEMD_INVOCATION_ID":
"f2e7e38ea3374a869ee7f51eaf745e1d", "SYSLOG_PID": "12456", "_PID":
"12456", "MESSAGE": "@cee:
{\"hostname\":\"something.example.org\",\"pri\":\"DEBUG\",\"syslog!pri\":6,\"time\":\"2021-07-09T00:38:08.262005743Z\",\"pname\":\"lt-reConServer\",\"subsys\":\"RECON\",\"proc!id\":12456,\"proc!tid\":139979031448192,\"file!name\":\"Conversation.cxx\",\"file!line\":45,\"msg\":\"Conversation
created, handle=1\"}", "_SOURCE_REALTIME_TIMESTAMP": "1625791088262026" }
$.:
$/:








> I also don't know what the default rsyslog.conf is on every system, so
> please include  that as well.
>
> on 8.1911 you can start rsyslog with the command line option -o
> /path/to/file and that file will then contain the combined config
> (including ny included files)
>
> At this point, I suspect that what is different is where the include is
> for the different distros, one including the file before it writes
> things to the default files and the other after, but  that's a guess
> without seeing the full configs.
>


Please find the output from -o underneath

It includes both omelasticsearch and omfwd

The omfwd is working for me if I send it over TCP to the newer rsyslog



## full conf created by rsyslog version 8.1911.0-7.el8_4.2 at 2021-07-09
02:41:25 ##

##### BEGIN CONFIG: /etc/rsyslog.conf
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

module(load="imuxsock"
SysSock.Use="off") # Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
module(load="imjournal"
StateFile="imjournal.state") # File to store the position in the
journal
#module(load="imklog") # reads kernel messages (the same are read from
journald)
#module(load="immark") # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")

# Use default timestamp format
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

# Include all config files in /etc/rsyslog.d/
include(file="/etc/rsyslog.d/*.conf" mode="optional")
##### BEGIN CONFIG: /etc/rsyslog.d/reConServer-test.conf

# sudo dnf install rsyslog-elasticsearch

module(load="mmjsonparse")

*.* :mmjsonparse:

template(name="isJSON" type="list") {
property(name="$!all-json")
}

#module(load="omelasticsearch")
#*.* action(type="omelasticsearch"
# template="isJSON"
# server="my-host"
# serverport="9200"
# searchIndex="log"
# searchType="_doc"
# uid="admin"
# pwd="secret")


*.* /tmp/debugfmt;RSYSLOG_DebugFormat



##### END CONFIG: /etc/rsyslog.d/reConServer-test.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log


# ### sample forwarding rule ###
*.* action(type="omfwd"



queue.filename="fwdRule1"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
queue.type="LinkedList"
action.resumeRetryCount="-1"



Target="my-host" Port="514" Protocol="tcp")

##### END CONFIG: /etc/rsyslog.conf
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
ok, this shows the exact config that rsyslog is seeing, and it shows that the
mmjsonparse call is not able to parse the message. I'm not exactly sure why.

what is the other rsyslog instance you are using? you said 8.32, but here you
mention sending from this config to the 'newer' rsyslog??

David Lang

On Fri, 9 Jul 2021, Daniel Pocock via rsyslog wrote:

> Date: Fri, 9 Jul 2021 02:43:59 +0200
> From: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> To: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Daniel Pocock <daniel@pocock.pro>
> Subject: Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues
>
>
> On 09/07/2021 02:32, David Lang wrote:
>> 8.32 is quite a bit older than 8.1911 (2-3 years)
>>
>> Please log a message with the template RSYSLOG_DebugFormat on both so we
>> can see what's different
>>
>
> This came from CentOS 8.1 with RSYSLOG_DebugFormat:
>
> Debug line with all properties:
> FROMHOST: 'something', fromhost-ip: '127.0.0.1', HOSTNAME: 'something',
> PRI: 30,
> syslogtag 'lt-reConServer[12456]:', programname: 'lt-reConServer',
> APP-NAME: 'lt-reConServer', PROCID: '12456', MSGID: '-',
> TIMESTAMP: 'Jul 9 02:38:08', STRUCTURED-DATA: '-',
> msg: '@cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> escaped msg: '@cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> inputname: imjournal rawmsg: '@cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T00:38:08.262005743Z","pname":"lt-reConServer","subsys":"RECON","proc!id":12456,"proc!tid":139979031448192,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> $!:{ "_BOOT_ID": "94fbf657a095412d80b4c387cbd90230", "_MACHINE_ID":
> "e339bc1ec88911eb92d2fb6499360034", "PRIORITY": "6", "SYSLOG_FACILITY":
> "3", "_TRANSPORT": "syslog", "_CAP_EFFECTIVE": "0", "_HOSTNAME":
> "something.example.org", "_AUDIT_LOGINUID": "1000",
> "_SYSTEMD_OWNER_UID": "1000", "_SYSTEMD_SLICE": "user-1000.slice",
> "_SYSTEMD_USER_SLICE": "-.slice", "_UID": "1000", "_GID": "1000",
> "_SELINUX_CONTEXT":
> "unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023",
> "SYSLOG_IDENTIFIER": "lt-reConServer", "_COMM": "lt-reConServer",
> "_EXE":
> "\/home\/daniel\/ws\/resiprocate\/resip-github\/apps\/reConServer\/.libs\/lt-reConServer",
> "_CMDLINE":
> "\/home\/daniel\/ws\/resiprocate\/resip-github\/apps\/reConServer\/.libs\/lt-reConServer
> apps\/reConServer\/reConServer.config.test-local", "_AUDIT_SESSION":
> "3", "_SYSTEMD_CGROUP":
> "\/user.slice\/user-1000.slice\/session-3.scope", "_SYSTEMD_SESSION":
> "3", "_SYSTEMD_UNIT": "session-3.scope", "_SYSTEMD_INVOCATION_ID":
> "f2e7e38ea3374a869ee7f51eaf745e1d", "SYSLOG_PID": "12456", "_PID":
> "12456", "MESSAGE": "@cee:
> {\"hostname\":\"something.example.org\",\"pri\":\"DEBUG\",\"syslog!pri\":6,\"time\":\"2021-07-09T00:38:08.262005743Z\",\"pname\":\"lt-reConServer\",\"subsys\":\"RECON\",\"proc!id\":12456,\"proc!tid\":139979031448192,\"file!name\":\"Conversation.cxx\",\"file!line\":45,\"msg\":\"Conversation
> created, handle=1\"}", "_SOURCE_REALTIME_TIMESTAMP": "1625791088262026" }
> $.:
> $/:
>
>
>
>
>
>
>
>
>> I also don't know what the default rsyslog.conf is on every system, so
>> please include  that as well.
>>
>> on 8.1911 you can start rsyslog with the command line option -o
>> /path/to/file and that file will then contain the combined config
>> (including ny included files)
>>
>> At this point, I suspect that what is different is where the include is
>> for the different distros, one including the file before it writes
>> things to the default files and the other after, but  that's a guess
>> without seeing the full configs.
>>
>
>
> Please find the output from -o underneath
>
> It includes both omelasticsearch and omfwd
>
> The omfwd is working for me if I send it over TCP to the newer rsyslog
>
>
>
> ## full conf created by rsyslog version 8.1911.0-7.el8_4.2 at 2021-07-09
> 02:41:25 ##
>
> ##### BEGIN CONFIG: /etc/rsyslog.conf
> # rsyslog configuration file
>
> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> # If you experience problems, see
> http://www.rsyslog.com/doc/troubleshoot.html
>
> #### MODULES ####
>
> module(load="imuxsock"
> SysSock.Use="off") # Turn off message reception via local log socket;
> # local messages are retrieved through imjournal now.
> module(load="imjournal"
> StateFile="imjournal.state") # File to store the position in the
> journal
> #module(load="imklog") # reads kernel messages (the same are read from
> journald)
> #module(load="immark") # provides --MARK-- message capability
>
> # Provides UDP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imudp.html
> #module(load="imudp") # needs to be done just once
> #input(type="imudp" port="514")
>
> # Provides TCP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imtcp.html
> #module(load="imtcp") # needs to be done just once
> #input(type="imtcp" port="514")
>
> #### GLOBAL DIRECTIVES ####
>
> # Where to place auxiliary files
> global(workDirectory="/var/lib/rsyslog")
>
> # Use default timestamp format
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>
> # Include all config files in /etc/rsyslog.d/
> include(file="/etc/rsyslog.d/*.conf" mode="optional")
> ##### BEGIN CONFIG: /etc/rsyslog.d/reConServer-test.conf
>
> # sudo dnf install rsyslog-elasticsearch
>
> module(load="mmjsonparse")
>
> *.* :mmjsonparse:
>
> template(name="isJSON" type="list") {
> property(name="$!all-json")
> }
>
> #module(load="omelasticsearch")
> #*.* action(type="omelasticsearch"
> # template="isJSON"
> # server="my-host"
> # serverport="9200"
> # searchIndex="log"
> # searchType="_doc"
> # uid="admin"
> # pwd="secret")
>
>
> *.* /tmp/debugfmt;RSYSLOG_DebugFormat
>
>
>
> ##### END CONFIG: /etc/rsyslog.d/reConServer-test.conf
>
>
> #### RULES ####
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg :omusrmsg:*
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
>
> # ### sample forwarding rule ###
> *.* action(type="omfwd"
>
>
>
> queue.filename="fwdRule1"
> queue.maxdiskspace="1g"
> queue.saveonshutdown="on"
> queue.type="LinkedList"
> action.resumeRetryCount="-1"
>
>
>
> Target="my-host" Port="514" Protocol="tcp")
>
> ##### END CONFIG: /etc/rsyslog.conf
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 09/07/2021 03:14, David Lang wrote:
> ok, this shows the exact config that rsyslog is seeing, and it shows
> that the mmjsonparse call is not able to parse the message. I'm not
> exactly sure why.
>
> what is the other rsyslog instance you are using? you said 8.32, but
> here you mention sending  from this config to the 'newer' rsyslog??
>



The Ubuntu system has 8.32.0-1ubuntu4 with default Ubuntu settings

The Syslog messages that are parsed correctly are exactly the same
message bodies. They are sent like this, notice it uses mmjsonparse in
Ubuntu:


application (reConServer / reSIProcate)
syslog() API call
rsyslog-8.1911.0-7.el8 (CentOS)
omfwd, TCP
rsyslog 8.32.0-1ubuntu4
imtcp
mmjsonparse

If there is any change you can suggest, I can tweak the message body in
the unit tests and get almost immediate feedback

https://github.com/resiprocate/resiprocate/commit/7db19670a39f459997c72185b5e3d501825d6e6d

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
what does RSYSLOG_DebugFormat look on Ubuntu with 8.32?

Rainer

El vie, 9 jul 2021 a las 10:56, Daniel Pocock via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
>
>
> On 09/07/2021 03:14, David Lang wrote:
> > ok, this shows the exact config that rsyslog is seeing, and it shows
> > that the mmjsonparse call is not able to parse the message. I'm not
> > exactly sure why.
> >
> > what is the other rsyslog instance you are using? you said 8.32, but
> > here you mention sending from this config to the 'newer' rsyslog??
> >
>
>
>
> The Ubuntu system has 8.32.0-1ubuntu4 with default Ubuntu settings
>
> The Syslog messages that are parsed correctly are exactly the same
> message bodies. They are sent like this, notice it uses mmjsonparse in
> Ubuntu:
>
>
> application (reConServer / reSIProcate)
> syslog() API call
> rsyslog-8.1911.0-7.el8 (CentOS)
> omfwd, TCP
> rsyslog 8.32.0-1ubuntu4
> imtcp
> mmjsonparse
>
> If there is any change you can suggest, I can tweak the message body in
> the unit tests and get almost immediate feedback
>
> https://github.com/resiprocate/resiprocate/commit/7db19670a39f459997c72185b5e3d501825d6e6d
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 09/07/2021 11:01, Rainer Gerhards wrote:
> what does RSYSLOG_DebugFormat look on Ubuntu with 8.32?



Debug line with all properties:
FROMHOST: 'something.example.org', fromhost-ip: '1.2.3.4', HOSTNAME:
'something', PRI: 30,
syslogtag 'lt-reConServer[37142]:', programname: 'lt-reConServer',
APP-NAME: 'lt-reConServer', PROCID: '37142', MSGID: '-',
TIMESTAMP: 'Jul 9 11:03:41', STRUCTURED-DATA: '-',
msg: ' @cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
escaped msg: ' @cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
inputname: imtcp rawmsg: '<30>Jul 9 11:03:41 some-host
lt-reConServer[37142]: @cee:
{"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
created, handle=1"}'
$!:{ "hostname": "something.example.org", "pri": "DEBUG", "syslog!pri":
6, "time": "2021-07-09T09:03:41.513601990Z", "pname": "lt-reConServer",
"subsys": "RECON", "proc!id": 37142, "proc!tid": 139892894497408,
"file!name": "Conversation.cxx", "file!line": 45, "msg": "Conversation
created, handle=1" }
$.:
$/:

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
sry, I misunderstood the conversation. So on the machine with the old
version, it works, but on the new not. OK.

In this case, can you create a debug log on CentOS8 and post it?

https://www.rsyslog.com/doc/master/troubleshooting/debug.html

Rainer

El vie, 9 jul 2021 a las 11:07, Daniel Pocock via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
>
>
> On 09/07/2021 11:01, Rainer Gerhards wrote:
> > what does RSYSLOG_DebugFormat look on Ubuntu with 8.32?
>
>
>
> Debug line with all properties:
> FROMHOST: 'something.example.org', fromhost-ip: '1.2.3.4', HOSTNAME:
> 'something', PRI: 30,
> syslogtag 'lt-reConServer[37142]:', programname: 'lt-reConServer',
> APP-NAME: 'lt-reConServer', PROCID: '37142', MSGID: '-',
> TIMESTAMP: 'Jul 9 11:03:41', STRUCTURED-DATA: '-',
> msg: ' @cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> escaped msg: ' @cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> inputname: imtcp rawmsg: '<30>Jul 9 11:03:41 some-host
> lt-reConServer[37142]: @cee:
> {"hostname":"something.example.org","pri":"DEBUG","syslog!pri":6,"time":"2021-07-09T09:03:41.513601990Z","pname":"lt-reConServer","subsys":"RECON","proc!id":37142,"proc!tid":139892894497408,"file!name":"Conversation.cxx","file!line":45,"msg":"Conversation
> created, handle=1"}'
> $!:{ "hostname": "something.example.org", "pri": "DEBUG", "syslog!pri":
> 6, "time": "2021-07-09T09:03:41.513601990Z", "pname": "lt-reConServer",
> "subsys": "RECON", "proc!id": 37142, "proc!tid": 139892894497408,
> "file!name": "Conversation.cxx", "file!line": 45, "msg": "Conversation
> created, handle=1" }
> $.:
> $/:
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 09/07/2021 11:17, Rainer Gerhards wrote:
> sry, I misunderstood the conversation. So on the machine with the old
> version, it works, but on the new not. OK.

> In this case, can you create a debug log on CentOS8 and post it?
>
> https://www.rsyslog.com/doc/master/troubleshooting/debug.html
>

Thanks for the fast feedback, I found the root cause

Debian and Ubuntu have the mmjsonparse module in the main rsyslog
package so it just works.

CentOS users have to install an extra package,

sudo dnf install rsyslog-mmjsonparse

rsyslogd[44038]: could not load module 'mmjsonparse', errors: trying to
load module /usr/lib64/rsyslog/mmjsonparse.so:
/usr/lib64/rsyslog/mmjsonparse.so: cannot open shared object file: No
such file or directory [v8.1911.0-7.el8_4.2 try
https://www.rsyslog.com/e/2066 ]



I installed the extra package and now the logging to OpenSearch (using
omelasticsearch) is working on the CentOS 8.1 host with rsyslog
8.1911.0-7.el8_4.2
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
over the years the distros have put more effort into minimizing the optional
dependencies that get pulled in, so more of the rsyslog modules have become
separate packages when they pull in other things.

it's nice when it's this easy to fix, but you also got a lesson in
troubleshooting.

Have fun.

David Lang

On Fri, 9 Jul 2021, Daniel Pocock via rsyslog wrote:

> Date: Fri, 9 Jul 2021 12:36:43 +0200
> From: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Daniel Pocock <daniel@pocock.pro>
> Subject: Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues
>
>
>
> On 09/07/2021 11:17, Rainer Gerhards wrote:
>> sry, I misunderstood the conversation. So on the machine with the old
>> version, it works, but on the new not. OK.
>
>> In this case, can you create a debug log on CentOS8 and post it?
>>
>> https://www.rsyslog.com/doc/master/troubleshooting/debug.html
>>
>
> Thanks for the fast feedback, I found the root cause
>
> Debian and Ubuntu have the mmjsonparse module in the main rsyslog
> package so it just works.
>
> CentOS users have to install an extra package,
>
> sudo dnf install rsyslog-mmjsonparse
>
> rsyslogd[44038]: could not load module 'mmjsonparse', errors: trying to
> load module /usr/lib64/rsyslog/mmjsonparse.so:
> /usr/lib64/rsyslog/mmjsonparse.so: cannot open shared object file: No
> such file or directory [v8.1911.0-7.el8_4.2 try
> https://www.rsyslog.com/e/2066 ]
>
>
>
> I installed the extra package and now the logging to OpenSearch (using
> omelasticsearch) is working on the CentOS 8.1 host with rsyslog
> 8.1911.0-7.el8_4.2
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: RHEL / CentOS 8.1 omelasticsearch JSON issues [ In reply to ]
On 10/07/2021 00:20, David Lang wrote:
> over the years the distros have put more effort into minimizing the
> optional dependencies that get pulled in, so more of the rsyslog modules
> have become separate packages when they pull in other things.
>
> it's nice when it's this easy to fix, but you also got a lesson in
> troubleshooting.

For some reason it hadn't occurred to me to check for an error log from
rsyslog itself. I was kind of embarrassed about not spotting this
myself earlier in the process, after all, the LogAnalyzer package[1] in
Debian is one of my side interests, along with Ganglia[2] and other
other monitoring efforts.

Thanks for the fast response, it helps me move ahead with other public
projects.

Thanks on behalf of all those who will use whatever I'm cooking up in
reSIProcate land. Is anybody looking for an open alternative[3] to Zoom
and Hangouts?

Daniel

1. https://tracker.debian.org/pkg/loganalyzer
2. https://github.com/ganglia
3. https://jscommunicator.org/
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.