Hello,
So I loaded the impstats.so module and restarted Rsyslog.
Rsyslog stopped forwarding logs in less than 5 min after this first restart
so I didn't get any rsyslogd-pstats logs (I use the default option to send
stats to the syslog stream with interval set to 5min).
I restarted Rsyslog and this time it kept working a bit longer. After 1 hr
Rsyslog stopped forwarding logs.
Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:
2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0 discarded.nf=0
maxqsize=145 *(last log)*
2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
Another filter in Splunk ( "rsyslogd-pstats" submitted NOT "origin=imptcp"
NOT "origin=imtcp" ) gave me this:
2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=91 disallowed=0
2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=6009 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=83 disallowed=0
2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=5139 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=66 disallowed=0
2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=4657 ratelimit.discarded=0
ratelimit.numratelimiters=0
* I have the same issue with the another vCenter 6.7U3. *
Rsyslog stops forwarding logs after a while.
Both have the same Rsyslog version:
rsyslogd 8.37.0, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64
This is my /etc/rsyslog.conf (identical on both vCenter):
################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad impstats.so
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514
#
# cron log entries for GEN003160
#
cron.* -/var/log/cron
#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log
*However*, in a 3rd vCenter, which is running a slightly different version
(VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.
This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:
################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514
The difference is at the end:
The conf file *with the issue* has these lines:
#
# cron log entries for GEN003160
#
cron.* -/var/log/cron
#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log
I don't really know what that means and if it can explain the issue.
Lucien
On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <lucien.courcol@gmail.com>
wrote:
> Thx David Lang,
>
> I'll work on that tomorrow.
>
> I'll use this guide to get impstats logs:
> https://www.rsyslog.com/how-to-use-impstats/
>
> Lucien
>
> On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:
>
>> wihtout seeing your rsyslog.conf we can't begin to guess what's
>> happening.
>> configuring impstats and showing us the output when it's stopped would
>> help us
>> to see what's happening.
>>
>> David Lang
>>
>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
>>
>> > Date: Wed, 11 Sep 2019 14:45:38 +0200
>> > From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
>> > To: rsyslog@lists.adiscon.com
>> > Cc: Lucien Courcol <lucien.courcol@gmail.com>
>> > Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
>> ~10min
>> > after starting
>> >
>> > Hello,
>> >
>> > We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
>> and
>> > we noticed a gap of logs in our syslog server (kiwi) since then.
>> >
>> > I did a bit of troubleshooting but Rsyslog (the syslog client running on
>> > VCSA) is completely new to me.
>> >
>> > I use this command to restart Rsyslog:
>> >
>> > systemctl restart rsyslog
>> >
>> > Right after starting up Rsyslog, logs are being sent to our syslog
>> server.
>> >
>> > ~10min later, no more logs are sent.
>> >
>> > The vCenter log file in our syslog server stops getting updated.
>> > I did a tcpdump in our vCenter and I see that the vCenter stops sending
>> > logs.
>> > Using UDP or TCP doesn't fix the issue.
>> > I looked for errors in various log files in the vCenter but can't find
>> > anything.
>> >
>> > This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
>> after
>> > restarting Rsyslog
>> >
>> > 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
>> > http://www.rsyslog.com"] exiting on signal 15.
>> > 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable
>> TZ
>> > is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>> > http://www.rsyslog.com/e/2442 ]
>> > 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
>> > socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
>> > 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> > http://www.rsyslog.com"] start
>> >
>> > Rsyslog is still running based on this command
>> >
>> > systemctl status rsyslog.service
>> >
>> > ? rsyslog.service - System Logging Service
>> > Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
>> vendor
>> > preset: enabled)
>> > Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
>> > Docs: man:rsyslogd(8)
>> > http://www.rsyslog.com/doc/
>> > Main PID: 22235 (rsyslogd)
>> > Tasks: 12
>> > Memory: 5.7M
>> > CPU: 191ms
>> > CGroup: /system.slice/rsyslog.service
>> > ??22235 /usr/sbin/rsyslogd -n
>> >
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
>> > process exited, code=killed, status=9/KILL
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
>> > Service.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
>> > entered failed state.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
>> > with result 'signal'.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
>> > Service...
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
>> > Service.
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
>> variable
>> > TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>> > http://www.rsyslog.com/e/2442 ]
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
>> > UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
>> [v8.37.0]
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> > http://www.rsyslog.com"] start
>> > (real hostname has been replaced by vcenter.domain.local)
>> >
>> > I created a ticket at VMware support, but the agent wasn't able to find
>> any
>> > errors as well and she suggested to take a backup of our vCenter and
>> > reinstall with a restore to get a fresh install of Photon OS since
>> Rsyslog
>> > is integrated in Photon OS. I'm not going to do that now, maybe as a
>> last
>> > troubleshooting step.
>> >
>> > In the meantime, do you guys have an idea? Wrong Rsyslog config?
>> >
>> > Thx for your help.
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.