Mailing List Archive

Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting
Hello,

We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and
we noticed a gap of logs in our syslog server (kiwi) since then.

I did a bit of troubleshooting but Rsyslog (the syslog client running on
VCSA) is completely new to me.

I use this command to restart Rsyslog:

systemctl restart rsyslog

Right after starting up Rsyslog, logs are being sent to our syslog server.

~10min later, no more logs are sent.

The vCenter log file in our syslog server stops getting updated.
I did a tcpdump in our vCenter and I see that the vCenter stops sending
logs.
Using UDP or TCP doesn't fix the issue.
I looked for errors in various log files in the vCenter but can't find
anything.

This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after
restarting Rsyslog

2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
http://www.rsyslog.com"] exiting on signal 15.
2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ
is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
http://www.rsyslog.com/e/2442 ]
2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
http://www.rsyslog.com"] start

Rsyslog is still running based on this command

systemctl status rsyslog.service

? rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor
preset: enabled)
Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 22235 (rsyslogd)
Tasks: 12
Memory: 5.7M
CPU: 191ms
CGroup: /system.slice/rsyslog.service
??22235 /usr/sbin/rsyslogd -n

Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
process exited, code=killed, status=9/KILL
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
Service.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
entered failed state.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
with result 'signal'.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
Service...
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
Service.
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable
TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
http://www.rsyslog.com/e/2442 ]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
http://www.rsyslog.com"] start
(real hostname has been replaced by vcenter.domain.local)

I created a ticket at VMware support, but the agent wasn't able to find any
errors as well and she suggested to take a backup of our vCenter and
reinstall with a restore to get a fresh install of Photon OS since Rsyslog
is integrated in Photon OS. I'm not going to do that now, maybe as a last
troubleshooting step.

In the meantime, do you guys have an idea? Wrong Rsyslog config?

Thx for your help.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
wihtout seeing your rsyslog.conf we can't begin to guess what's happening.
configuring impstats and showing us the output when it's stopped would help us
to see what's happening.

David Lang

On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:

> Date: Wed, 11 Sep 2019 14:45:38 +0200
> From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Lucien Courcol <lucien.courcol@gmail.com>
> Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min
> after starting
>
> Hello,
>
> We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and
> we noticed a gap of logs in our syslog server (kiwi) since then.
>
> I did a bit of troubleshooting but Rsyslog (the syslog client running on
> VCSA) is completely new to me.
>
> I use this command to restart Rsyslog:
>
> systemctl restart rsyslog
>
> Right after starting up Rsyslog, logs are being sent to our syslog server.
>
> ~10min later, no more logs are sent.
>
> The vCenter log file in our syslog server stops getting updated.
> I did a tcpdump in our vCenter and I see that the vCenter stops sending
> logs.
> Using UDP or TCP doesn't fix the issue.
> I looked for errors in various log files in the vCenter but can't find
> anything.
>
> This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after
> restarting Rsyslog
>
> 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
> software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
> http://www.rsyslog.com"] exiting on signal 15.
> 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ
> is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> http://www.rsyslog.com/e/2442 ]
> 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
> socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
> 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> http://www.rsyslog.com"] start
>
> Rsyslog is still running based on this command
>
> systemctl status rsyslog.service
>
> ? rsyslog.service - System Logging Service
> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor
> preset: enabled)
> Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
> Docs: man:rsyslogd(8)
> http://www.rsyslog.com/doc/
> Main PID: 22235 (rsyslogd)
> Tasks: 12
> Memory: 5.7M
> CPU: 191ms
> CGroup: /system.slice/rsyslog.service
> ??22235 /usr/sbin/rsyslogd -n
>
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
> process exited, code=killed, status=9/KILL
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
> Service.
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
> entered failed state.
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
> with result 'signal'.
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
> Service...
> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
> Service.
> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable
> TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> http://www.rsyslog.com/e/2442 ]
> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
> UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> http://www.rsyslog.com"] start
> (real hostname has been replaced by vcenter.domain.local)
>
> I created a ticket at VMware support, but the agent wasn't able to find any
> errors as well and she suggested to take a backup of our vCenter and
> reinstall with a restore to get a fresh install of Photon OS since Rsyslog
> is integrated in Photon OS. I'm not going to do that now, maybe as a last
> troubleshooting step.
>
> In the meantime, do you guys have an idea? Wrong Rsyslog config?
>
> Thx for your help.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
Thx David Lang,

I'll work on that tomorrow.

I'll use this guide to get impstats logs:
https://www.rsyslog.com/how-to-use-impstats/

Lucien

On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:

> wihtout seeing your rsyslog.conf we can't begin to guess what's happening.
> configuring impstats and showing us the output when it's stopped would
> help us
> to see what's happening.
>
> David Lang
>
> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
>
> > Date: Wed, 11 Sep 2019 14:45:38 +0200
> > From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: Lucien Courcol <lucien.courcol@gmail.com>
> > Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
> ~10min
> > after starting
> >
> > Hello,
> >
> > We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and
> > we noticed a gap of logs in our syslog server (kiwi) since then.
> >
> > I did a bit of troubleshooting but Rsyslog (the syslog client running on
> > VCSA) is completely new to me.
> >
> > I use this command to restart Rsyslog:
> >
> > systemctl restart rsyslog
> >
> > Right after starting up Rsyslog, logs are being sent to our syslog
> server.
> >
> > ~10min later, no more logs are sent.
> >
> > The vCenter log file in our syslog server stops getting updated.
> > I did a tcpdump in our vCenter and I see that the vCenter stops sending
> > logs.
> > Using UDP or TCP doesn't fix the issue.
> > I looked for errors in various log files in the vCenter but can't find
> > anything.
> >
> > This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
> after
> > restarting Rsyslog
> >
> > 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
> > software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
> > http://www.rsyslog.com"] exiting on signal 15.
> > 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable
> TZ
> > is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> > http://www.rsyslog.com/e/2442 ]
> > 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
> > socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
> > 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> > http://www.rsyslog.com"] start
> >
> > Rsyslog is still running based on this command
> >
> > systemctl status rsyslog.service
> >
> > ? rsyslog.service - System Logging Service
> > Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
> vendor
> > preset: enabled)
> > Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
> > Docs: man:rsyslogd(8)
> > http://www.rsyslog.com/doc/
> > Main PID: 22235 (rsyslogd)
> > Tasks: 12
> > Memory: 5.7M
> > CPU: 191ms
> > CGroup: /system.slice/rsyslog.service
> > ??22235 /usr/sbin/rsyslogd -n
> >
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
> > process exited, code=killed, status=9/KILL
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
> > Service.
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
> > entered failed state.
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
> > with result 'signal'.
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
> > Service...
> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
> > Service.
> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
> variable
> > TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> > http://www.rsyslog.com/e/2442 ]
> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
> > UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> > http://www.rsyslog.com"] start
> > (real hostname has been replaced by vcenter.domain.local)
> >
> > I created a ticket at VMware support, but the agent wasn't able to find
> any
> > errors as well and she suggested to take a backup of our vCenter and
> > reinstall with a restore to get a fresh install of Photon OS since
> Rsyslog
> > is integrated in Photon OS. I'm not going to do that now, maybe as a last
> > troubleshooting step.
> >
> > In the meantime, do you guys have an idea? Wrong Rsyslog config?
> >
> > Thx for your help.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
Hello,

So I loaded the impstats.so module and restarted Rsyslog.

Rsyslog stopped forwarding logs in less than 5 min after this first restart
so I didn't get any rsyslogd-pstats logs (I use the default option to send
stats to the syslog stream with interval set to 5min).

I restarted Rsyslog and this time it kept working a bit longer. After 1 hr
Rsyslog stopped forwarding logs.

Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:

2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0 discarded.nf=0
maxqsize=145 *(last log)*
2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0 discarded.nf=0
maxqsize=145

Another filter in Splunk ( "rsyslogd-pstats" submitted NOT "origin=imptcp"
NOT "origin=imtcp" ) gave me this:

2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=91 disallowed=0
2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=6009 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=83 disallowed=0
2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=5139 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=66 disallowed=0
2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=4657 ratelimit.discarded=0
ratelimit.numratelimiters=0

* I have the same issue with the another vCenter 6.7U3. *
Rsyslog stops forwarding logs after a while.

Both have the same Rsyslog version:

rsyslogd 8.37.0, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64

This is my /etc/rsyslog.conf (identical on both vCenter):

################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad impstats.so
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514
#
# cron log entries for GEN003160
#
cron.* -/var/log/cron

#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log


*However*, in a 3rd vCenter, which is running a slightly different version
(VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.

This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:

################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514

The difference is at the end:

The conf file *with the issue* has these lines:

#
# cron log entries for GEN003160
#
cron.* -/var/log/cron

#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log


I don't really know what that means and if it can explain the issue.

Lucien


On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <lucien.courcol@gmail.com>
wrote:

> Thx David Lang,
>
> I'll work on that tomorrow.
>
> I'll use this guide to get impstats logs:
> https://www.rsyslog.com/how-to-use-impstats/
>
> Lucien
>
> On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:
>
>> wihtout seeing your rsyslog.conf we can't begin to guess what's
>> happening.
>> configuring impstats and showing us the output when it's stopped would
>> help us
>> to see what's happening.
>>
>> David Lang
>>
>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
>>
>> > Date: Wed, 11 Sep 2019 14:45:38 +0200
>> > From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
>> > To: rsyslog@lists.adiscon.com
>> > Cc: Lucien Courcol <lucien.courcol@gmail.com>
>> > Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
>> ~10min
>> > after starting
>> >
>> > Hello,
>> >
>> > We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
>> and
>> > we noticed a gap of logs in our syslog server (kiwi) since then.
>> >
>> > I did a bit of troubleshooting but Rsyslog (the syslog client running on
>> > VCSA) is completely new to me.
>> >
>> > I use this command to restart Rsyslog:
>> >
>> > systemctl restart rsyslog
>> >
>> > Right after starting up Rsyslog, logs are being sent to our syslog
>> server.
>> >
>> > ~10min later, no more logs are sent.
>> >
>> > The vCenter log file in our syslog server stops getting updated.
>> > I did a tcpdump in our vCenter and I see that the vCenter stops sending
>> > logs.
>> > Using UDP or TCP doesn't fix the issue.
>> > I looked for errors in various log files in the vCenter but can't find
>> > anything.
>> >
>> > This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
>> after
>> > restarting Rsyslog
>> >
>> > 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
>> > http://www.rsyslog.com"] exiting on signal 15.
>> > 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable
>> TZ
>> > is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>> > http://www.rsyslog.com/e/2442 ]
>> > 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
>> > socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
>> > 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> > http://www.rsyslog.com"] start
>> >
>> > Rsyslog is still running based on this command
>> >
>> > systemctl status rsyslog.service
>> >
>> > ? rsyslog.service - System Logging Service
>> > Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
>> vendor
>> > preset: enabled)
>> > Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
>> > Docs: man:rsyslogd(8)
>> > http://www.rsyslog.com/doc/
>> > Main PID: 22235 (rsyslogd)
>> > Tasks: 12
>> > Memory: 5.7M
>> > CPU: 191ms
>> > CGroup: /system.slice/rsyslog.service
>> > ??22235 /usr/sbin/rsyslogd -n
>> >
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
>> > process exited, code=killed, status=9/KILL
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
>> > Service.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
>> > entered failed state.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
>> > with result 'signal'.
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
>> > Service...
>> > Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
>> > Service.
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
>> variable
>> > TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>> > http://www.rsyslog.com/e/2442 ]
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
>> > UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
>> [v8.37.0]
>> > Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
>> > software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> > http://www.rsyslog.com"] start
>> > (real hostname has been replaced by vcenter.domain.local)
>> >
>> > I created a ticket at VMware support, but the agent wasn't able to find
>> any
>> > errors as well and she suggested to take a backup of our vCenter and
>> > reinstall with a restore to get a fresh install of Photon OS since
>> Rsyslog
>> > is integrated in Photon OS. I'm not going to do that now, maybe as a
>> last
>> > troubleshooting step.
>> >
>> > In the meantime, do you guys have an idea? Wrong Rsyslog config?
>> >
>> > Thx for your help.
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
when you configure impstats, you can configure it to write to a local file and
not to syslog, that should decouple it from whatever is causing it to freeze.I
would also set it to log more frequently.

all the options for impstats are listed at
https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html

each timeperiod it spits out a good chunk of logs, we will want to see at least
one complete set, ideally after it stops processing things otherwise.

putting a queue on a write to disk ( *.* ?esxLoc;esxFmt ) is usually
counterproductive, rsyslog spends more time managing the queue than just writing
the file.

David Lang


On Thu, 12 Sep 2019, Lucien Courcol wrote:

> Date: Thu, 12 Sep 2019 12:59:38 +0200
> From: Lucien Courcol <lucien.courcol@gmail.com>
> To: David Lang <david@lang.hm>
> Cc: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
> ~10min after starting
>
> Hello,
>
> So I loaded the impstats.so module and restarted Rsyslog.
>
> Rsyslog stopped forwarding logs in less than 5 min after this first restart
> so I didn't get any rsyslogd-pstats logs (I use the default option to send
> stats to the syslog stream with interval set to 5min).
>
> I restarted Rsyslog and this time it kept working a bit longer. After 1 hr
> Rsyslog stopped forwarding logs.
>
> Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:
>
> 2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
> origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0 discarded.nf=0
> maxqsize=145 *(last log)*
> 2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
> origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0 discarded.nf=0
> maxqsize=145
> 2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
> origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0 discarded.nf=0
> maxqsize=145
> 2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
> origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0 discarded.nf=0
> maxqsize=145
>
> Another filter in Splunk ( "rsyslogd-pstats" submitted NOT "origin=imptcp"
> NOT "origin=imtcp" ) gave me this:
>
> 2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
> 2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=91 disallowed=0
> 2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> origin=imuxsock submitted=6009 ratelimit.discarded=0
> ratelimit.numratelimiters=0
> 2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=0 disallowed=0
> 2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=83 disallowed=0
> 2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> origin=imuxsock submitted=5139 ratelimit.discarded=0
> ratelimit.numratelimiters=0
> 2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=0 disallowed=0
> 2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
> imudp(*:514): origin=imudp submitted=66 disallowed=0
> 2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> origin=imuxsock submitted=4657 ratelimit.discarded=0
> ratelimit.numratelimiters=0
>
> * I have the same issue with the another vCenter 6.7U3. *
> Rsyslog stops forwarding logs after a while.
>
> Both have the same Rsyslog version:
>
> rsyslogd 8.37.0, compiled with:
> PLATFORM: x86_64-unknown-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: No
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> systemd support: Yes
> Number of Bits in RainerScript integers: 64
>
> This is my /etc/rsyslog.conf (identical on both vCenter):
>
> ################################################################################
> ############################# VMware Rsyslog Configuration
> ####################
> ################################################################################
> ###### Module declarations ######
> $ModLoad impstats.so
> $ModLoad imuxsock.so
> $ModLoad imtcp.so # TLS
> $ModLoad imptcp.so # TCP
> $ModLoad imudp.so # UDP
> $ModLoad omrelp.so # RELP
> ###### Common configuration ######
> $EscapeControlCharactersOnReceive off
> ###### Template declarations ######
> $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %app-name% %msg%\n"
> $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template vpxdFmt,"%msg%\n"
> $template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %app-name% %msg%\n"
> $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
> $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %hostname% %app-name% %msg%\n"
> $template defaultSystemLoc,"/var/log/vmware/messages"
> ###### Rule declarations ######
> # TCP/UDP/rsyslog input ruleset declaration
> $RuleSet all
> # Make gtls driver the default
> $DefaultNetstreamDriver gtls
> # Shared certificate authority certificate
> $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
> # Client certificate
> $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
> # Client key
> $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
> # Include the configuration for syslog relay
> # _must_ be first to relay all messages
> $IncludeConfig /etc/vmware-syslog/syslog.conf
> # vmware services
> :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "rbd" ?defaultLoc;defaultFmt
> & stop
> :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
> & stop
> :programname, isequal, "vmon" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "pod" stop
> :programname, isequal, "updatemgr" stop
> # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "vpxd-svcs" stop
> # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "hvc" stop
> # vpxd logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "vpxd" stop
> # For local host's syslog and system logs use the following rules
> # localhost
> if $fromhost contains $$myhostname then ?defaultSystemLoc
> & stop
> #localhost
> :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
> & stop
> # ESX rules
> # Define large LinkedList action queue with 2K msgs cap to accomodate 100
> ESXs
> $ActionQueueSize 2000
> # Do not choke ESXs, rather start dropping messages after queue is 97.5%
> full
> $ActionQueueDiscardMark 1950
> $ActionQueueDiscardSeverity 0
> $ActionQueueTimeoutEnqueue 1
> # VC syslog server log collection
> *.* ?esxLoc;esxFmt
> ###### Input server declarations ######
> # Setup input flow
> $DefaultRuleset all
> $InputPTCPServerBindRuleset all
> $InputPTCPServerRun 514
> $InputUDPServerBindRuleset all
> $UDPServerRun 514
> $InputTCPServerBindRuleset all
> $InputTCPServerStreamDriverMode 1
> $InputTCPServerStreamDriverAuthMode anon
> $InputTCPServerRun 1514
> #
> # cron log entries for GEN003160
> #
> cron.* -/var/log/cron
>
> #
> # auth.log entries for GEN003660
> #
> auth.* -/var/log/auth.log
>
>
> *However*, in a 3rd vCenter, which is running a slightly different version
> (VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.
>
> This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:
>
> ################################################################################
> ############################# VMware Rsyslog Configuration
> ####################
> ################################################################################
> ###### Module declarations ######
> $ModLoad imuxsock.so
> $ModLoad imtcp.so # TLS
> $ModLoad imptcp.so # TCP
> $ModLoad imudp.so # UDP
> $ModLoad omrelp.so # RELP
> ###### Common configuration ######
> $EscapeControlCharactersOnReceive off
> ###### Template declarations ######
> $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %app-name% %msg%\n"
> $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template vpxdFmt,"%msg%\n"
> $template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> $template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %app-name% %msg%\n"
> $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
> $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> %hostname% %app-name% %msg%\n"
> $template defaultSystemLoc,"/var/log/vmware/messages"
> ###### Rule declarations ######
> # TCP/UDP/rsyslog input ruleset declaration
> $RuleSet all
> # Make gtls driver the default
> $DefaultNetstreamDriver gtls
> # Shared certificate authority certificate
> $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
> # Client certificate
> $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
> # Client key
> $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
> # Include the configuration for syslog relay
> # _must_ be first to relay all messages
> $IncludeConfig /etc/vmware-syslog/syslog.conf
> # vmware services
> :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "rbd" ?defaultLoc;defaultFmt
> & stop
> :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
> & stop
> :programname, isequal, "vmon" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
> & stop
> :programname, isequal, "pod" stop
> :programname, isequal, "updatemgr" stop
> # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "vpxd-svcs" stop
> # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "hvc" stop
> # vpxd logs to its local logs, hence avoiding duplicate logging.
> :programname, isequal, "vpxd" stop
> # For local host's syslog and system logs use the following rules
> # localhost
> if $fromhost contains $$myhostname then ?defaultSystemLoc
> & stop
> #localhost
> :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
> & stop
> # ESX rules
> # Define large LinkedList action queue with 2K msgs cap to accomodate 100
> ESXs
> $ActionQueueSize 2000
> # Do not choke ESXs, rather start dropping messages after queue is 97.5%
> full
> $ActionQueueDiscardMark 1950
> $ActionQueueDiscardSeverity 0
> $ActionQueueTimeoutEnqueue 1
> # VC syslog server log collection
> *.* ?esxLoc;esxFmt
> ###### Input server declarations ######
> # Setup input flow
> $DefaultRuleset all
> $InputPTCPServerBindRuleset all
> $InputPTCPServerRun 514
> $InputUDPServerBindRuleset all
> $UDPServerRun 514
> $InputTCPServerBindRuleset all
> $InputTCPServerStreamDriverMode 1
> $InputTCPServerStreamDriverAuthMode anon
> $InputTCPServerRun 1514
>
> The difference is at the end:
>
> The conf file *with the issue* has these lines:
>
> #
> # cron log entries for GEN003160
> #
> cron.* -/var/log/cron
>
> #
> # auth.log entries for GEN003660
> #
> auth.* -/var/log/auth.log
>
>
> I don't really know what that means and if it can explain the issue.
>
> Lucien
>
>
> On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <lucien.courcol@gmail.com>
> wrote:
>
>> Thx David Lang,
>>
>> I'll work on that tomorrow.
>>
>> I'll use this guide to get impstats logs:
>> https://www.rsyslog.com/how-to-use-impstats/
>>
>> Lucien
>>
>> On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:
>>
>>> wihtout seeing your rsyslog.conf we can't begin to guess what's
>>> happening.
>>> configuring impstats and showing us the output when it's stopped would
>>> help us
>>> to see what's happening.
>>>
>>> David Lang
>>>
>>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
>>>
>>>> Date: Wed, 11 Sep 2019 14:45:38 +0200
>>>> From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: rsyslog@lists.adiscon.com
>>>> Cc: Lucien Courcol <lucien.courcol@gmail.com>
>>>> Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
>>> ~10min
>>>> after starting
>>>>
>>>> Hello,
>>>>
>>>> We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
>>> and
>>>> we noticed a gap of logs in our syslog server (kiwi) since then.
>>>>
>>>> I did a bit of troubleshooting but Rsyslog (the syslog client running on
>>>> VCSA) is completely new to me.
>>>>
>>>> I use this command to restart Rsyslog:
>>>>
>>>> systemctl restart rsyslog
>>>>
>>>> Right after starting up Rsyslog, logs are being sent to our syslog
>>> server.
>>>>
>>>> ~10min later, no more logs are sent.
>>>>
>>>> The vCenter log file in our syslog server stops getting updated.
>>>> I did a tcpdump in our vCenter and I see that the vCenter stops sending
>>>> logs.
>>>> Using UDP or TCP doesn't fix the issue.
>>>> I looked for errors in various log files in the vCenter but can't find
>>>> anything.
>>>>
>>>> This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
>>> after
>>>> restarting Rsyslog
>>>>
>>>> 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
>>>> software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
>>>> http://www.rsyslog.com"] exiting on signal 15.
>>>> 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable
>>> TZ
>>>> is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>>>> http://www.rsyslog.com/e/2442 ]
>>>> 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
>>>> socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
>>>> 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
>>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>>>> http://www.rsyslog.com"] start
>>>>
>>>> Rsyslog is still running based on this command
>>>>
>>>> systemctl status rsyslog.service
>>>>
>>>> ? rsyslog.service - System Logging Service
>>>> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
>>> vendor
>>>> preset: enabled)
>>>> Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
>>>> Docs: man:rsyslogd(8)
>>>> http://www.rsyslog.com/doc/
>>>> Main PID: 22235 (rsyslogd)
>>>> Tasks: 12
>>>> Memory: 5.7M
>>>> CPU: 191ms
>>>> CGroup: /system.slice/rsyslog.service
>>>> ??22235 /usr/sbin/rsyslogd -n
>>>>
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
>>>> process exited, code=killed, status=9/KILL
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
>>>> Service.
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
>>>> entered failed state.
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
>>>> with result 'signal'.
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
>>>> Service...
>>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
>>>> Service.
>>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
>>> variable
>>>> TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>>>> http://www.rsyslog.com/e/2442 ]
>>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
>>>> UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
>>> [v8.37.0]
>>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
>>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>>>> http://www.rsyslog.com"] start
>>>> (real hostname has been replaced by vcenter.domain.local)
>>>>
>>>> I created a ticket at VMware support, but the agent wasn't able to find
>>> any
>>>> errors as well and she suggested to take a backup of our vCenter and
>>>> reinstall with a restore to get a fresh install of Photon OS since
>>> Rsyslog
>>>> is integrated in Photon OS. I'm not going to do that now, maybe as a
>>> last
>>>> troubleshooting step.
>>>>
>>>> In the meantime, do you guys have an idea? Wrong Rsyslog config?
>>>>
>>>> Thx for your help.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>> you DON'T LIKE THAT.
>>
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
Hello David,

Find attached 2 impstats log files (output to local file, interval of 1
min.)
Each file comes from a different vCenter with the same issue.
Rsyslog version is the same: 8.37.0

rsyslog-stats-vcenter-lab.log => Rsyslog stopped forwarding logs at 17:26:33
rsyslog-stats-vcenter-corp.log => Rsyslog stopped forwarding logs
at 17:09:42

Lucien

On Thu, Sep 12, 2019 at 1:08 PM David Lang <david@lang.hm> wrote:

> when you configure impstats, you can configure it to write to a local file
> and
> not to syslog, that should decouple it from whatever is causing it to
> freeze.I
> would also set it to log more frequently.
>
> all the options for impstats are listed at
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
>
> each timeperiod it spits out a good chunk of logs, we will want to see at
> least
> one complete set, ideally after it stops processing things otherwise.
>
> putting a queue on a write to disk ( *.* ?esxLoc;esxFmt ) is usually
> counterproductive, rsyslog spends more time managing the queue than just
> writing
> the file.
>
> David Lang
>
>
> On Thu, 12 Sep 2019, Lucien Courcol wrote:
>
> > Date: Thu, 12 Sep 2019 12:59:38 +0200
> > From: Lucien Courcol <lucien.courcol@gmail.com>
> > To: David Lang <david@lang.hm>
> > Cc: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
> > ~10min after starting
> >
> > Hello,
> >
> > So I loaded the impstats.so module and restarted Rsyslog.
> >
> > Rsyslog stopped forwarding logs in less than 5 min after this first
> restart
> > so I didn't get any rsyslogd-pstats logs (I use the default option to
> send
> > stats to the syslog stream with interval set to 5min).
> >
> > I restarted Rsyslog and this time it kept working a bit longer. After 1
> hr
> > Rsyslog stopped forwarding logs.
> >
> > Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:
> >
> > 2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
> > origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0
> discarded.nf=0
> > maxqsize=145 *(last log)*
> > 2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
> > origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0
> discarded.nf=0
> > maxqsize=145
> > 2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
> > origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0
> discarded.nf=0
> > maxqsize=145
> > 2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
> > origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0
> discarded.nf=0
> > maxqsize=145
> >
> > Another filter in Splunk ( "rsyslogd-pstats" submitted NOT
> "origin=imptcp"
> > NOT "origin=imtcp" ) gave me this:
> >
> > 2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
> > 2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=91 disallowed=0
> > 2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> > origin=imuxsock submitted=6009 ratelimit.discarded=0
> > ratelimit.numratelimiters=0
> > 2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=0 disallowed=0
> > 2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=83 disallowed=0
> > 2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> > origin=imuxsock submitted=5139 ratelimit.discarded=0
> > ratelimit.numratelimiters=0
> > 2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=0 disallowed=0
> > 2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
> > imudp(*:514): origin=imudp submitted=66 disallowed=0
> > 2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
> > origin=imuxsock submitted=4657 ratelimit.discarded=0
> > ratelimit.numratelimiters=0
> >
> > * I have the same issue with the another vCenter 6.7U3. *
> > Rsyslog stops forwarding logs after a while.
> >
> > Both have the same Rsyslog version:
> >
> > rsyslogd 8.37.0, compiled with:
> > PLATFORM: x86_64-unknown-linux-gnu
> > PLATFORM (lsb_release -d):
> > FEATURE_REGEXP: Yes
> > GSSAPI Kerberos 5 support: No
> > FEATURE_DEBUG (debug build, slow code): No
> > 32bit Atomic operations supported: Yes
> > 64bit Atomic operations supported: Yes
> > memory allocator: system default
> > Runtime Instrumentation (slow code): No
> > uuid support: Yes
> > systemd support: Yes
> > Number of Bits in RainerScript integers: 64
> >
> > This is my /etc/rsyslog.conf (identical on both vCenter):
> >
> >
> ################################################################################
> > ############################# VMware Rsyslog Configuration
> > ####################
> >
> ################################################################################
> > ###### Module declarations ######
> > $ModLoad impstats.so
> > $ModLoad imuxsock.so
> > $ModLoad imtcp.so # TLS
> > $ModLoad imptcp.so # TCP
> > $ModLoad imudp.so # UDP
> > $ModLoad omrelp.so # RELP
> > ###### Common configuration ######
> > $EscapeControlCharactersOnReceive off
> > ###### Template declarations ######
> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> > %app-name% %msg%\n"
> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template vpxdFmt,"%msg%\n"
> > $template
> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339%
> %syslogseverity-text%
> > %app-name% %msg%\n"
> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> > %hostname% %app-name% %msg%\n"
> > $template defaultSystemLoc,"/var/log/vmware/messages"
> > ###### Rule declarations ######
> > # TCP/UDP/rsyslog input ruleset declaration
> > $RuleSet all
> > # Make gtls driver the default
> > $DefaultNetstreamDriver gtls
> > # Shared certificate authority certificate
> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
> > # Client certificate
> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
> > # Client key
> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
> > # Include the configuration for syslog relay
> > # _must_ be first to relay all messages
> > $IncludeConfig /etc/vmware-syslog/syslog.conf
> > # vmware services
> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt
> > & stop
> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
> > & stop
> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "pod" stop
> > :programname, isequal, "updatemgr" stop
> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "vpxd-svcs" stop
> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "hvc" stop
> > # vpxd logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "vpxd" stop
> > # For local host's syslog and system logs use the following rules
> > # localhost
> > if $fromhost contains $$myhostname then ?defaultSystemLoc
> > & stop
> > #localhost
> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
> > & stop
> > # ESX rules
> > # Define large LinkedList action queue with 2K msgs cap to accomodate 100
> > ESXs
> > $ActionQueueSize 2000
> > # Do not choke ESXs, rather start dropping messages after queue is 97.5%
> > full
> > $ActionQueueDiscardMark 1950
> > $ActionQueueDiscardSeverity 0
> > $ActionQueueTimeoutEnqueue 1
> > # VC syslog server log collection
> > *.* ?esxLoc;esxFmt
> > ###### Input server declarations ######
> > # Setup input flow
> > $DefaultRuleset all
> > $InputPTCPServerBindRuleset all
> > $InputPTCPServerRun 514
> > $InputUDPServerBindRuleset all
> > $UDPServerRun 514
> > $InputTCPServerBindRuleset all
> > $InputTCPServerStreamDriverMode 1
> > $InputTCPServerStreamDriverAuthMode anon
> > $InputTCPServerRun 1514
> > #
> > # cron log entries for GEN003160
> > #
> > cron.* -/var/log/cron
> >
> > #
> > # auth.log entries for GEN003660
> > #
> > auth.* -/var/log/auth.log
> >
> >
> > *However*, in a 3rd vCenter, which is running a slightly different
> version
> > (VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.
> >
> > This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:
> >
> >
> ################################################################################
> > ############################# VMware Rsyslog Configuration
> > ####################
> >
> ################################################################################
> > ###### Module declarations ######
> > $ModLoad imuxsock.so
> > $ModLoad imtcp.so # TLS
> > $ModLoad imptcp.so # TCP
> > $ModLoad imudp.so # UDP
> > $ModLoad omrelp.so # RELP
> > ###### Common configuration ######
> > $EscapeControlCharactersOnReceive off
> > ###### Template declarations ######
> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> > %app-name% %msg%\n"
> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template vpxdFmt,"%msg%\n"
> > $template
> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339%
> %syslogseverity-text%
> > %app-name% %msg%\n"
> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
> > %hostname% %app-name% %msg%\n"
> > $template defaultSystemLoc,"/var/log/vmware/messages"
> > ###### Rule declarations ######
> > # TCP/UDP/rsyslog input ruleset declaration
> > $RuleSet all
> > # Make gtls driver the default
> > $DefaultNetstreamDriver gtls
> > # Shared certificate authority certificate
> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
> > # Client certificate
> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
> > # Client key
> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
> > # Include the configuration for syslog relay
> > # _must_ be first to relay all messages
> > $IncludeConfig /etc/vmware-syslog/syslog.conf
> > # vmware services
> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt
> > & stop
> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
> > & stop
> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
> > & stop
> > :programname, isequal, "pod" stop
> > :programname, isequal, "updatemgr" stop
> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "vpxd-svcs" stop
> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "hvc" stop
> > # vpxd logs to its local logs, hence avoiding duplicate logging.
> > :programname, isequal, "vpxd" stop
> > # For local host's syslog and system logs use the following rules
> > # localhost
> > if $fromhost contains $$myhostname then ?defaultSystemLoc
> > & stop
> > #localhost
> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
> > & stop
> > # ESX rules
> > # Define large LinkedList action queue with 2K msgs cap to accomodate 100
> > ESXs
> > $ActionQueueSize 2000
> > # Do not choke ESXs, rather start dropping messages after queue is 97.5%
> > full
> > $ActionQueueDiscardMark 1950
> > $ActionQueueDiscardSeverity 0
> > $ActionQueueTimeoutEnqueue 1
> > # VC syslog server log collection
> > *.* ?esxLoc;esxFmt
> > ###### Input server declarations ######
> > # Setup input flow
> > $DefaultRuleset all
> > $InputPTCPServerBindRuleset all
> > $InputPTCPServerRun 514
> > $InputUDPServerBindRuleset all
> > $UDPServerRun 514
> > $InputTCPServerBindRuleset all
> > $InputTCPServerStreamDriverMode 1
> > $InputTCPServerStreamDriverAuthMode anon
> > $InputTCPServerRun 1514
> >
> > The difference is at the end:
> >
> > The conf file *with the issue* has these lines:
> >
> > #
> > # cron log entries for GEN003160
> > #
> > cron.* -/var/log/cron
> >
> > #
> > # auth.log entries for GEN003660
> > #
> > auth.* -/var/log/auth.log
> >
> >
> > I don't really know what that means and if it can explain the issue.
> >
> > Lucien
> >
> >
> > On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <
> lucien.courcol@gmail.com>
> > wrote:
> >
> >> Thx David Lang,
> >>
> >> I'll work on that tomorrow.
> >>
> >> I'll use this guide to get impstats logs:
> >> https://www.rsyslog.com/how-to-use-impstats/
> >>
> >> Lucien
> >>
> >> On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:
> >>
> >>> wihtout seeing your rsyslog.conf we can't begin to guess what's
> >>> happening.
> >>> configuring impstats and showing us the output when it's stopped would
> >>> help us
> >>> to see what's happening.
> >>>
> >>> David Lang
> >>>
> >>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
> >>>
> >>>> Date: Wed, 11 Sep 2019 14:45:38 +0200
> >>>> From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
> >>>> To: rsyslog@lists.adiscon.com
> >>>> Cc: Lucien Courcol <lucien.courcol@gmail.com>
> >>>> Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
> >>> ~10min
> >>>> after starting
> >>>>
> >>>> Hello,
> >>>>
> >>>> We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
> >>> and
> >>>> we noticed a gap of logs in our syslog server (kiwi) since then.
> >>>>
> >>>> I did a bit of troubleshooting but Rsyslog (the syslog client running
> on
> >>>> VCSA) is completely new to me.
> >>>>
> >>>> I use this command to restart Rsyslog:
> >>>>
> >>>> systemctl restart rsyslog
> >>>>
> >>>> Right after starting up Rsyslog, logs are being sent to our syslog
> >>> server.
> >>>>
> >>>> ~10min later, no more logs are sent.
> >>>>
> >>>> The vCenter log file in our syslog server stops getting updated.
> >>>> I did a tcpdump in our vCenter and I see that the vCenter stops
> sending
> >>>> logs.
> >>>> Using UDP or TCP doesn't fix the issue.
> >>>> I looked for errors in various log files in the vCenter but can't find
> >>>> anything.
> >>>>
> >>>> This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
> >>> after
> >>>> restarting Rsyslog
> >>>>
> >>>> 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
> >>>> http://www.rsyslog.com"] exiting on signal 15.
> >>>> 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment
> variable
> >>> TZ
> >>>> is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> >>>> http://www.rsyslog.com/e/2442 ]
> >>>> 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired
> UNIX
> >>>> socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
> >>>> 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> >>>> http://www.rsyslog.com"] start
> >>>>
> >>>> Rsyslog is still running based on this command
> >>>>
> >>>> systemctl status rsyslog.service
> >>>>
> >>>> ? rsyslog.service - System Logging Service
> >>>> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
> >>> vendor
> >>>> preset: enabled)
> >>>> Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min
> ago
> >>>> Docs: man:rsyslogd(8)
> >>>> http://www.rsyslog.com/doc/
> >>>> Main PID: 22235 (rsyslogd)
> >>>> Tasks: 12
> >>>> Memory: 5.7M
> >>>> CPU: 191ms
> >>>> CGroup: /system.slice/rsyslog.service
> >>>> ??22235 /usr/sbin/rsyslogd -n
> >>>>
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
> >>>> process exited, code=killed, status=9/KILL
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System
> Logging
> >>>> Service.
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
> >>>> entered failed state.
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service:
> Failed
> >>>> with result 'signal'.
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System
> Logging
> >>>> Service...
> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System
> Logging
> >>>> Service.
> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
> >>> variable
> >>>> TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
> >>>> http://www.rsyslog.com/e/2442 ]
> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock:
> Acquired
> >>>> UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
> >>> [v8.37.0]
> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
> >>>> http://www.rsyslog.com"] start
> >>>> (real hostname has been replaced by vcenter.domain.local)
> >>>>
> >>>> I created a ticket at VMware support, but the agent wasn't able to
> find
> >>> any
> >>>> errors as well and she suggested to take a backup of our vCenter and
> >>>> reinstall with a restore to get a fresh install of Photon OS since
> >>> Rsyslog
> >>>> is integrated in Photon OS. I'm not going to do that now, maybe as a
> >>> last
> >>>> troubleshooting step.
> >>>>
> >>>> In the meantime, do you guys have an idea? Wrong Rsyslog config?
> >>>>
> >>>> Thx for your help.
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> >>> you DON'T LIKE THAT.
> >>
> >>
> >
Re: Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting [ In reply to ]
Hello David,

Have you received my previous email?
Did I configure the impstats module correctly?
I've added attachments and I was wondering if it was blocked or moved to
your spam folder.

Lucien

On Thu, Sep 12, 2019 at 6:00 PM Lucien Courcol <lucien.courcol@gmail.com>
wrote:

> Hello David,
>
> Find attached 2 impstats log files (output to local file, interval of 1
> min.)
> Each file comes from a different vCenter with the same issue.
> Rsyslog version is the same: 8.37.0
>
> rsyslog-stats-vcenter-lab.log => Rsyslog stopped forwarding logs at
> 17:26:33
> rsyslog-stats-vcenter-corp.log => Rsyslog stopped forwarding logs
> at 17:09:42
>
> Lucien
>
> On Thu, Sep 12, 2019 at 1:08 PM David Lang <david@lang.hm> wrote:
>
>> when you configure impstats, you can configure it to write to a local
>> file and
>> not to syslog, that should decouple it from whatever is causing it to
>> freeze.I
>> would also set it to log more frequently.
>>
>> all the options for impstats are listed at
>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
>>
>> each timeperiod it spits out a good chunk of logs, we will want to see at
>> least
>> one complete set, ideally after it stops processing things otherwise.
>>
>> putting a queue on a write to disk ( *.* ?esxLoc;esxFmt ) is usually
>> counterproductive, rsyslog spends more time managing the queue than just
>> writing
>> the file.
>>
>> David Lang
>>
>>
>> On Thu, 12 Sep 2019, Lucien Courcol wrote:
>>
>> > Date: Thu, 12 Sep 2019 12:59:38 +0200
>> > From: Lucien Courcol <lucien.courcol@gmail.com>
>> > To: David Lang <david@lang.hm>
>> > Cc: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
>> > Subject: Re: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops
>> working
>> > ~10min after starting
>> >
>> > Hello,
>> >
>> > So I loaded the impstats.so module and restarted Rsyslog.
>> >
>> > Rsyslog stopped forwarding logs in less than 5 min after this first
>> restart
>> > so I didn't get any rsyslogd-pstats logs (I use the default option to
>> send
>> > stats to the syslog stream with interval set to 5min).
>> >
>> > I restarted Rsyslog and this time it kept working a bit longer. After 1
>> hr
>> > Rsyslog stopped forwarding logs.
>> >
>> > Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:
>> >
>> > 2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
>> > origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0
>> discarded.nf=0
>> > maxqsize=145 *(last log)*
>> > 2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
>> > origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0
>> discarded.nf=0
>> > maxqsize=145
>> > 2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
>> > origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0
>> discarded.nf=0
>> > maxqsize=145
>> > 2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
>> > origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0
>> discarded.nf=0
>> > maxqsize=145
>> >
>> > Another filter in Splunk ( "rsyslogd-pstats" submitted NOT
>> "origin=imptcp"
>> > NOT "origin=imtcp" ) gave me this:
>> >
>> > 2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
>> > 2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=91 disallowed=0
>> > 2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
>> > origin=imuxsock submitted=6009 ratelimit.discarded=0
>> > ratelimit.numratelimiters=0
>> > 2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=0 disallowed=0
>> > 2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=83 disallowed=0
>> > 2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
>> > origin=imuxsock submitted=5139 ratelimit.discarded=0
>> > ratelimit.numratelimiters=0
>> > 2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=0 disallowed=0
>> > 2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
>> > imudp(*:514): origin=imudp submitted=66 disallowed=0
>> > 2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
>> > origin=imuxsock submitted=4657 ratelimit.discarded=0
>> > ratelimit.numratelimiters=0
>> >
>> > * I have the same issue with the another vCenter 6.7U3. *
>> > Rsyslog stops forwarding logs after a while.
>> >
>> > Both have the same Rsyslog version:
>> >
>> > rsyslogd 8.37.0, compiled with:
>> > PLATFORM: x86_64-unknown-linux-gnu
>> > PLATFORM (lsb_release -d):
>> > FEATURE_REGEXP: Yes
>> > GSSAPI Kerberos 5 support: No
>> > FEATURE_DEBUG (debug build, slow code): No
>> > 32bit Atomic operations supported: Yes
>> > 64bit Atomic operations supported: Yes
>> > memory allocator: system default
>> > Runtime Instrumentation (slow code): No
>> > uuid support: Yes
>> > systemd support: Yes
>> > Number of Bits in RainerScript integers: 64
>> >
>> > This is my /etc/rsyslog.conf (identical on both vCenter):
>> >
>> >
>> ################################################################################
>> > ############################# VMware Rsyslog Configuration
>> > ####################
>> >
>> ################################################################################
>> > ###### Module declarations ######
>> > $ModLoad impstats.so
>> > $ModLoad imuxsock.so
>> > $ModLoad imtcp.so # TLS
>> > $ModLoad imptcp.so # TCP
>> > $ModLoad imudp.so # UDP
>> > $ModLoad omrelp.so # RELP
>> > ###### Common configuration ######
>> > $EscapeControlCharactersOnReceive off
>> > ###### Template declarations ######
>> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
>> > %app-name% %msg%\n"
>> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template vpxdFmt,"%msg%\n"
>> > $template
>> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339%
>> %syslogseverity-text%
>> > %app-name% %msg%\n"
>> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
>> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
>> > %hostname% %app-name% %msg%\n"
>> > $template defaultSystemLoc,"/var/log/vmware/messages"
>> > ###### Rule declarations ######
>> > # TCP/UDP/rsyslog input ruleset declaration
>> > $RuleSet all
>> > # Make gtls driver the default
>> > $DefaultNetstreamDriver gtls
>> > # Shared certificate authority certificate
>> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
>> > # Client certificate
>> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
>> > # Client key
>> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
>> > # Include the configuration for syslog relay
>> > # _must_ be first to relay all messages
>> > $IncludeConfig /etc/vmware-syslog/syslog.conf
>> > # vmware services
>> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt
>> > & stop
>> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
>> > & stop
>> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "pod" stop
>> > :programname, isequal, "updatemgr" stop
>> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "vpxd-svcs" stop
>> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "hvc" stop
>> > # vpxd logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "vpxd" stop
>> > # For local host's syslog and system logs use the following rules
>> > # localhost
>> > if $fromhost contains $$myhostname then ?defaultSystemLoc
>> > & stop
>> > #localhost
>> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
>> > & stop
>> > # ESX rules
>> > # Define large LinkedList action queue with 2K msgs cap to accomodate
>> 100
>> > ESXs
>> > $ActionQueueSize 2000
>> > # Do not choke ESXs, rather start dropping messages after queue is 97.5%
>> > full
>> > $ActionQueueDiscardMark 1950
>> > $ActionQueueDiscardSeverity 0
>> > $ActionQueueTimeoutEnqueue 1
>> > # VC syslog server log collection
>> > *.* ?esxLoc;esxFmt
>> > ###### Input server declarations ######
>> > # Setup input flow
>> > $DefaultRuleset all
>> > $InputPTCPServerBindRuleset all
>> > $InputPTCPServerRun 514
>> > $InputUDPServerBindRuleset all
>> > $UDPServerRun 514
>> > $InputTCPServerBindRuleset all
>> > $InputTCPServerStreamDriverMode 1
>> > $InputTCPServerStreamDriverAuthMode anon
>> > $InputTCPServerRun 1514
>> > #
>> > # cron log entries for GEN003160
>> > #
>> > cron.* -/var/log/cron
>> >
>> > #
>> > # auth.log entries for GEN003660
>> > #
>> > auth.* -/var/log/auth.log
>> >
>> >
>> > *However*, in a 3rd vCenter, which is running a slightly different
>> version
>> > (VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.
>> >
>> > This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:
>> >
>> >
>> ################################################################################
>> > ############################# VMware Rsyslog Configuration
>> > ####################
>> >
>> ################################################################################
>> > ###### Module declarations ######
>> > $ModLoad imuxsock.so
>> > $ModLoad imtcp.so # TLS
>> > $ModLoad imptcp.so # TCP
>> > $ModLoad imudp.so # UDP
>> > $ModLoad omrelp.so # RELP
>> > ###### Common configuration ######
>> > $EscapeControlCharactersOnReceive off
>> > ###### Template declarations ######
>> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
>> > %app-name% %msg%\n"
>> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template vpxdFmt,"%msg%\n"
>> > $template
>> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
>> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339%
>> %syslogseverity-text%
>> > %app-name% %msg%\n"
>> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
>> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
>> > %hostname% %app-name% %msg%\n"
>> > $template defaultSystemLoc,"/var/log/vmware/messages"
>> > ###### Rule declarations ######
>> > # TCP/UDP/rsyslog input ruleset declaration
>> > $RuleSet all
>> > # Make gtls driver the default
>> > $DefaultNetstreamDriver gtls
>> > # Shared certificate authority certificate
>> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
>> > # Client certificate
>> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
>> > # Client key
>> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
>> > # Include the configuration for syslog relay
>> > # _must_ be first to relay all messages
>> > $IncludeConfig /etc/vmware-syslog/syslog.conf
>> > # vmware services
>> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt
>> > & stop
>> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
>> > & stop
>> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
>> > & stop
>> > :programname, isequal, "pod" stop
>> > :programname, isequal, "updatemgr" stop
>> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "vpxd-svcs" stop
>> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "hvc" stop
>> > # vpxd logs to its local logs, hence avoiding duplicate logging.
>> > :programname, isequal, "vpxd" stop
>> > # For local host's syslog and system logs use the following rules
>> > # localhost
>> > if $fromhost contains $$myhostname then ?defaultSystemLoc
>> > & stop
>> > #localhost
>> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
>> > & stop
>> > # ESX rules
>> > # Define large LinkedList action queue with 2K msgs cap to accomodate
>> 100
>> > ESXs
>> > $ActionQueueSize 2000
>> > # Do not choke ESXs, rather start dropping messages after queue is 97.5%
>> > full
>> > $ActionQueueDiscardMark 1950
>> > $ActionQueueDiscardSeverity 0
>> > $ActionQueueTimeoutEnqueue 1
>> > # VC syslog server log collection
>> > *.* ?esxLoc;esxFmt
>> > ###### Input server declarations ######
>> > # Setup input flow
>> > $DefaultRuleset all
>> > $InputPTCPServerBindRuleset all
>> > $InputPTCPServerRun 514
>> > $InputUDPServerBindRuleset all
>> > $UDPServerRun 514
>> > $InputTCPServerBindRuleset all
>> > $InputTCPServerStreamDriverMode 1
>> > $InputTCPServerStreamDriverAuthMode anon
>> > $InputTCPServerRun 1514
>> >
>> > The difference is at the end:
>> >
>> > The conf file *with the issue* has these lines:
>> >
>> > #
>> > # cron log entries for GEN003160
>> > #
>> > cron.* -/var/log/cron
>> >
>> > #
>> > # auth.log entries for GEN003660
>> > #
>> > auth.* -/var/log/auth.log
>> >
>> >
>> > I don't really know what that means and if it can explain the issue.
>> >
>> > Lucien
>> >
>> >
>> > On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <
>> lucien.courcol@gmail.com>
>> > wrote:
>> >
>> >> Thx David Lang,
>> >>
>> >> I'll work on that tomorrow.
>> >>
>> >> I'll use this guide to get impstats logs:
>> >> https://www.rsyslog.com/how-to-use-impstats/
>> >>
>> >> Lucien
>> >>
>> >> On Wed, Sep 11, 2019 at 7:56 PM David Lang <david@lang.hm> wrote:
>> >>
>> >>> wihtout seeing your rsyslog.conf we can't begin to guess what's
>> >>> happening.
>> >>> configuring impstats and showing us the output when it's stopped would
>> >>> help us
>> >>> to see what's happening.
>> >>>
>> >>> David Lang
>> >>>
>> >>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
>> >>>
>> >>>> Date: Wed, 11 Sep 2019 14:45:38 +0200
>> >>>> From: Lucien Courcol via rsyslog <rsyslog@lists.adiscon.com>
>> >>>> To: rsyslog@lists.adiscon.com
>> >>>> Cc: Lucien Courcol <lucien.courcol@gmail.com>
>> >>>> Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
>> >>> ~10min
>> >>>> after starting
>> >>>>
>> >>>> Hello,
>> >>>>
>> >>>> We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
>> >>> and
>> >>>> we noticed a gap of logs in our syslog server (kiwi) since then.
>> >>>>
>> >>>> I did a bit of troubleshooting but Rsyslog (the syslog client
>> running on
>> >>>> VCSA) is completely new to me.
>> >>>>
>> >>>> I use this command to restart Rsyslog:
>> >>>>
>> >>>> systemctl restart rsyslog
>> >>>>
>> >>>> Right after starting up Rsyslog, logs are being sent to our syslog
>> >>> server.
>> >>>>
>> >>>> ~10min later, no more logs are sent.
>> >>>>
>> >>>> The vCenter log file in our syslog server stops getting updated.
>> >>>> I did a tcpdump in our vCenter and I see that the vCenter stops
>> sending
>> >>>> logs.
>> >>>> Using UDP or TCP doesn't fix the issue.
>> >>>> I looked for errors in various log files in the vCenter but can't
>> find
>> >>>> anything.
>> >>>>
>> >>>> This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
>> >>> after
>> >>>> restarting Rsyslog
>> >>>>
>> >>>> 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
>> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
>> >>>> http://www.rsyslog.com"] exiting on signal 15.
>> >>>> 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment
>> variable
>> >>> TZ
>> >>>> is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
>> >>>> http://www.rsyslog.com/e/2442 ]
>> >>>> 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired
>> UNIX
>> >>>> socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
>> >>>> 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
>> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> >>>> http://www.rsyslog.com"] start
>> >>>>
>> >>>> Rsyslog is still running based on this command
>> >>>>
>> >>>> systemctl status rsyslog.service
>> >>>>
>> >>>> ? rsyslog.service - System Logging Service
>> >>>> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
>> >>> vendor
>> >>>> preset: enabled)
>> >>>> Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min
>> ago
>> >>>> Docs: man:rsyslogd(8)
>> >>>> http://www.rsyslog.com/doc/
>> >>>> Main PID: 22235 (rsyslogd)
>> >>>> Tasks: 12
>> >>>> Memory: 5.7M
>> >>>> CPU: 191ms
>> >>>> CGroup: /system.slice/rsyslog.service
>> >>>> ??22235 /usr/sbin/rsyslogd -n
>> >>>>
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service:
>> Main
>> >>>> process exited, code=killed, status=9/KILL
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System
>> Logging
>> >>>> Service.
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service:
>> Unit
>> >>>> entered failed state.
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service:
>> Failed
>> >>>> with result 'signal'.
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System
>> Logging
>> >>>> Service...
>> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System
>> Logging
>> >>>> Service.
>> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
>> >>> variable
>> >>>> TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0
>> try
>> >>>> http://www.rsyslog.com/e/2442 ]
>> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock:
>> Acquired
>> >>>> UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
>> >>> [v8.37.0]
>> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
>> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
>> >>>> http://www.rsyslog.com"] start
>> >>>> (real hostname has been replaced by vcenter.domain.local)
>> >>>>
>> >>>> I created a ticket at VMware support, but the agent wasn't able to
>> find
>> >>> any
>> >>>> errors as well and she suggested to take a backup of our vCenter and
>> >>>> reinstall with a restore to get a fresh install of Photon OS since
>> >>> Rsyslog
>> >>>> is integrated in Photon OS. I'm not going to do that now, maybe as a
>> >>> last
>> >>>> troubleshooting step.
>> >>>>
>> >>>> In the meantime, do you guys have an idea? Wrong Rsyslog config?
>> >>>>
>> >>>> Thx for your help.
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com/professional-services/
>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if
>> >>> you DON'T LIKE THAT.
>> >>
>> >>
>> >
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.