Caveat: I've recently inherited a bunch of ubuntu systems with very little historical knowledge available to me. The issue described below has occurred only since the inline ubuntu upgrade was done. However, Ive no vision of whether this is solely an issue with ubuntu implementation of rsyslog, or it's a higher version of rsyslog that has this issue across all platforms
Systems: Ubuntu 20 (recent upgrade in-line from ubuntu 18)
Scenario:
While servers were ubuntu 18, clients had a a application that wrote logs into a non-standard system log /var/log/node/Tlog.log (name obfuscated [cid:image001.gif@01DA9232.6FE9BBE0] )
Previous rsyslog.conf configurations were in place to capture and include this log into rsyslog for central syslogging purposes where the log would appear on the central syslog server in /var/log/external/<client>/node/Tlog-<date>.log
client config:
$ModLoad imfile
...
$InputFileName /var/log/node/Tlog.log
$InputFileTag tserv-stdout
$InputFileStateFile tserv-stdout
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor
That worked well for years allegedly/apparently.
a few months ago these systems were upgraded to ubuntu 20. It was shortly afterwards noted that the central logging wasn't working well... logs were very sporadic if at all.
Then it was discovered that somewhere along the line the ubuntu 20 systems rsyslog config was needed to be altered (who knew?)
So now the client has
module(
load = "imfile"
pollingInterval = "1"
statefile.directory = "/var/log/node"
)
...
input(
type = "imfile"
tag = "tserv-stdout"
facility = "local4"
severity = "info"
file = "/var/log/node/Tlog.log"
)
That seemed to fix matters - logs to Tlog.log on the client were appearing in the central syslog log as well.
then about a week ago it just stopped working. All configs are as above ie the ones that were working. rsyslogd is running. nothing obvious is in any other log to suggest issues with rsyslog. Other system logs are updated centrally as normal. It's just this extraneous log that has stopped "working". Tlog.log on the client is constantly updated via its app (ie it is not a silent/inactive log)
Any thoughts/pointers as to what the proverbial is happening, or how better to troubleshoot it?
I've chucked a local4.* /var/log/node/Tlog.log entry into rsyslog.conf and logger -p local4.info logs locally and centrally as expected using that - so its not that eg networks have decided to block that access etc .
and FWIW, both systems (client and rsyslog server) have this version of rsyslogd
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
If anyone can rid me of this turbulent issue.... I would be immensely grateful...
ian
Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
Systems: Ubuntu 20 (recent upgrade in-line from ubuntu 18)
Scenario:
While servers were ubuntu 18, clients had a a application that wrote logs into a non-standard system log /var/log/node/Tlog.log (name obfuscated [cid:image001.gif@01DA9232.6FE9BBE0] )
Previous rsyslog.conf configurations were in place to capture and include this log into rsyslog for central syslogging purposes where the log would appear on the central syslog server in /var/log/external/<client>/node/Tlog-<date>.log
client config:
$ModLoad imfile
...
$InputFileName /var/log/node/Tlog.log
$InputFileTag tserv-stdout
$InputFileStateFile tserv-stdout
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor
That worked well for years allegedly/apparently.
a few months ago these systems were upgraded to ubuntu 20. It was shortly afterwards noted that the central logging wasn't working well... logs were very sporadic if at all.
Then it was discovered that somewhere along the line the ubuntu 20 systems rsyslog config was needed to be altered (who knew?)
So now the client has
module(
load = "imfile"
pollingInterval = "1"
statefile.directory = "/var/log/node"
)
...
input(
type = "imfile"
tag = "tserv-stdout"
facility = "local4"
severity = "info"
file = "/var/log/node/Tlog.log"
)
That seemed to fix matters - logs to Tlog.log on the client were appearing in the central syslog log as well.
then about a week ago it just stopped working. All configs are as above ie the ones that were working. rsyslogd is running. nothing obvious is in any other log to suggest issues with rsyslog. Other system logs are updated centrally as normal. It's just this extraneous log that has stopped "working". Tlog.log on the client is constantly updated via its app (ie it is not a silent/inactive log)
Any thoughts/pointers as to what the proverbial is happening, or how better to troubleshoot it?
I've chucked a local4.* /var/log/node/Tlog.log entry into rsyslog.conf and logger -p local4.info logs locally and centrally as expected using that - so its not that eg networks have decided to block that access etc .
and FWIW, both systems (client and rsyslog server) have this version of rsyslogd
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
If anyone can rid me of this turbulent issue.... I would be immensely grateful...
ian
Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.