Mailing List Archive: imfile rsyslog config sporadic since upgrade to ubuntu20

imfile rsyslog config sporadic since upgrade to ubuntu20

Apr 19, 2024, 12:20 AM

Post #1 of 14 (78 views)

Caveat: I've recently inherited a bunch of ubuntu systems with very little historical knowledge available to me. The issue described below has occurred only since the inline ubuntu upgrade was done. However, Ive no vision of whether this is solely an issue with ubuntu implementation of rsyslog, or it's a higher version of rsyslog that has this issue across all platforms

Systems: Ubuntu 20 (recent upgrade in-line from ubuntu 18)

Scenario:
While servers were ubuntu 18, clients had a a application that wrote logs into a non-standard system log /var/log/node/Tlog.log (name obfuscated [cid:image001.gif@01DA9232.6FE9BBE0] )
Previous rsyslog.conf configurations were in place to capture and include this log into rsyslog for central syslogging purposes where the log would appear on the central syslog server in /var/log/external/<client>/node/Tlog-<date>.log

client config:

$ModLoad imfile
...
$InputFileName /var/log/node/Tlog.log
$InputFileTag tserv-stdout
$InputFileStateFile tserv-stdout
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor

That worked well for years allegedly/apparently.

a few months ago these systems were upgraded to ubuntu 20. It was shortly afterwards noted that the central logging wasn't working well... logs were very sporadic if at all.

Then it was discovered that somewhere along the line the ubuntu 20 systems rsyslog config was needed to be altered (who knew?)

So now the client has

module(
load = "imfile"
pollingInterval = "1"
statefile.directory = "/var/log/node"
)
...
input(
type = "imfile"
tag = "tserv-stdout"
facility = "local4"
severity = "info"
file = "/var/log/node/Tlog.log"
)

That seemed to fix matters - logs to Tlog.log on the client were appearing in the central syslog log as well.

then about a week ago it just stopped working. All configs are as above ie the ones that were working. rsyslogd is running. nothing obvious is in any other log to suggest issues with rsyslog. Other system logs are updated centrally as normal. It's just this extraneous log that has stopped "working". Tlog.log on the client is constantly updated via its app (ie it is not a silent/inactive log)

Any thoughts/pointers as to what the proverbial is happening, or how better to troubleshoot it?

I've chucked a local4.* /var/log/node/Tlog.log entry into rsyslog.conf and logger -p local4.info logs locally and centrally as expected using that - so its not that eg networks have decided to block that access etc .

and FWIW, both systems (client and rsyslog server) have this version of rsyslogd

rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64

If anyone can rid me of this turbulent issue.... I would be immensely grateful...

ian
Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.

Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 19, 2024, 12:39 AM

Post #2 of 14 (78 views)

Permalink

> Caveat: I've recently inherited a bunch of ubuntu systems with very little
> historical knowledge available to me. The issue described below has occurred
> only since the inline ubuntu upgrade was done. However, Ive no vision of
> whether this is solely an issue with ubuntu implementation of rsyslog, or it's
> a higher version of rsyslog that has this issue across all platforms
>
> Systems: Ubuntu 20 (recent upgrade in-line from ubuntu 18)
>
> Scenario:
>
> While servers were ubuntu 18, clients had a a application that wrote logs into
> a non-standard system log /var/log/node/Tlog.log (name obfuscated
> [cid:image001.gif@01DA9232.6FE9BBE0] )
>
> Previous rsyslog.conf configurations were in place to capture and include this
> log into rsyslog for central syslogging purposes where the log would appear on
> the central syslog server in /var/log/external/<client>/node/Tlog-<date>.log
>
> client config:
>
> $ModLoad imfile
> ...
> $InputFileName /var/log/node/Tlog.log
> $InputFileTag tserv-stdout
> $InputFileStateFile tserv-stdout
> $InputFileSeverity info
> $InputFileFacility local4
> $InputRunFileMonitor
>
> That worked well for years allegedly/apparently.
>
> a few months ago these systems were upgraded to ubuntu 20. It was shortly
> afterwards noted that the central logging wasn't working well... logs were
> very sporadic if at all.
>
> Then it was discovered that somewhere along the line the ubuntu 20 systems
> rsyslog config was needed to be altered (who knew?)
>
> So now the client has
>
> module(
> load = "imfile"
> pollingInterval = "1"
> statefile.directory = "/var/log/node"
> )
> ...
> input(
> type = "imfile"
> tag = "tserv-stdout"
> facility = "local4"
> severity = "info"
> file = "/var/log/node/Tlog.log"
> )

Rsyslog tries very hard to not break backwards compatibility, so you should not
have needed to change the config. There have been bugs over the years, but in
general, a config should just keep working.

> That seemed to fix matters - logs to Tlog.log on the client were appearing in
> the central syslog log as well.

good, that should mean that the new style config is working

> then about a week ago it just stopped working. All configs are as above ie the
> ones that were working. rsyslogd is running. nothing obvious is in any other
> log to suggest issues with rsyslog. Other system logs are updated centrally as
> normal. It's just this extraneous log that has stopped "working". Tlog.log on
> the client is constantly updated via its app (ie it is not a silent/inactive
> log)

other logs from the same systems?

> Any thoughts/pointers as to what the proverbial is happening, or how better to
> troubleshoot it?

> I've chucked a local4.* /var/log/node/Tlog.log entry into rsyslog.conf and
> logger -p local4.info logs locally and centrally as expected using that - so
> its not that eg networks have decided to block that access etc .

This is a good start. But at this point I am not understanding the problem. You
say that with this config it is logging both locally and centrally as expected,
what isn't working as expected?

> and FWIW, both systems (client and rsyslog server) have this version of
> rsyslogd
>
> rsyslogd 8.2001.0 (aka 2020.01) compiled with:
> PLATFORM: x86_64-pc-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> systemd support: Yes
> Config file: /etc/rsyslog.conf
> PID file: /run/rsyslogd.pid
> Number of Bits in RainerScript integers: 64

one thing here is that, as this notes, you are using a version of rsyslog
initially released in Jan 2020, that has some number of changes backported to it
by Canonical, this is not a version provided by the rsyslog project (just based
on it), if you can upgrade to a current version to get the 4+ years of changes
and bugfixes (which do include a complete imfile rewrite amoung other things),
we would be in better shape to help. If you can't do that, we will do what we
can, but are far more limited in what we can do with such an old version.

based on your test, it sounds as if imfile is reading things, but not matching
something else on your central system. can you provide more info about the
config there?

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 19, 2024, 2:24 AM

Post #3 of 14 (78 views)

Permalink

>>Rsyslog tries very hard to not break backwards compatibility, so you should not have needed to change the config. There have been bugs over the years, but in >>general, a config should just keep working.

That of course makes perfect sense. Though as it turned it - come the upgrade 18-> 20 ... it didn’t work at all.

>> That seemed to fix matters - logs to Tlog.log on the client were
>> appearing in the central syslog log as well.
>good, that should mean that the new style config is working

It should.

But ... it ain't now....

>> other logs from the same systems?
other system logs handled by rsyslog.conf all work as expected.
Adding local4.* to that /var/log/node/Tlog.log works for
logger -p local4.info TEST
AND
that log gets held centrally.

But the app that wroites to /var/log/node/Tlog.log doesn’t use native local4.<whatever> ... no idea what it does but it wortes direct to Tlog.log (that’s down to devs years ago etc Id imagine)

>>This is a good start. But at this point I am not understanding the problem. You say that with this config it is logging both locally and centrally as expected, what >>isn't working as expected?

No.

anything set up "as standard" in rsyslog.con f works, and logs centrally. As expected.

This Tlog.log is written to via some other means

There is a historical config (up to Ubuntu 18) where a rsyslog.d config file using imfile DID work and logged centrally

Then that stopped working on the upgrade to Ubuntu 20.
But we found that an different working configuration was required - so implemebnted that and the devs tell me it all then worked.

But a week or so ago that updated config stopped working.

ie the imfile stuff to capture a nmon standard rsyslog log no longer works.

>>based on your test, it sounds as if imfile is reading things, but not matching something else on your central system. can you provide more info about the config >>there?

You asked! ????

# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf

#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

$ModLoad imrelp
$InputRELPServerRun 514

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup adm

#
# Where to place spool files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

$template syslog,"/var/log/external/%fromhost%/syslog-%$YEAR%%$MONTH%%$DAY%.log"
$template apacheError,"/var/log/external/%fromhost%/apache/%programname%-error-%$YEAR%%$MONTH%%$DAY%.log"
$template apacheAccess,"/var/log/external/%fromhost%/apache/%programname%-access-%$YEAR%%$MONTH%%$DAY%.log"
$template mailError, "/var/log/external/%fromhost%/mail/error-%$YEAR%%$MONTH%%$DAY%.log"
$template nodeStd, "/var/log/external/%fromhost%/node/TStd-%$YEAR%%$MONTH%%$DAY%.log"
$template nodeTService, "/var/log/external/%fromhost%/node/TLog-%$YEAR%%$MONTH%%$DAY%.log"

local4.* ?nodeService
#& ~
& stop

local5.* ?nodeStd
#& ~
& stop

local7.* ?apacheError
#& ~
& stop

local6.* ?apacheAccess
#& ~
& stop

*.* ?syslog

That hasn’t changed for about 11 years.
That rsyslog central server is also recently upgraded to Ubuntu20 from Ubuntu18

FWIW Ive only talked about TLog - but TStd does the same thing (and has a similar imfile/local5 config on the client as top the imfile/local4)

Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 19, 2024, 3:59 AM

Post #4 of 14 (78 views)

Permalink

Is there any chance that they are getting logged under a different hostname?

David Lang

On Fri, 19 Apr 2024, Ian Diddams via rsyslog wrote:

> Date: Fri, 19 Apr 2024 09:24:03 +0000
> From: Ian Diddams via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: Ian Diddams <ian.diddams@celebrus.com>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since
> upgrade to ubuntu20
>
>>> Rsyslog tries very hard to not break backwards compatibility, so you should not have needed to change the config. There have been bugs over the years, but in >>general, a config should just keep working.
>
> That of course makes perfect sense. Though as it turned it - come the upgrade 18-> 20 ... it didn’t work at all.
>
>
>>> That seemed to fix matters - logs to Tlog.log on the client were
>>> appearing in the central syslog log as well.
>> good, that should mean that the new style config is working
>
> It should.
>
> But ... it ain't now....
>
>
>>> other logs from the same systems?
> other system logs handled by rsyslog.conf all work as expected.
> Adding local4.* to that /var/log/node/Tlog.log works for
> logger -p local4.info TEST
> AND
> that log gets held centrally.
>
> But the app that wroites to /var/log/node/Tlog.log doesn’t use native local4.<whatever> ... no idea what it does but it wortes direct to Tlog.log (that’s down to devs years ago etc Id imagine)
>
>
>>> This is a good start. But at this point I am not understanding the problem. You say that with this config it is logging both locally and centrally as expected, what >>isn't working as expected?
>
>
> No.
>
> anything set up "as standard" in rsyslog.con f works, and logs centrally. As expected.
>
> This Tlog.log is written to via some other means
>
> There is a historical config (up to Ubuntu 18) where a rsyslog.d config file using imfile DID work and logged centrally
>
> Then that stopped working on the upgrade to Ubuntu 20.
> But we found that an different working configuration was required - so implemebnted that and the devs tell me it all then worked.
>
> But a week or so ago that updated config stopped working.
>
> ie the imfile stuff to capture a nmon standard rsyslog log no longer works.
>
>
>>> based on your test, it sounds as if imfile is reading things, but not matching something else on your central system. can you provide more info about the config >>there?
>
> You asked! ????
>
>
> # /etc/rsyslog.conf Configuration file for rsyslog.
> #
> # For more information see
> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
> #
> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>
>
> #################
> #### MODULES ####
> #################
>
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog # provides kernel logging support (previously done by rklogd)
> #$ModLoad immark # provides --MARK-- message capability
>
> # provides UDP syslog reception
> $ModLoad imudp
> $UDPServerRun 514
>
> # provides TCP syslog reception
> #$ModLoad imtcp
> #$InputTCPServerRun 514
>
> $ModLoad imrelp
> $InputRELPServerRun 514
>
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
>
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Filter duplicated messages
> $RepeatedMsgReduction on
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> #$PrivDropToUser syslog
> #$PrivDropToGroup adm
>
> #
> # Where to place spool files
> #
> $WorkDirectory /var/spool/rsyslog
>
> #
> # Include all config files in /etc/rsyslog.d/
> #
> $IncludeConfig /etc/rsyslog.d/*.conf
>
> $template syslog,"/var/log/external/%fromhost%/syslog-%$YEAR%%$MONTH%%$DAY%.log"
> $template apacheError,"/var/log/external/%fromhost%/apache/%programname%-error-%$YEAR%%$MONTH%%$DAY%.log"
> $template apacheAccess,"/var/log/external/%fromhost%/apache/%programname%-access-%$YEAR%%$MONTH%%$DAY%.log"
> $template mailError, "/var/log/external/%fromhost%/mail/error-%$YEAR%%$MONTH%%$DAY%.log"
> $template nodeStd, "/var/log/external/%fromhost%/node/TStd-%$YEAR%%$MONTH%%$DAY%.log"
> $template nodeTService, "/var/log/external/%fromhost%/node/TLog-%$YEAR%%$MONTH%%$DAY%.log"
>
> local4.* ?nodeService
> #& ~
> & stop
>
> local5.* ?nodeStd
> #& ~
> & stop
>
> local7.* ?apacheError
> #& ~
> & stop
>
> local6.* ?apacheAccess
> #& ~
> & stop
>
> *.* ?syslog
>
> That hasn’t changed for about 11 years.
> That rsyslog central server is also recently upgraded to Ubuntu20 from Ubuntu18
>
> FWIW Ive only talked about TLog - but TStd does the same thing (and has a similar imfile/local5 config on the client as top the imfile/local4)
>
>
>
>
>
> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 19, 2024, 4:44 AM

Post #5 of 14 (78 views)

Permalink

specifically look for 127.0.0.1 or localhost

If you can log anything that's local4 on the server to a single file (ideally
using the template RSYSLOG_DebugFormat so we can see all the variables that are
parsed from it) it may be easier to find the log than your current dynafile
approach that puts them in different directories based on the hostname.

David Lang

On Fri, 19 Apr 2024, David Lang via rsyslog wrote:

> Date: Fri, 19 Apr 2024 03:59:53 -0700 (PDT)
> From: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> To: Ian Diddams via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since
> upgrade to ubuntu20
>
> Is there any chance that they are getting logged under a different hostname?
>
> David Lang
>
> On Fri, 19 Apr 2024, Ian Diddams via rsyslog wrote:
>
>> Date: Fri, 19 Apr 2024 09:24:03 +0000
>> From: Ian Diddams via rsyslog <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: Ian Diddams <ian.diddams@celebrus.com>
>> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since
>> upgrade to ubuntu20
>>
>>>> Rsyslog tries very hard to not break backwards compatibility, so you
> should not have needed to change the config. There have been bugs over the
> years, but in >>general, a config should just keep working.
>>
>> That of course makes perfect sense. Though as it turned it - come the
> upgrade 18-> 20 ... it didn’t work at all.
>>
>>
>>>> That seemed to fix matters - logs to Tlog.log on the client were
>>>> appearing in the central syslog log as well.
>>> good, that should mean that the new style config is working
>>
>> It should.
>>
>> But ... it ain't now....
>>
>>
>>>> other logs from the same systems?
>> other system logs handled by rsyslog.conf all work as expected.
>> Adding local4.* to that /var/log/node/Tlog.log works for
>> logger -p local4.info TEST
>> AND
>> that log gets held centrally.
>>
>> But the app that wroites to /var/log/node/Tlog.log doesn’t use native
> local4.<whatever> ... no idea what it does but it wortes direct to Tlog.log
> (that’s down to devs years ago etc Id imagine)
>>
>>
>>>> This is a good start. But at this point I am not understanding the
> problem. You say that with this config it is logging both locally and
> centrally as expected, what >>isn't working as expected?
>>
>>
>> No.
>>
>> anything set up "as standard" in rsyslog.con f works, and logs centrally.
> As expected.
>>
>> This Tlog.log is written to via some other means
>>
>> There is a historical config (up to Ubuntu 18) where a rsyslog.d config
> file using imfile DID work and logged centrally
>>
>> Then that stopped working on the upgrade to Ubuntu 20.
>> But we found that an different working configuration was required - so
> implemebnted that and the devs tell me it all then worked.
>>
>> But a week or so ago that updated config stopped working.
>>
>> ie the imfile stuff to capture a nmon standard rsyslog log no longer works.
>>
>>
>>>> based on your test, it sounds as if imfile is reading things, but not
> matching something else on your central system. can you provide more info
> about the config >>there?
>>
>> You asked! ????
>>
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> $ModLoad imuxsock # provides support for local system logging
>> $ModLoad imklog # provides kernel logging support (previously done by
> rklogd)
>> #$ModLoad immark # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> $ModLoad imudp
>> $UDPServerRun 514
>>
>> # provides TCP syslog reception
>> #$ModLoad imtcp
>> #$InputTCPServerRun 514
>>
>> $ModLoad imrelp
>> $InputRELPServerRun 514
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> #$PrivDropToUser syslog
>> #$PrivDropToGroup adm
>>
>> #
>> # Where to place spool files
>> #
>> $WorkDirectory /var/spool/rsyslog
>>
>> #
>> # Include all config files in /etc/rsyslog.d/
>> #
>> $IncludeConfig /etc/rsyslog.d/*.conf
>>
>> $template
> syslog,"/var/log/external/%fromhost%/syslog-%$YEAR%%$MONTH%%$DAY%.log"
>> $template
> apacheError,"/var/log/external/%fromhost%/apache/%programname%-error-%$YEAR%%$MONTH%%$DAY%.log"
>> $template
> apacheAccess,"/var/log/external/%fromhost%/apache/%programname%-access-%$YEAR%%$MONTH%%$DAY%.log"
>> $template mailError,
> "/var/log/external/%fromhost%/mail/error-%$YEAR%%$MONTH%%$DAY%.log"
>> $template nodeStd,
> "/var/log/external/%fromhost%/node/TStd-%$YEAR%%$MONTH%%$DAY%.log"
>> $template nodeTService,
> "/var/log/external/%fromhost%/node/TLog-%$YEAR%%$MONTH%%$DAY%.log"
>>
>> local4.* ?nodeService
>> #& ~
>> & stop
>>
>> local5.* ?nodeStd
>> #& ~
>> & stop
>>
>> local7.* ?apacheError
>> #& ~
>> & stop
>>
>> local6.* ?apacheAccess
>> #& ~
>> & stop
>>
>> *.* ?syslog
>>
>> That hasn’t changed for about 11 years.
>> That rsyslog central server is also recently upgraded to Ubuntu20 from
> Ubuntu18
>>
>> FWIW Ive only talked about TLog - but TStd does the same thing (and has a
> similar imfile/local5 config on the client as top the imfile/local4)
>>
>>
>>
>>
>>
>> Confidentiality notice: This email (and any attachment) is intended for the
> addressee(s) named above. It may contain information of a confidential or
> legally privileged nature. Unauthorised disclosure or use of this email (or
> any attachment) is prohibited and may be unlawful. If you are not the
> intended recipient, please delete the email from your systems, destroy any
> copies and inform the sender immediately. Privacy notice: To find information
> on how we collect, process and store data, please see our privacy statement
> on our website https://www.celebrus.com/privacy-statement Disclaimer: All
> attachments have been scanned for viruses. However, Celebrus Technologies Plc
> cannot accept liability for any loss or damage you may incur as a result of
> virus infection.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 12:32 AM

Post #6 of 14 (60 views)

Permalink

>> Is there any chance that they are getting logged under a different hostname?

I've done an extensive search in the rsyslog server this morning, and the answer is sadly no.

Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 12:56 AM

Post #7 of 14 (60 views)

Permalink

On Mon, 22 Apr 2024, Ian Diddams wrote:

>>> Is there any chance that they are getting logged under a different hostname?
>
> I've done an extensive search in the rsyslog server this morning, and the answer is sadly no.

try logging the local4 facility to a different, fixed file (rather than a
dynamic filename)

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 1:42 AM

Post #8 of 14 (60 views)

Permalink

>> specifically look for 127.0.0.1 or localhost
see previous message . Nothing.

OK, Ive just set this up

local4.* /tmp/Tlocal.log

I created that log and chmod 777 for it to remove any silly potential issues

and restarted rsyslog on the central server

No change. Client logs have new entries. Server /var/log/Tlocal.log doesn’t at all.

So it's either the server not accepting what is sent on local4 - but "logger -p local4.info <string>" on the client DOES get centrally logged so that’s unlikely
OR
the client config isn’t capturing the additions to that local log for sending
i.e.

module(
load = "imfile"
pollingInterval = "1"
statefile.directory = "/var/log/node"
)

input(
type = "imfile"
tag = "tserv-stdout"
facility = "local4"
severity = "info"
file = "/var/log/node/Tlog.log"
)

cheers

ian
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David Lang via rsyslog
Sent: Friday, April 19, 2024 12:44 PM
To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>
Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

specifically look for 127.0.0.1 or localhost

If you can log anything that's local4 on the server to a single file (ideally using the template RSYSLOG_DebugFormat so we can see all the variables that are parsed from it) it may be easier to find the log than your current dynafile approach that puts them in different directories based on the hostname.

David Lang
Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 3:47 AM

Post #9 of 14 (60 views)

Permalink

Having experienced something similar a few years ago (imfile not seeing
new messages added to a certain log file), I'll throw this idea: in my
case, it was because the statefile associated with this log file got
corrupted or something. Granted, I was messing around with the log file
that imfile was watching.
Deleting the statefile and restarting rsyslog unlocked the situation. If
you do so, keep in mind that the whole log file will be processed as if
it was a new file.

On 2024-04-22 10:42, Ian Diddams via rsyslog wrote :
>>> specifically look for 127.0.0.1 or localhost
> see previous message . Nothing.
>
>
> OK, Ive just set this up
>
>
> local4.* /tmp/Tlocal.log
>
> I created that log and chmod 777 for it to remove any silly potential
> issues
>
> and restarted rsyslog on the central server
>
> No change. Client logs have new entries. Server /var/log/Tlocal.log
> doesn’t
> at all.
>
> So it's either the server not accepting what is sent on local4 - but
> "logger -p local4.info <string>" on the client DOES get centrally
> logged so
> that’s unlikely
> OR
> the client config isn’t capturing the additions to that local log for
> sending
> i.e.
>
> module(
> load = "imfile"
> pollingInterval = "1"
> statefile.directory = "/var/log/node"
> )
>
> input(
> type = "imfile"
> tag = "tserv-stdout"
> facility = "local4"
> severity = "info"
> file = "/var/log/node/Tlog.log"
> )
>
> cheers
>
> ian
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David
> Lang
> via rsyslog
> Sent: Friday, April 19, 2024 12:44 PM
> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic
> since
> upgrade to ubuntu20
>
> specifically look for 127.0.0.1 or localhost
>
> If you can log anything that's local4 on the server to a single file
> (ideally using the template RSYSLOG_DebugFormat so we can see all the
> variables that are parsed from it) it may be easier to find the log
> than
> your current dynafile approach that puts them in different directories
> based
> on the hostname.
>
> David Lang
> Confidentiality notice: This email (and any attachment) is intended for
> the
> addressee(s) named above. It may contain information of a confidential
> or
> legally privileged nature. Unauthorised disclosure or use of this email
> (or
> any attachment) is prohibited and may be unlawful. If you are not the
> intended recipient, please delete the email from your systems, destroy
> any
> copies and inform the sender immediately. Privacy notice: To find
> information on how we collect, process and store data, please see our
> privacy statement on our website
> https://www.celebrus.com/privacy-statement
> Disclaimer: All attachments have been scanned for viruses. However,
> Celebrus
> Technologies Plc cannot accept liability for any loss or damage you may
> incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 4:21 AM

Post #10 of 14 (60 views)

Permalink

depending on your distro SELinux or AppArmor may be preventing rsyslog from
writing to /tmp, stick it in /var/log to avoid that.

can you get a short debug dump during a time that you send such a log?

just to be sure, this local4 is not showing up on the client, but is showing up
on the central server??

David Lang

On Mon, 22 Apr 2024, Ian Diddams via rsyslog wrote:

> Date: Mon, 22 Apr 2024 08:42:00 +0000
> From: Ian Diddams via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Ian Diddams <ian.diddams@celebrus.com>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since
> upgrade to ubuntu20
>
>>> specifically look for 127.0.0.1 or localhost
> see previous message . Nothing.
>
>
> OK, Ive just set this up
>
>
> local4.* /tmp/Tlocal.log
>
> I created that log and chmod 777 for it to remove any silly potential issues
>
> and restarted rsyslog on the central server
>
> No change. Client logs have new entries. Server /var/log/Tlocal.log doesn’t at all.
>
> So it's either the server not accepting what is sent on local4 - but "logger -p local4.info <string>" on the client DOES get centrally logged so that’s unlikely
> OR
> the client config isn’t capturing the additions to that local log for sending
> i.e.
>
> module(
> load = "imfile"
> pollingInterval = "1"
> statefile.directory = "/var/log/node"
> )
>
> input(
> type = "imfile"
> tag = "tserv-stdout"
> facility = "local4"
> severity = "info"
> file = "/var/log/node/Tlog.log"
> )
>
> cheers
>
> ian
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David Lang via rsyslog
> Sent: Friday, April 19, 2024 12:44 PM
> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20
>
> specifically look for 127.0.0.1 or localhost
>
> If you can log anything that's local4 on the server to a single file (ideally using the template RSYSLOG_DebugFormat so we can see all the variables that are parsed from it) it may be easier to find the log than your current dynafile approach that puts them in different directories based on the hostname.
>
> David Lang
> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 2:22 PM

Post #11 of 14 (51 views)

Permalink

>Having experienced something similar a few years ago (imfile not seeing new messages added to a certain log file), I'll throw this idea: in my case, it was because >the statefile associated with this log file got corrupted or something. Granted, I was messing around with the log file that imfile was watching.
>Deleting the statefile and restarting rsyslog unlocked the situation. If you do so, keep in mind that the whole log file will be processed as if it was a new file.

For clarity, in my examples. Do you mean by statefile

>> statefile.directory = "/var/log/node"
or
>> file = "/var/log/node/Tlog.log"

Cheers

ian

On 2024-04-22 10:42, Ian Diddams via rsyslog wrote :
>>> specifically look for 127.0.0.1 or localhost
> see previous message . Nothing.
>
>
> OK, Ive just set this up
>
>
> local4.* /tmp/Tlocal.log
>
> I created that log and chmod 777 for it to remove any silly potential
> issues
>
> and restarted rsyslog on the central server
>
> No change. Client logs have new entries. Server /var/log/Tlocal.log
> doesn’t at all.
>
> So it's either the server not accepting what is sent on local4 - but
> "logger -p local4.info <string>" on the client DOES get centrally
> logged so that’s unlikely OR the client config isn’t capturing the
> additions to that local log for sending i.e.
>
> module(
> load = "imfile"
> pollingInterval = "1"
> statefile.directory = "/var/log/node"
> )
>
> input(
> type = "imfile"
> tag = "tserv-stdout"
> facility = "local4"
> severity = "info"
> file = "/var/log/node/Tlog.log"
> )
>
> cheers
>
> ian
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David
> Lang via rsyslog
> Sent: Friday, April 19, 2024 12:44 PM
> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic
> since upgrade to ubuntu20
>
> specifically look for 127.0.0.1 or localhost
>
> If you can log anything that's local4 on the server to a single file
> (ideally using the template RSYSLOG_DebugFormat so we can see all the
> variables that are parsed from it) it may be easier to find the log
> than your current dynafile approach that puts them in different
> directories based on the hostname.
>
> David Lang
> Confidentiality notice: This email (and any attachment) is intended
> for the
> addressee(s) named above. It may contain information of a confidential
> or legally privileged nature. Unauthorised disclosure or use of this
> email (or any attachment) is prohibited and may be unlawful. If you
> are not the intended recipient, please delete the email from your
> systems, destroy any copies and inform the sender immediately. Privacy
> notice: To find information on how we collect, process and store data,
> please see our privacy statement on our website
> https://www.celebrus.com/privacy-statement
> Disclaimer: All attachments have been scanned for viruses. However,
> Celebrus Technologies Plc cannot accept liability for any loss or
> damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 22, 2024, 3:01 PM

Post #12 of 14 (51 views)

Permalink

If a file is rotated without rsyslog’s knowledge the state file becomes out of date. The state file is just a high-water mark for the file and is not guaranteed to be updated in all cases.

State files are created in the statefile.directory and are convieniently named imfile-state* where the rest of the file name is a unique identifier that is specific to the input file.

If a log file is truncated, and the state file does not update, rsyslog won’t process new file content until the file reaches the previous high water size. You can see this in the state file itself as they are simply JSON objects containing offset values.

Our solution (for these file based instances) was to stop rsyslog, then rotate/delete the log file, then delete the state file, then restart rsyslog.

Regards,

> On Apr 22, 2024, at 16:22, Ian Diddams via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
>> Having experienced something similar a few years ago (imfile not seeing new messages added to a certain log file), I'll throw this idea: in my case, it was because >the statefile associated with this log file got corrupted or something. Granted, I was messing around with the log file that imfile was watching.
>> Deleting the statefile and restarting rsyslog unlocked the situation. If you do so, keep in mind that the whole log file will be processed as if it was a new file.
>
> For clarity, in my examples. Do you mean by statefile
>
>>> statefile.directory = "/var/log/node"
> or
>>> file = "/var/log/node/Tlog.log"
>
> Cheers
>
> ian
>
> On 2024-04-22 10:42, Ian Diddams via rsyslog wrote :
>>>> specifically look for 127.0.0.1 or localhost
>> see previous message . Nothing.
>>
>>
>> OK, Ive just set this up
>>
>>
>> local4.* /tmp/Tlocal.log
>>
>> I created that log and chmod 777 for it to remove any silly potential
>> issues
>>
>> and restarted rsyslog on the central server
>>
>> No change. Client logs have new entries. Server /var/log/Tlocal.log
>> doesn’t at all.
>>
>> So it's either the server not accepting what is sent on local4 - but
>> "logger -p local4.info <string>" on the client DOES get centrally
>> logged so that’s unlikely OR the client config isn’t capturing the
>> additions to that local log for sending i.e.
>>
>> module(
>> load = "imfile"
>> pollingInterval = "1"
>> statefile.directory = "/var/log/node"
>> )
>>
>> input(
>> type = "imfile"
>> tag = "tserv-stdout"
>> facility = "local4"
>> severity = "info"
>> file = "/var/log/node/Tlog.log"
>> )
>>
>> cheers
>>
>> ian
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David
>> Lang via rsyslog
>> Sent: Friday, April 19, 2024 12:44 PM
>> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: David Lang <david@lang.hm>
>> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic
>> since upgrade to ubuntu20
>>
>> specifically look for 127.0.0.1 or localhost
>>
>> If you can log anything that's local4 on the server to a single file
>> (ideally using the template RSYSLOG_DebugFormat so we can see all the
>> variables that are parsed from it) it may be easier to find the log
>> than your current dynafile approach that puts them in different
>> directories based on the hostname.
>>
>> David Lang
>> Confidentiality notice: This email (and any attachment) is intended
>> for the
>> addressee(s) named above. It may contain information of a confidential
>> or legally privileged nature. Unauthorised disclosure or use of this
>> email (or any attachment) is prohibited and may be unlawful. If you
>> are not the intended recipient, please delete the email from your
>> systems, destroy any copies and inform the sender immediately. Privacy
>> notice: To find information on how we collect, process and store data,
>> please see our privacy statement on our website
>> https://www.celebrus.com/privacy-statement
>> Disclaimer: All attachments have been scanned for viruses. However,
>> Celebrus Technologies Plc cannot accept liability for any loss or
>> damage you may incur as a result of virus infection.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 23, 2024, 7:10 AM

Post #13 of 14 (50 views)

Permalink

John - think that’s the badger!!

the logrotate.d file has been in use for about a decade AIUI but only now has it failed to work properly - which either suggests rsyslog wasn’t really that choosy before ubuntu20/latest rsyslog version but now is... or "we" got lucky somehow.

Anyway ive updated the rotate config on the client to

postrotate
stop rsyslog
remove imfile
blat the logs
recreate the logs
restart rsyslog

and it seems to be doing what it is supposed to do now.

Ill keep a beady eye on it over the next few days.

Cheers!
-----Original Message-----
From: John Chivian <jchivian@chivian.com>
Sent: Monday, April 22, 2024 11:02 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Ian Diddams <ian.diddams@celebrus.com>
Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

If a file is rotated without rsyslog’s knowledge the state file becomes out of date. The state file is just a high-water mark for the file and is not guaranteed to be updated in all cases.

State files are created in the statefile.directory and are convieniently named imfile-state* where the rest of the file name is a unique identifier that is specific to the input file.

If a log file is truncated, and the state file does not update, rsyslog won’t process new file content until the file reaches the previous high water size. You can see this in the state file itself as they are simply JSON objects containing offset values.

Our solution (for these file based instances) was to stop rsyslog, then rotate/delete the log file, then delete the state file, then restart rsyslog.

Regards,

> On Apr 22, 2024, at 16:22, Ian Diddams via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
>> Having experienced something similar a few years ago (imfile not seeing new messages added to a certain log file), I'll throw this idea: in my case, it was because >the statefile associated with this log file got corrupted or something. Granted, I was messing around with the log file that imfile was watching.
>> Deleting the statefile and restarting rsyslog unlocked the situation. If you do so, keep in mind that the whole log file will be processed as if it was a new file.
>
> For clarity, in my examples. Do you mean by statefile
>
>>> statefile.directory = "/var/log/node"
> or
>>> file = "/var/log/node/Tlog.log"
>
> Cheers
>
> ian
>
> On 2024-04-22 10:42, Ian Diddams via rsyslog wrote :
>>>> specifically look for 127.0.0.1 or localhost
>> see previous message . Nothing.
>>
>>
>> OK, Ive just set this up
>>
>>
>> local4.* /tmp/Tlocal.log
>>
>> I created that log and chmod 777 for it to remove any silly potential
>> issues
>>
>> and restarted rsyslog on the central server
>>
>> No change. Client logs have new entries. Server /var/log/Tlocal.log
>> doesn’t at all.
>>
>> So it's either the server not accepting what is sent on local4 - but
>> "logger -p local4.info <string>" on the client DOES get centrally
>> logged so that’s unlikely OR the client config isn’t capturing the
>> additions to that local log for sending i.e.
>>
>> module(
>> load = "imfile"
>> pollingInterval = "1"
>> statefile.directory = "/var/log/node"
>> )
>>
>> input(
>> type = "imfile"
>> tag = "tserv-stdout"
>> facility = "local4"
>> severity = "info"
>> file = "/var/log/node/Tlog.log"
>> )
>>
>> cheers
>>
>> ian
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David
>> Lang via rsyslog
>> Sent: Friday, April 19, 2024 12:44 PM
>> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: David Lang <david@lang.hm>
>> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic
>> since upgrade to ubuntu20
>>
>> specifically look for 127.0.0.1 or localhost
>>
>> If you can log anything that's local4 on the server to a single file
>> (ideally using the template RSYSLOG_DebugFormat so we can see all the
>> variables that are parsed from it) it may be easier to find the log
>> than your current dynafile approach that puts them in different
>> directories based on the hostname.
>>
>> David Lang
>> Confidentiality notice: This email (and any attachment) is intended
>> for the
>> addressee(s) named above. It may contain information of a
>> confidential or legally privileged nature. Unauthorised disclosure or
>> use of this email (or any attachment) is prohibited and may be
>> unlawful. If you are not the intended recipient, please delete the
>> email from your systems, destroy any copies and inform the sender
>> immediately. Privacy
>> notice: To find information on how we collect, process and store
>> data, please see our privacy statement on our website
>> https://www.celebrus.com/privacy-statement
>> Disclaimer: All attachments have been scanned for viruses. However,
>> Celebrus Technologies Plc cannot accept liability for any loss or
>> damage you may incur as a result of virus infection.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20 [ In reply to ]

rsyslog at lists

Apr 23, 2024, 7:38 AM

Post #14 of 14 (50 views)

Permalink

I'm pretty sure this is one of the areas affected by the improvements since 2020
in imfile.

David Lang

On Tue, 23 Apr 2024, Ian Diddams via rsyslog wrote:

> Date: Tue, 23 Apr 2024 14:10:32 +0000
> From: Ian Diddams via rsyslog <rsyslog@lists.adiscon.com>
> To: John Chivian <jchivian@chivian.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Ian Diddams <ian.diddams@celebrus.com>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since
> upgrade to ubuntu20
>
> John - think that’s the badger!!
>
> the logrotate.d file has been in use for about a decade AIUI but only now has it failed to work properly - which either suggests rsyslog wasn’t really that choosy before ubuntu20/latest rsyslog version but now is... or "we" got lucky somehow.
>
> Anyway ive updated the rotate config on the client to
>
> postrotate
> stop rsyslog
> remove imfile
> blat the logs
> recreate the logs
> restart rsyslog
>
>
> and it seems to be doing what it is supposed to do now.
>
> Ill keep a beady eye on it over the next few days.
>
> Cheers!
> -----Original Message-----
> From: John Chivian <jchivian@chivian.com>
> Sent: Monday, April 22, 2024 11:02 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Ian Diddams <ian.diddams@celebrus.com>
> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20
>
> If a file is rotated without rsyslog’s knowledge the state file becomes out of date. The state file is just a high-water mark for the file and is not guaranteed to be updated in all cases.
>
> State files are created in the statefile.directory and are convieniently named imfile-state* where the rest of the file name is a unique identifier that is specific to the input file.
>
> If a log file is truncated, and the state file does not update, rsyslog won’t process new file content until the file reaches the previous high water size. You can see this in the state file itself as they are simply JSON objects containing offset values.
>
> Our solution (for these file based instances) was to stop rsyslog, then rotate/delete the log file, then delete the state file, then restart rsyslog.
>
> Regards,
>
>
>> On Apr 22, 2024, at 16:22, Ian Diddams via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>>> Having experienced something similar a few years ago (imfile not seeing new messages added to a certain log file), I'll throw this idea: in my case, it was because >the statefile associated with this log file got corrupted or something. Granted, I was messing around with the log file that imfile was watching.
>>> Deleting the statefile and restarting rsyslog unlocked the situation. If you do so, keep in mind that the whole log file will be processed as if it was a new file.
>>
>> For clarity, in my examples. Do you mean by statefile
>>
>>>> statefile.directory = "/var/log/node"
>> or
>>>> file = "/var/log/node/Tlog.log"
>>
>> Cheers
>>
>> ian
>>
>> On 2024-04-22 10:42, Ian Diddams via rsyslog wrote :
>>>>> specifically look for 127.0.0.1 or localhost
>>> see previous message . Nothing.
>>>
>>>
>>> OK, Ive just set this up
>>>
>>>
>>> local4.* /tmp/Tlocal.log
>>>
>>> I created that log and chmod 777 for it to remove any silly potential
>>> issues
>>>
>>> and restarted rsyslog on the central server
>>>
>>> No change. Client logs have new entries. Server /var/log/Tlocal.log
>>> doesn’t at all.
>>>
>>> So it's either the server not accepting what is sent on local4 - but
>>> "logger -p local4.info <string>" on the client DOES get centrally
>>> logged so that’s unlikely OR the client config isn’t capturing the
>>> additions to that local log for sending i.e.
>>>
>>> module(
>>> load = "imfile"
>>> pollingInterval = "1"
>>> statefile.directory = "/var/log/node"
>>> )
>>>
>>> input(
>>> type = "imfile"
>>> tag = "tserv-stdout"
>>> facility = "local4"
>>> severity = "info"
>>> file = "/var/log/node/Tlog.log"
>>> )
>>>
>>> cheers
>>>
>>> ian
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David
>>> Lang via rsyslog
>>> Sent: Friday, April 19, 2024 12:44 PM
>>> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>
>>> Cc: David Lang <david@lang.hm>
>>> Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic
>>> since upgrade to ubuntu20
>>>
>>> specifically look for 127.0.0.1 or localhost
>>>
>>> If you can log anything that's local4 on the server to a single file
>>> (ideally using the template RSYSLOG_DebugFormat so we can see all the
>>> variables that are parsed from it) it may be easier to find the log
>>> than your current dynafile approach that puts them in different
>>> directories based on the hostname.
>>>
>>> David Lang
>>> Confidentiality notice: This email (and any attachment) is intended
>>> for the
>>> addressee(s) named above. It may contain information of a
>>> confidential or legally privileged nature. Unauthorised disclosure or
>>> use of this email (or any attachment) is prohibited and may be
>>> unlawful. If you are not the intended recipient, please delete the
>>> email from your systems, destroy any copies and inform the sender
>>> immediately. Privacy
>>> notice: To find information on how we collect, process and store
>>> data, please see our privacy statement on our website
>>> https://www.celebrus.com/privacy-statement
>>> Disclaimer: All attachments have been scanned for viruses. However,
>>> Celebrus Technologies Plc cannot accept liability for any loss or
>>> damage you may incur as a result of virus infection.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>> you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> Confidentiality notice: This email (and any attachment) is intended for the addressee(s) named above. It may contain information of a confidential or legally privileged nature. Unauthorised disclosure or use of this email (or any attachment) is prohibited and may be unlawful. If you are not the intended recipient, please delete the email from your systems, destroy any copies and inform the sender immediately. Privacy notice: To find information on how we collect, process and store data, please see our privacy statement on our website https://www.celebrus.com/privacy-statement Disclaimer: All attachments have been scanned for viruses. However, Celebrus Technologies Plc cannot accept liability for any loss or damage you may incur as a result of virus infection.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Mailing List Archive

Mailing List Archive

Attached Files: