Mailing List Archive

rsyslogd rotate / killall -HUP rsyslogd
Hi,

is there a rule when to issue a rotate / reload command? I'm referring
to "postrotate" in logrotate.

https://docs.rackspace.com/support/how-to/understanding-logrotate-utility/

Thank you!

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
It's not exactly clear what you mean :-)

In case of rsyslog, the killall -HUP (be wary about the killall command
- getting used to it can have disastrous effects if you're on a wrong
system - https://en.wikipedia.org/wiki/Killall ) makes the rsyslogd
process reopen the log files (and re-read _some_ configuration files;
but only very selected subset - in order to apply new configuration you
generally need to restart the rsyslogd completely).

Which means that typically

1. rsyslogd works as usual

2. logrotate wakes up and rotates the files - the log files change their
names (for example - /var/log/messages becomes /var/log/messages.1). But
files are still open - rsyslogd can still write to the descriptors it
opened long ago. The data lands in the same files, it's just that they
are named differently now. So in our example - rsyslog is writing to
/var/log/messages.1.

3. logrotate sends SIGHUP to rsyslogd as part of postrotate action

4. rsyslogd receives SIGHUP, closes its file descriptors and opens new
ones with the filenames specified in configuration (so closes
/var/log/messages.1 and creates /var/log/messages again).

5. optionally - logrotate compresses rotated files

That's how you normally do it.

On 19.01.2022 11:11, Helmut Schneider via rsyslog wrote:
> Hi,
>
> is there a rule when to issue a rotate / reload command? I'm referring
> to "postrotate" in logrotate.
>
> https://docs.rackspace.com/support/how-to/understanding-logrotate-utility/
>
>
> Thank you!
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
Am 19.01.2022 um 12:11 schrieb Mariusz Kruk via rsyslog:
> It's not exactly clear what you mean :-)
[...]
> 3. logrotate sends SIGHUP to rsyslogd as part of postrotate action

If logrotate rotes a log, do I always have to reload rsyslod or only
under some circumstances?

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
On 19.01.2022 13:10, Helmut Schneider via rsyslog wrote:
> Am 19.01.2022 um 12:11 schrieb Mariusz Kruk via rsyslog:
>> It's not exactly clear what you mean :-)
> [...]
>> 3. logrotate sends SIGHUP to rsyslogd as part of postrotate action
>
> If logrotate rotes a log, do I always have to reload rsyslod or only
> under some circumstances?


Well,  you never "have to" ;-)

But seriously. As I wrote before - when you rotate the file, logrotate
does a simple syscall changing the link in the directory index but the
file itself (inode, contents) stays the same. And if it's open at time
of rotating the descriptors stay open and the processes that have the
file open still work on the file they had opened at the moment of
rotation. But if any process wants to open the file by the old name - it
won't find it because file _under that name_ is no longer present in the
directory. So if "old" process doesn't close/re-open the file, it will
still be writing to the old file under a new name when newly created
processes (or old processes which open the file later) will refer to a
new file.

Let's imagine a sequence of events:

1) rsyslogd starts, opens /var/log/messages for writing

2) after some time, logrotate renames the file to /var/log/messages.1

3) since no SIGHUP has been sent to rsyslogd process, it keeps the file
open but the file is named /var/log/messages.1

4) another process (for example, a user using cat/more/less in his shell
session) opens /var/log/messages

5) there is no such file as /var/log/messages

In extreme case without closing/re-opening files after enough time the
original file could get "rotated out" : /var/log/messages ->
/var/log/messages.1 -> /var/log/messages.2 -> ... up to a point when it
would reach the limit of your rotated files backlog. In this case the
file would get unlinked by the logrotate process but it would still stay
opened in the rsyslogd process! So the disk space would still be used,
rsyslogd process would still be writing to it consuming said space but
you'd have no way to open such file anymore (short of some clever tricks
with file descriptors) and check its contents because there would be no
directory entry for it anymore. Only after closing the last descriptor
pointing to this file the underlying operating system would delete the
file contents and free the inode.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
Am 19.01.2022 um 13:21 schrieb Mariusz Kruk via rsyslog:
> On 19.01.2022 13:10, Helmut Schneider via rsyslog wrote:
>> Am 19.01.2022 um 12:11 schrieb Mariusz Kruk via rsyslog:
>>> It's not exactly clear what you mean :-)
>> [...]
>>> 3. logrotate sends SIGHUP to rsyslogd as part of postrotate action
>>
>> If logrotate rotes a log, do I always have to reload rsyslod or only
>> under some circumstances?
>
> Well,  you never "have to" ;-)
>
> But seriously. As I wrote before - when you rotate the file, logrotate
> does a simple syscall changing the link in the directory index but the
> file itself (inode, contents) stays the same. And if it's open at time
> of rotating the descriptors stay open and the processes that have the
> file open still work on the file they had opened at the moment of
> rotation. But if any process wants to open the file by the old name - it
> won't find it because file _under that name_ is no longer present in the
> directory. So if "old" process doesn't close/re-open the file, it will
> still be writing to the old file under a new name when newly created
> processes (or old processes which open the file later) will refer to a
> new file.

Sounds to me like "killall -HUP rsyslogd" is recommended on each run of
logrotate, correct? At least it doesn't hurt. :)

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
On 19.01.2022 17:43, Helmut Schneider via rsyslog wrote:
> Am 19.01.2022 um 13:21 schrieb Mariusz Kruk via rsyslog:
>> On 19.01.2022 13:10, Helmut Schneider via rsyslog wrote:
>>> Am 19.01.2022 um 12:11 schrieb Mariusz Kruk via rsyslog:
>>>> It's not exactly clear what you mean :-)
>>> [...]
>>>> 3. logrotate sends SIGHUP to rsyslogd as part of postrotate action
>>>
>>> If logrotate rotes a log, do I always have to reload rsyslod or only
>>> under some circumstances?
>>
>> Well,  you never "have to" ;-)
>>
>> But seriously. As I wrote before - when you rotate the file, logrotate
>> does a simple syscall changing the link in the directory index but the
>> file itself (inode, contents) stays the same. And if it's open at time
>> of rotating the descriptors stay open and the processes that have the
>> file open still work on the file they had opened at the moment of
>> rotation. But if any process wants to open the file by the old name - it
>> won't find it because file _under that name_ is no longer present in the
>> directory. So if "old" process doesn't close/re-open the file, it will
>> still be writing to the old file under a new name when newly created
>> processes (or old processes which open the file later) will refer to a
>> new file.
>
> Sounds to me like "killall -HUP rsyslogd" is recommended on each run
> of logrotate, correct? At least it doesn't hurt. :)
>
It's actually necessary for quite a significant subset of daemons. It's
quite typical for well-written daemons to close/reopen their log files
in response to SIGHUP. And you do it in postrotate action in logrotate
quite often.

The more problematic ones are those that don't do it (like tomcat which
- at least some time ago when I worked with it - could "rotate" files
meaning it would create new files on its own in some conditions but it
could not limit the number of files and delete oldest ones) and need
full restart to actually close/open log files (and such restart of
course can be time consuming and introduces significant downtime to the
service).

But that's completely different story (not related to rsyslog itself).

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslogd rotate / killall -HUP rsyslogd [ In reply to ]
Am 20.01.2022 um 13:23 schrieb Mariusz Kruk via rsyslog:
>
> On 19.01.2022 17:43, Helmut Schneider via rsyslog wrote:
>> Am 19.01.2022 um 13:21 schrieb Mariusz Kruk via rsyslog:
>>> On 19.01.2022 13:10, Helmut Schneider via rsyslog wrote:
>>>> Am 19.01.2022 um 12:11 schrieb Mariusz Kruk via rsyslog:
>>>>> It's not exactly clear what you mean :-)
>>>> [...]
>>>>> 3. logrotate sends SIGHUP to rsyslogd as part of postrotate action
>>>>
>>>> If logrotate rotes a log, do I always have to reload rsyslod or only
>>>> under some circumstances?
>>>
>>> Well,  you never "have to" ;-)
>>>
>>> But seriously. As I wrote before - when you rotate the file, logrotate
>>> does a simple syscall changing the link in the directory index but the
>>> file itself (inode, contents) stays the same. And if it's open at time
>>> of rotating the descriptors stay open and the processes that have the
>>> file open still work on the file they had opened at the moment of
>>> rotation. But if any process wants to open the file by the old name - it
>>> won't find it because file _under that name_ is no longer present in the
>>> directory. So if "old" process doesn't close/re-open the file, it will
>>> still be writing to the old file under a new name when newly created
>>> processes (or old processes which open the file later) will refer to a
>>> new file.
>>
>> Sounds to me like "killall -HUP rsyslogd" is recommended on each run
>> of logrotate, correct? At least it doesn't hurt. :)
>>
> It's actually necessary for quite a significant subset of daemons. It's
> quite typical for well-written daemons to close/reopen their log files
> in response to SIGHUP. And you do it in postrotate action in logrotate
> quite often.
>
> The more problematic ones are those that don't do it (like tomcat which
> - at least some time ago when I worked with it - could "rotate" files
> meaning it would create new files on its own in some conditions but it
> could not limit the number of files and delete oldest ones) and need
> full restart to actually close/open log files (and such restart of
> course can be time consuming and introduces significant downtime to the
> service).
>
> But that's completely different story (not related to rsyslog itself).

Thank you!

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.