Mailing List Archive

On Mon, 6 Oct 2008, Rainer Gerhards wrote:

RG>
RG> [...]
RG>
RG> Initially, I thought to use a special signal (e.g. USR2) for this
RG> purpose. But still logrotate, as a well-known example, would cause a
RG> daemon restart where none is needed. So I now think it would be best to
RG> change HUP in the way described above. Those that need to apply a config
RG> change would then need to do a regular daemon restart - the same thing
RG> that is done with other daemons. In order to keep compatible, I would
RG> add an $HUPisRestart option, by default off, which could be used to
RG> re-enable previous behavior.
RG>

Still, I'd like to have a way to force full restart without really tearing
down and reloading the server. What about (optionally?) binding current
HUP behaviour to USR2 adding a default on option like $USR2isRestart?

Cheers, and a nice forthweek to all, yours

lorenzo


+-------------------------+----------------------------------------------+
| Lorenzo M. Catucci | Centro di Calcolo e Documentazione |
| catucci@ccd.uniroma2.it | Università degli Studi di Roma "Tor Vergata" |
| | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY |
| Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 |
+-------------------------+----------------------------------------------+

Thanks for the feedback, but please let me ask one question:

On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote:

> Still, I'd like to have a way to force full restart without really tearing
> down and reloading the server. What about (optionally?) binding current
> HUP behaviour to USR2 adding a default on option like $USR2isRestart?

Why is this useful? I am trying to understand the use cases and why a
"normal" restart is not desirable.

Rainer

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 6 Oct 2008, Rainer Gerhards wrote:

RG> Thanks for the feedback, but please let me ask one question:
RG>
RG> On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote:
RG>
RG> > Still, I'd like to have a way to force full restart without really tearing
RG> > down and reloading the server. What about (optionally?) binding current
RG> > HUP behaviour to USR2 adding a default on option like $USR2isRestart?
RG>
RG> Why is this useful? I am trying to understand the use cases and why a
RG> "normal" restart is not desirable.
RG>

Habits, pid stability, monitoring systems, one less thing to break... etc.

Yours,

lorenzo

+-------------------------+----------------------------------------------+
| Lorenzo M. Catucci | Centro di Calcolo e Documentazione |
| catucci@ccd.uniroma2.it | Università degli Studi di Roma "Tor Vergata" |
| | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY |
| Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 |
+-------------------------+----------------------------------------------+

Lorenzo,

> Habits, pid stability, monitoring systems, one less thing to break...

I understand habits and pid stability. What do you mean by "monitoring
system" and "one less thing to break"?

Anyone else with an opinion?

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

In general, I understand HUP to be exactly what you're talking - a
reload rather than a restart. I see little to lose by enforcing that
in rsyslogd.

The pid stability argument has no relevance for me or the tools I use
since I just pull it out of pidfiles anyway.

-HKS



On Mon, Oct 6, 2008 at 7:56 AM, Rainer Gerhards
<rgerhards@hq.adiscon.com> wrote:
> Lorenzo,
>
>> Habits, pid stability, monitoring systems, one less thing to break...
>
> I understand habits and pid stability. What do you mean by "monitoring
> system" and "one less thing to break"?
>
> Anyone else with an opinion?
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 6 Oct 2008, Rainer Gerhards wrote:

> Hi List,
>
> I again have a backwards compatibility question.
>
> HUP, in sysklogd, means a full restart of the syslogd. This is done
> because the configuration may have been changed and to apply the new
> one, the previous one must be shut down. This is even more so the case
> in rsyslog, which has multiple dynamically loadable plugins and all
> that...
>
> rsyslog has implemented HUP processing to be compatible to sysklogd.
>
> However, in practice (as far as I know) HUP is almost always being used
> to close the output files after log rotation. An automatted config
> reaload indication usually does not happen (but operators use HUP after
> doing a config change).
>
> HUP, as it currently is, is an extremely expensive operation. Keep in
> mind that it is a full daemon restart, so, among other things, it
> unloads modules and tears down (TCP, RELP, TLS, ...) connections. In
> most cases, though, this would not be needed.
>
> I now consider changing the way HUP works. I would like to model it in a
> way that closes files and clears caches (where useful), but not do a
> full restart.
>
> Initially, I thought to use a special signal (e.g. USR2) for this
> purpose. But still logrotate, as a well-known example, would cause a
> daemon restart where none is needed. So I now think it would be best to
> change HUP in the way described above. Those that need to apply a config
> change would then need to do a regular daemon restart - the same thing
> that is done with other daemons. In order to keep compatible, I would
> add an $HUPisRestart option, by default off, which could be used to
> re-enable previous behavior.
>
> I would appreciate feedback on this plan. Are there any concerns? Is
> this a good idea or a bad one? Does it break something I have not yet
> mentioned?

could you do something along the lines of checking to see if the config
file changed, and if so do a full restart, otherwise just the flush/close?

if you know when you started you may be able to just check the
last-modified time of the config file and do a restart if it's been
modified after you started.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

> could you do something along the lines of checking to see if the
config
> file changed, and if so do a full restart, otherwise just the
> flush/close?
>
> if you know when you started you may be able to just check the
> last-modified time of the config file and do a restart if it's been
> modified after you started.

That's a very interesting idea. Do you think this criterion is
sufficient?

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 6 Oct 2008, Rainer Gerhards wrote:

>> could you do something along the lines of checking to see if the
> config
>> file changed, and if so do a full restart, otherwise just the
>> flush/close?
>>
>> if you know when you started you may be able to just check the
>> last-modified time of the config file and do a restart if it's been
>> modified after you started.
>
> That's a very interesting idea. Do you think this criterion is
> sufficient?

I was thinking of other possible items (doing a checksum of the file,
doing a syntactic check to see if the new is equivalent to the old, etc)
and I think that they don't end up adding noticable value

the only way the last-modified check would fail is if your clock goes
backwards (and goes backwards far enough that the 'new' file is modified
before you last started). two ways to deal with that

1. store the last-modified time of the file when you start up and re-read
it if it has changed (still vunerable if the clock moved back to that
exact second, but a very small windows)

2. document the issue and tell people that if their clock goes backwards
and the HUP doesn't re-read the config file do a 'touch' of the file and
try again.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

This sounds very reasonable. Thanks for the good suggestion!

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of david@lang.hm
> Sent: Monday, October 06, 2008 5:16 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] HUP action
>
> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
>
> >> could you do something along the lines of checking to see if the
> > config
> >> file changed, and if so do a full restart, otherwise just the
> >> flush/close?
> >>
> >> if you know when you started you may be able to just check the
> >> last-modified time of the config file and do a restart if it's been
> >> modified after you started.
> >
> > That's a very interesting idea. Do you think this criterion is
> > sufficient?
>
> I was thinking of other possible items (doing a checksum of the file,
> doing a syntactic check to see if the new is equivalent to the old,
> etc)
> and I think that they don't end up adding noticable value
>
> the only way the last-modified check would fail is if your clock goes
> backwards (and goes backwards far enough that the 'new' file is
> modified
> before you last started). two ways to deal with that
>
> 1. store the last-modified time of the file when you start up and re-
> read
> it if it has changed (still vunerable if the clock moved back to that
> exact second, but a very small windows)
>
> 2. document the issue and tell people that if their clock goes
> backwards
> and the HUP doesn't re-read the config file do a 'touch' of the file
> and
> try again.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

2008/10/6 <david@lang.hm>:
> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
>
>>> could you do something along the lines of checking to see if the
>> config
>>> file changed, and if so do a full restart, otherwise just the
>>> flush/close?
>>>
>>> if you know when you started you may be able to just check the
>>> last-modified time of the config file and do a restart if it's been
>>> modified after you started.
>>
>> That's a very interesting idea. Do you think this criterion is
>> sufficient?

Don't forget $IncludeConfig

Michael


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Args... thanks Michael, this makes life a little less easy ;) So we do
not longer have a clear indication.

Would it make sense to require that the main config file is touched when
something in the includes is changed? This could be documented. And a
restart of course pulls everything.

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, October 06, 2008 5:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] HUP action
>
> 2008/10/6 <david@lang.hm>:
> > On Mon, 6 Oct 2008, Rainer Gerhards wrote:
> >
> >>> could you do something along the lines of checking to see if the
> >> config
> >>> file changed, and if so do a full restart, otherwise just the
> >>> flush/close?
> >>>
> >>> if you know when you started you may be able to just check the
> >>> last-modified time of the config file and do a restart if it's
been
> >>> modified after you started.
> >>
> >> That's a very interesting idea. Do you think this criterion is
> >> sufficient?
>
> Don't forget $IncludeConfig
>
> Michael
>
>
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, Oct 6, 2008 at 08:49, (private) HKS <hks.private@gmail.com> wrote:
> In general, I understand HUP to be exactly what you're talking - a
> reload rather than a restart. I see little to lose by enforcing that
> in rsyslogd.

Ditto - IMHO SIGHUP has come to signify 'change what you can without
restarting' or a "hot restart". Some daemons are capable of fully
re-configuring on the fly, i.e. "warm restart". It would be rather
complex but one could mark particular configuration pragmas as
requiring a full tear-down and issue a warning if a warm restart was
issued and those had changed. The third option (which many
poorly-written init scripts do anyway) is a "cold restart", where the
process is killed off and manually re-started, allowing full-blown
reconfiguration.

I don't care what signals are used for what, beyond the [de facto?]
standard of 'skip a beat' when given SIGHUP.

Beyond the scope of this discussion, I've not looked at your
configuration parsing but wonder if you're not parsing it into an
internal, static structure and handling bits from there. If you were,
it would be trivial to parse your new configuration, see how/whether
it differs from the running one, and shift over if able - like
failsafe firmware in embedded systems. That would also offer
interesting implications for alternative configuration syntaxes.

RB
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 2008-10-06 at 10:29 -0600, RB wrote:
> Beyond the scope of this discussion, I've not looked at your
> configuration parsing but wonder if you're not parsing it into an
> internal, static structure and handling bits from there. If you were,
> it would be trivial to parse your new configuration, see how/whether
> it differs from the running one, and shift over if able - like
> failsafe firmware in embedded systems. That would also offer
> interesting implications for alternative configuration syntaxes.

That is quite complex, because the configuration defines what can be
configured. With the plug-in architecture, we do not have any fixed
memory structures. So, we would need to spawn a new process, load
plugins there, check what in-memory structures they create and, if they
differ, than use this new configuration instead of the old one.

Rainer

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

2008/10/6 Rainer Gerhards <rgerhards@hq.adiscon.com>:
> Args... thanks Michael, this makes life a little less easy ;) So we do
> not longer have a clear indication.
>
> Would it make sense to require that the main config file is touched when
> something in the includes is changed? This could be documented. And a
> restart of course pulls everything.

I don't think that would be such a good idea.

It is not very intuitive and people are not used to such a behaviour.
They (at least I do) expect "reload" to reload the complete configuration.


Michael

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

So let me try to sum up where we are now:

- it is considered useful to have a full daemon restart be done via HUP
- we can not sufficiently simply detect whether the configuration
has changed or not
- there are situations where it is useful to have the ability to
just close files, clear caches etc
- people do not like existing things be used in new ways
(least surprise principle)

So I conclude:

- SIGHUP, as ugly as it is, must stay with existing semantics
- a new signal can be defined to just do file closure etc

Unfortunately, this means that most systems will still use the terribly
expensive during e.g. logrotation. However, this is considered
acceptable because a) it always was this way, b) a higher demand
environment then has options to avoid that. Over time, package
maintainers my get maintainers of logrotate involved to change the HUP
to the new signal.

Am I grasping this right?

Rainer

On Mon, 2008-10-06 at 18:37 +0200, Michael Biebl wrote:
> 2008/10/6 Rainer Gerhards <rgerhards@hq.adiscon.com>:
> > Args... thanks Michael, this makes life a little less easy ;) So we do
> > not longer have a clear indication.
> >
> > Would it make sense to require that the main config file is touched when
> > something in the includes is changed? This could be documented. And a
> > restart of course pulls everything.
>
> I don't think that would be such a good idea.
>
> It is not very intuitive and people are not used to such a behaviour.
> They (at least I do) expect "reload" to reload the complete configuration.
>
>
> Michael
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Rainer Gerhards wrote:
> - it is considered useful to have a full daemon restart be done via HUP
> - we can not sufficiently simply detect whether the configuration
> has changed or not

no knowing the code, i suggest md5/sha1/... checksum the whole file
after integrating everything including rsyslog.d/* into one flat
file.

> - there are situations where it is useful to have the ability to
> just close files, clear caches etc
> - people do not like existing things be used in new ways
> (least surprise principle)

i, currently, do not care that much about this issue - at least
regarding the reload vs. restart issue.

cheers,
raoul
--
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc. email. r.bhatia@ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at
Barawitzkagasse 10/2/2/11 email. office@ipax.at
1190 Wien tel. +43 1 3670030
FN 277995t HG Wien fax. +43 1 3670030 15
____________________________________________________________________
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX]
> Sent: Wednesday, October 08, 2008 12:49 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] HUP action
>
> Rainer Gerhards wrote:
> > - it is considered useful to have a full daemon restart be done via
> HUP
> > - we can not sufficiently simply detect whether the configuration
> > has changed or not
>
> no knowing the code, i suggest md5/sha1/... checksum the whole file
> after integrating everything including rsyslog.d/* into one flat
> file.

Quite honestly, I do not like the extra effort. I now have to define two
signals in any case, so let the user use the right one for what he
intends to do ;)

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

> > -----Original Message-----
> > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > bounces@lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX]
> > Sent: Wednesday, October 08, 2008 12:49 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] HUP action
> >
> > Rainer Gerhards wrote:
> > > - it is considered useful to have a full daemon restart be done
via
> > HUP
> > > - we can not sufficiently simply detect whether the configuration
> > > has changed or not
> >
> > no knowing the code, i suggest md5/sha1/... checksum the whole file
> > after integrating everything including rsyslog.d/* into one flat
> > file.
>
> Quite honestly, I do not like the extra effort. I now have to define
> two
> signals in any case, so let the user use the right one for what he
> intends to do ;)

Sorry, I was a bit brief: doing the checksum requires interpretation of
the config files, at least as far as the $IncludeConfig statements are
concerned. This is costly, as is the whole gathering process to build
the checksum. So to be on the efficient side, I need to have a dedicated
HUP for "non-reload" in any case. Thus I think there is not much added
benefit in doing the checksumming.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 6 Oct 2008, Rainer Gerhards wrote:

> So let me try to sum up where we are now:
>
> - it is considered useful to have a full daemon restart be done via HUP
> - we can not sufficiently simply detect whether the configuration
> has changed or not
> - there are situations where it is useful to have the ability to
> just close files, clear caches etc
> - people do not like existing things be used in new ways
> (least surprise principle)
>
> So I conclude:
>
> - SIGHUP, as ugly as it is, must stay with existing semantics
> - a new signal can be defined to just do file closure etc
>
> Unfortunately, this means that most systems will still use the terribly
> expensive during e.g. logrotation. However, this is considered
> acceptable because a) it always was this way, b) a higher demand
> environment then has options to avoid that. Over time, package
> maintainers my get maintainers of logrotate involved to change the HUP
> to the new signal.
>
> Am I grasping this right?

I learned this week that on a HUP rsyslog will loose any messages that it
has in it's memory queue (due to the fact that it basicly does a full
shutdown and restart). I suspect that many other people would be surprised
at this as well.

It looks like we have two options.

1. HUP will notice config file changes, but can loose queued logs

2. HUP will not notice config file changes, but will not loose queued logs

I think that there would be less surprise in the second case.

all the scripts that use HUP to roll logs don't really care about the
config files

I guess there is a third case, make a HUP not drop the queue, but if the
config file options for the queue change significantly, this can be a very
difficult thing to do

David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Fri, 2008-10-17 at 21:40 -0700, david@lang.hm wrote:
> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
>
> > So let me try to sum up where we are now:
> >
> > - it is considered useful to have a full daemon restart be done via HUP
> > - we can not sufficiently simply detect whether the configuration
> > has changed or not
> > - there are situations where it is useful to have the ability to
> > just close files, clear caches etc
> > - people do not like existing things be used in new ways
> > (least surprise principle)
> >
> > So I conclude:
> >
> > - SIGHUP, as ugly as it is, must stay with existing semantics
> > - a new signal can be defined to just do file closure etc
> >
> > Unfortunately, this means that most systems will still use the terribly
> > expensive during e.g. logrotation. However, this is considered
> > acceptable because a) it always was this way, b) a higher demand
> > environment then has options to avoid that. Over time, package
> > maintainers my get maintainers of logrotate involved to change the HUP
> > to the new signal.
> >
> > Am I grasping this right?
>
> I learned this week that on a HUP rsyslog will loose any messages that it
> has in it's memory queue (due to the fact that it basicly does a full
> shutdown and restart). I suspect that many other people would be surprised
> at this as well.

Well, actually it depends. In the default configuration, this is true.
But the spirit of the default configuration (suitable for most simple
cases) is "I don't care about message loss". If you care about message
loss, queues need to be configured so that no already-gathered messages
are lost between sessions. If done so, messages are written to files
before a shutdown and read from it upon restart. Of course, messages are
still lost while the system is doing the restart (new ones).

For a HUP and a very busy system, this can mean it takes a couple of
seconds, even minutes, to do a full restart.

After I noticed that the "message loss on HUP if not configured to avoid
it" is not obvious, I added some explicit doc to the man pages so that
people are warned they do a full restart.

>
> It looks like we have two options.
>
> 1. HUP will notice config file changes, but can loose queued logs
>
> 2. HUP will not notice config file changes, but will not loose queued logs
>
> I think that there would be less surprise in the second case.

see comments above

>
> all the scripts that use HUP to roll logs don't really care about the
> config files

that was my initial observation. I will define another signal to just
close the file. I hope I can get this done next week. Unfortunately, the
financial crisis begins to hit the real world and this causes me to
currently spent less time on new additions to rsyslog and more on
profit-generating projects, but that should be doable in a day or two
(the output module interface needs to be extend, a new administrative
linked list implemented and the signal handler be done - changes to
outputs are also necessary).

>
> I guess there is a third case, make a HUP not drop the queue, but if the
> config file options for the queue change significantly, this can be a very
> difficult thing to do

This is extremely difficult, but of course doable. I think, however,
this is only needed for a very limited number of cases. So this is
probably something that I will only work on if a sponsor for this work
shows up.

Rainer
>
> David Lang
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Sat, 18 Oct 2008, Rainer Gerhards wrote:

> On Fri, 2008-10-17 at 21:40 -0700, david@lang.hm wrote:
>> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
>>
>>> So let me try to sum up where we are now:
>>>
>>> - it is considered useful to have a full daemon restart be done via HUP
>>> - we can not sufficiently simply detect whether the configuration
>>> has changed or not
>>> - there are situations where it is useful to have the ability to
>>> just close files, clear caches etc
>>> - people do not like existing things be used in new ways
>>> (least surprise principle)
>>>
>>> So I conclude:
>>>
>>> - SIGHUP, as ugly as it is, must stay with existing semantics
>>> - a new signal can be defined to just do file closure etc
>>>
>>> Unfortunately, this means that most systems will still use the terribly
>>> expensive during e.g. logrotation. However, this is considered
>>> acceptable because a) it always was this way, b) a higher demand
>>> environment then has options to avoid that. Over time, package
>>> maintainers my get maintainers of logrotate involved to change the HUP
>>> to the new signal.
>>>
>>> Am I grasping this right?
>>
>> I learned this week that on a HUP rsyslog will loose any messages that it
>> has in it's memory queue (due to the fact that it basicly does a full
>> shutdown and restart). I suspect that many other people would be surprised
>> at this as well.
>
> Well, actually it depends. In the default configuration, this is true.
> But the spirit of the default configuration (suitable for most simple
> cases) is "I don't care about message loss".

there's a big difference between "I don't care about message loss if a
system crashed" and "I don't care about message loss at all"

admins should evaluate the probability of the different message loss
scenerios and decide on the appropriate cost/benifit for their situation.

it's very possible that someone may not care about loosing logs when a
system crashes (becouse it's a fairly rare event), but isn't willing to
loose logs just to rotate log files.

> If you care about message
> loss, queues need to be configured so that no already-gathered messages
> are lost between sessions. If done so, messages are written to files
> before a shutdown and read from it upon restart. Of course, messages are
> still lost while the system is doing the restart (new ones).
>
> For a HUP and a very busy system, this can mean it takes a couple of
> seconds, even minutes, to do a full restart.

and unfortunantly, the process of writing the pending queue to disk and
then reading it back after a restart can greatly lengthen the restart,
causing incoming logs to be missed (depending on how able the log sender
is to queue the log messages for your)

> After I noticed that the "message loss on HUP if not configured to avoid
> it" is not obvious, I added some explicit doc to the man pages so that
> people are warned they do a full restart.

thanks.

>>
>> It looks like we have two options.
>>
>> 1. HUP will notice config file changes, but can loose queued logs
>>
>> 2. HUP will not notice config file changes, but will not loose queued logs
>>
>> I think that there would be less surprise in the second case.
>
> see comments above

the purpose of this message was mostly to try and prompt the other people
who commented in this thread to think about the issue and comment on their
expectations.

>>
>> all the scripts that use HUP to roll logs don't really care about the
>> config files
>
> that was my initial observation. I will define another signal to just
> close the file. I hope I can get this done next week. Unfortunately, the
> financial crisis begins to hit the real world and this causes me to
> currently spent less time on new additions to rsyslog and more on
> profit-generating projects, but that should be doable in a day or two
> (the output module interface needs to be extend, a new administrative
> linked list implemented and the signal handler be done - changes to
> outputs are also necessary).

I am not trying to preasure you into rushing on this, just to get opinions
on which is the 'more expected' behavior for HUP

>>
>> I guess there is a third case, make a HUP not drop the queue, but if the
>> config file options for the queue change significantly, this can be a very
>> difficult thing to do
>
> This is extremely difficult, but of course doable. I think, however,
> this is only needed for a very limited number of cases. So this is
> probably something that I will only work on if a sponsor for this work
> shows up.

I would definantly not recommend this one becouse of the difficulty
involved, but in the interest of completeness I thought it needed to be
mentioned.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Sat, 2008-10-18 at 02:32 -0700, david@lang.hm wrote:
> On Sat, 18 Oct 2008, Rainer Gerhards wrote:
>
> > On Fri, 2008-10-17 at 21:40 -0700, david@lang.hm wrote:
> >> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
> >>
> >>> So let me try to sum up where we are now:
> >>>
> >>> - it is considered useful to have a full daemon restart be done via HUP
> >>> - we can not sufficiently simply detect whether the configuration
> >>> has changed or not
> >>> - there are situations where it is useful to have the ability to
> >>> just close files, clear caches etc
> >>> - people do not like existing things be used in new ways
> >>> (least surprise principle)
> >>>
> >>> So I conclude:
> >>>
> >>> - SIGHUP, as ugly as it is, must stay with existing semantics
> >>> - a new signal can be defined to just do file closure etc
> >>>
> >>> Unfortunately, this means that most systems will still use the terribly
> >>> expensive during e.g. logrotation. However, this is considered
> >>> acceptable because a) it always was this way, b) a higher demand
> >>> environment then has options to avoid that. Over time, package
> >>> maintainers my get maintainers of logrotate involved to change the HUP
> >>> to the new signal.
> >>>
> >>> Am I grasping this right?
> >>
> >> I learned this week that on a HUP rsyslog will loose any messages that it
> >> has in it's memory queue (due to the fact that it basicly does a full
> >> shutdown and restart). I suspect that many other people would be surprised
> >> at this as well.
> >
> > Well, actually it depends. In the default configuration, this is true.
> > But the spirit of the default configuration (suitable for most simple
> > cases) is "I don't care about message loss".
>
> there's a big difference between "I don't care about message loss if a
> system crashed" and "I don't care about message loss at all"

That's right. But I was not talking about a crash condition. I was
talking about a shutdown (or a restart). With the default settings,
rsyslog does what sysklogd does, that is (mostly) not care about message
loss. If you shut down the daemon, it tries to drain the queue for a
short timeout. If all messages can be processed during that time, we are
fine and terminate rsyslogd without message loss.

However, if there are too many messages queued, the daemon terminates
after the timeout has expired and discards whatever messages are left
unprocessed. This is done in an effort to keep compatible with sysklogd.
Also, in most scenarios it is highly unlikely that message loss will
occur, because the queue will not hold extended data for an extended
period of time. If, however, this is anticipated, there are ways to
configure rsyslog not to loose messages. For example, the timeout can be
increased and/or the queue can be made disk-assisted and data be written
to disk if they can not be processed during the timeout.

During queue drain, no new messages are accepted (otherwise, how should
we finally shut down?). Given a busy system, there is unavoidable
message loss. Either you use old messages you already had in the queue,
but have a quick restart and loose few new messages. Or you save the
already existing messages but experience loss of new messages. This is
another reason for the default, which I consider the best compromise
under the circumstances given. This is also a good indication why you
should not unnecessarily restart rsyslgod (or any other syslogd, because
this is not a project-specific problem but rather a general one).

With fully reliable protocols, you can of course make the sender block
while you are shutting down. But all of this requires very careful
planning. I assume folks who want to do that either have the necessary
knowledge themselvs or hire someone who does.

So there are three scenarios:

a) hup
b) regular shutdown
c) system abort

I think you did not take into account b), which is a quite common case.

Other than that, I fully agree to you post.

Rainer

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

On Mon, 20 Oct 2008, Rainer Gerhards wrote:

> On Sat, 2008-10-18 at 02:32 -0700, david@lang.hm wrote:
>> On Sat, 18 Oct 2008, Rainer Gerhards wrote:
>>
>>> On Fri, 2008-10-17 at 21:40 -0700, david@lang.hm wrote:
>>>> On Mon, 6 Oct 2008, Rainer Gerhards wrote:
>>>>
>>>>> So let me try to sum up where we are now:
>>>>>
>>>>> - it is considered useful to have a full daemon restart be done via HUP
>>>>> - we can not sufficiently simply detect whether the configuration
>>>>> has changed or not
>>>>> - there are situations where it is useful to have the ability to
>>>>> just close files, clear caches etc
>>>>> - people do not like existing things be used in new ways
>>>>> (least surprise principle)
>>>>>
>>>>> So I conclude:
>>>>>
>>>>> - SIGHUP, as ugly as it is, must stay with existing semantics
>>>>> - a new signal can be defined to just do file closure etc
>>>>>
>>>>> Unfortunately, this means that most systems will still use the terribly
>>>>> expensive during e.g. logrotation. However, this is considered
>>>>> acceptable because a) it always was this way, b) a higher demand
>>>>> environment then has options to avoid that. Over time, package
>>>>> maintainers my get maintainers of logrotate involved to change the HUP
>>>>> to the new signal.
>>>>>
>>>>> Am I grasping this right?
>>>>
>>>> I learned this week that on a HUP rsyslog will loose any messages that it
>>>> has in it's memory queue (due to the fact that it basicly does a full
>>>> shutdown and restart). I suspect that many other people would be surprised
>>>> at this as well.
>>>
>>> Well, actually it depends. In the default configuration, this is true.
>>> But the spirit of the default configuration (suitable for most simple
>>> cases) is "I don't care about message loss".
>>
>> there's a big difference between "I don't care about message loss if a
>> system crashed" and "I don't care about message loss at all"
>
> That's right. But I was not talking about a crash condition. I was
> talking about a shutdown (or a restart). With the default settings,
> rsyslog does what sysklogd does, that is (mostly) not care about message
> loss. If you shut down the daemon, it tries to drain the queue for a
> short timeout. If all messages can be processed during that time, we are
> fine and terminate rsyslogd without message loss.

the big difference is that rsysklog does not maintain a queue, so when it
shuts down it doesn't loose any messages that it has processed (even with
async writes it has passed the logs to the OS and so those logs are not
affected by the sysklog shutdown/HUP)

> However, if there are too many messages queued, the daemon terminates
> after the timeout has expired and discards whatever messages are left
> unprocessed. This is done in an effort to keep compatible with sysklogd.
> Also, in most scenarios it is highly unlikely that message loss will
> occur, because the queue will not hold extended data for an extended
> period of time. If, however, this is anticipated, there are ways to
> configure rsyslog not to loose messages. For example, the timeout can be
> increased and/or the queue can be made disk-assisted and data be written
> to disk if they can not be processed during the timeout.
>
> During queue drain, no new messages are accepted (otherwise, how should
> we finally shut down?). Given a busy system, there is unavoidable
> message loss. Either you use old messages you already had in the queue,
> but have a quick restart and loose few new messages. Or you save the
> already existing messages but experience loss of new messages. This is
> another reason for the default, which I consider the best compromise
> under the circumstances given. This is also a good indication why you
> should not unnecessarily restart rsyslgod (or any other syslogd, because
> this is not a project-specific problem but rather a general one).

this conflicts with the need to rotate the log files.

while we have discussed the possibility of using a different signal to
allow the logs to be rotated, I was bringing the subject up again becouse
I think the HUP behavior is unexpected enough that it may be less surprise
if a HUP doesn't re-read config files but does the close/open needed for
log rotation than the alturnative of having HUP remain as-in with a
different signal for log rotation.

I'm not suggesting more work, I'm suggesting that it may make sense to
just use HUP for the close/open and if someone needs the full config
reload they do a stop/start instead.

> With fully reliable protocols, you can of course make the sender block
> while you are shutting down. But all of this requires very careful
> planning. I assume folks who want to do that either have the necessary
> knowledge themselvs or hire someone who does.
>
> So there are three scenarios:
>
> a) hup
> b) regular shutdown
> c) system abort
>
> I think you did not take into account b), which is a quite common case.

there is case d) log rotation

when someone does a stop/start (case b) they expect logs in a queue to be
lost (you are fully stopping the process after all), I don't think that's
what people would expect from a HUP.

David Lang

> Other than that, I fully agree to you post.
>
> Rainer
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com