Mailing List Archive

Hi HKS-san,

+1

> 2 - This will avoid a lot of silly questions without any supporting
> information. At least when you get a complaint like "Rsyslogd won't
> start!" it will generally include a "It just says 'configuration file
> invalid: broken syntax at line 135.'

In addition,
'... at line 135. See man 5 rsyslog.conf or $doc/rsyslog_conf.html.'

Any ideas?

Best Rio.

On 2008/07/25, at 23:33, (private) HKS wrote:

> Hi Rainer,
>
> Interesting post. The tendency to ignore errors/diagnostics is not
> limited to rsyslog, it's something that plagues pretty much every
> piece of software ever created anywhere. Some users are ignorant of
> basic diagnostic tools like syslog, --help flags, tcpdump, and man
> pages. Others just need a bit of prodding before they slap themselves
> and realize that they forgot some basic troubleshooting. Some are lazy
> slobs.
>
> However, rsyslog lends itself to error reports that lack
> troubleshooting for one huge reason: errors are not reliably reported
> in the default configuration. It's happened a few times that I start
> rsyslogd at the command line, and yet there's nothing in the process
> table, and nothing dumped to my logs. I've been able to track down the
> problem with -dn switches, but that involves weeding through a lot of
> irrelevant information.
>
> There are three modifications I'd love to see that would help
> resolve this:
>
> 1 - Configuration errors should be fatal
> 2 - Configuration and other runtime errors should be printed to STDERR
> 3 - By default, rsyslogd should log all of its own errors (fatal or
> not) to /var/log/rsyslog.log. This should also be user-configurable
>
> My reasoning:
>
> 1 - If rsyslog doesn't understand your config file, obviously it won't
> be doing what you intend it to. There's little benefit in running with
> a horked configuration, but if you don't test thoroughly, the cost
> could be devastating. Some users won't know they're not logging things
> until they need them - then, if they keep their jobs, they probably
> won't keep rsyslog ("This software doesn't even log ssh attempts!").
>
> 2 - This will avoid a lot of silly questions without any supporting
> information. At least when you get a complaint like "Rsyslogd won't
> start!" it will generally include a "It just says 'configuration file
> invalid: broken syntax at line 135.'" For most users, this kind of
> error reporting will be enough to lead them to a solution. If nothing
> else, it removes the need to explain how to find the errors.
>
> 3 - For those more subtle problems, and things you might miss upon
> bootup. This also gives you a simple, easily accessible debugging
> source with minimal security implications and additional code and no
> required user configuration. I'm not sure how the code is written, but
> I'd prefer to see this happen even if the configuration file can't be
> read. Perhaps an $RsyslogErrors directive that, by default, references
> a different module that just dumps directly to a file? Users can
> configure it to put rsyslog errors through the actual logging engine
> and then manage it with typical syslog selector/action lines.
>
> The HTTP server is an interesting idea, but not something I would make
> use of. The potential security problems are enormous, and it would
> introduce too many additional factors to the debugging process: is the
> firewall right? Was there already a process listening on that port? Is
> the user's configuration file correct? These are the kinds of things
> you already have to deal with on rsyslog itself - why force the
> debugging process to go through them again?
>
> Hope that helps, and sorry for the long email.
> -HKS
>
>
>
>
> On Fri, Jul 25, 2008 at 6:31 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> Thanks, HKS, for the good answers. There is one thing I would like to
>> bring to your attention: I often see that people do not check for
>> rsyslog error messages themselves. That complicates setup. I do not
>> blame anyone, but would like to make you aware of the problem.
>>
>> I have just blogged about a potential (partial) solution and will
>> try to
>> implement it:
>>
>> http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.ht
>> ml
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of (private) HKS
>>> Sent: Thursday, July 24, 2008 11:36 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Help with Ommail Configuration
>>>
>>> Oh - one other possibility. rsyslogd has to have mail enabled at
>>> compile time to work with it at runtime. check your catchall
>>> logfile -
>>> if it has messages like this, you didn't compile it in:
>>>
>>> rsyslogd: could not load module '/usr/local/lib/rsyslog/ommail.so',
>>> dlopen: /usr/local/lib/rsyslog/ommail.so: cannot open shared object
>>> file: No such file or directory
>>>
>>> To fix this, you'll need to reinstall. This time, when you run
>>> ./configure be sure to include --enable-mail. make install clean and
>>> you should be good to go...
>>>
>>> -HKS
>>>
>>> On Thu, Jul 24, 2008 at 5:01 PM, (private) HKS <hks.private at gmail.com
>>> >
>>> wrote:
>>>> A few things to try:
>>>>
>>>> - You load ommail.so twice, once at the top and once right above
>>> your
>>>> $ActionMail... lines. I don't think this will break it, but it's
>>>> unnecessary - delete the second one.
>>>>
>>>> - $ActionExecOnlyOnceEveryInterval 21600 means that it's only going
>>>> to attempt sending a message once every 6 hours. For testing
>>> purposes,
>>>> this will be obnoxious. Until you're ready for production, change
>>>> it
>>>> to:
>>>> $ActionExecOnlyOnceEveryInterval 30
>>>>
>>>> - Send everything to make sure you're not missing it based on
>>>> something in the property-replacer/templates/whatever. Replace "if
>>>> $msg contains 'hard disk fatal failure' then :ommail:;mailBody"
>> with:
>>>> *.* :ommail:;mailBody
>>>>
>>>> Try again. Try logging a few messages from the localhost first
>>>> (with
>>>> RHEL you can just run "logger test" to log a user.notice message
>> with
>>>> content "test") and see if you get them.
>>>>
>>>> If you don't, check the mail logs on your mail server to see if it
>>>> ever received the message. If not, it's time to break out tcpdump
>> and
>>>> see if the packets are ever being generated.
>>>>
>>>> Hope that helps.
>>>>
>>>> -HKS
>>>>
>>>>
>>>>
>>>> On Thu, Jul 24, 2008 at 3:50 PM, Goutos, Kevin (DOB)
>>>> <Kevin.Goutos at budget.state.ny.us> wrote:
>>>>> Hello all,
>>>>>
>>>>> First off, I am not very Linux savvy. I don't have a lot of
>>> experience
>>>>> other then a basic course. This is probably way past my knowledge,
>>> but I
>>>>> really need to get it done. Appreciate any help you guys have to
>>> offer.
>>>>>
>>>>> I am working on a Red Hat Enterprise 4 box and I am running the
>>> latest
>>>>> edition of rsyslog. I currently have rsyslog configured to receive
>>>>> messages remotely via UDP. I am trying to set it up so that it
>>>>> will
>>> send
>>>>> out E-mail messages to the system Admin's based on the severity
>>> level of
>>>>> the log files I am receiving. I would like it so that any device
>>> that
>>>>> sends a log with a critical, alert, or emergency level facility
>> will
>>>>> send out an e-mail to a specific address.
>>>>>
>>>>> Here is my rsyslog.conf file. I used the sample code from Rainer
>>>>> Gerhards configuration page. I tried sending a test syslog with
>>> 'hard
>>>>> disk fatal failure' in it, but it is not sending out mail. Also
>>> tried
>>>>> without the templates below thinking it would just send me an
>>>>> email
>>> for
>>>>> every syslog that I received, but it doesn't appear to be sending
>>> mail.
>>>>> Any thoughts on what I am doing wrong. I'm sure there is a lot I
>>> need to
>>>>> do, so please let me know. Thanks!
>>>>>
>>>>> $template mailSubject,"disk problem on %hostname%"
>>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"'
>>>>>
>>>>> I edited out the 3 lines below in the code for security reasons..
>>>>> $ActionMailSMTPServer <ip of smtp server>
>>>>> $ActionMailFrom <from address>
>>>>> $ActionMailTo <my email>
>>>>>
>>>>>
>>>>>
>>>>> Here is my code from rsyslog.conf below.
>>>>>
>>>>>
>>>>>
>>>>> # if you experience problems, check
>>>>> # http://www.rsyslog.com/troubleshoot for assistance
>>>>>
>>>>> # rsyslog v3: load input modules
>>>>> # If you do not load inputs, nothing happens!
>>>>> # You may need to set the module load path if modules are not
>> found.
>>>>>
>>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>>> $ModLoad imuxsock.so # provides support for local system logging
>>> (e.g.
>>>>> via logger command)
>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>> $ModLoad ommail
>>>>>
>>>>> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated%
>>>>> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
>>>>>
>>>>> # Log all kernel messages to the console.
>>>>> # Logging much else clutters up the screen.
>>>>> #kern.*
>> /dev/console
>>>>>
>>>>> # Log anything (except mail) of level info or higher.
>>>>> # Don't log private authentication messages!
>>>>> *.info;mail.none;authpriv.none;cron.none
>>>>> -/var/log/messages
>>>>>
>>>>> # The authpriv file has restricted access.
>>>>> authpriv.*
>>> /var/log/secure
>>>>>
>>>>> # Log all the mail messages in one place.
>>>>> mail.*
>>>>> -/var/log/maillog
>>>>>
>>>>> # Log cron stuff
>>>>> cron.* -
>>> /var/log/cron
>>>>>
>>>>> # Everybody gets emergency messages
>>>>> *.emerg *
>>>>>
>>>>> # Save news errors of level crit and higher in a special file.
>>>>> uucp,news.crit
>>>>> -/var/log/spooler
>>>>>
>>>>> # Save boot messages also to boot.log
>>>>> local7.*
>>>>> /var/log/boot.log
>>>>>
>>>>> #Catch all incoming syslog messages
>>>>> *.*
>>>>> /var/log/catchall;TraditionalFormatWithPRI
>>>>>
>>>>> # Remote Logging (we use TCP for reliable delivery)
>>>>> # An on-disk queue is created for this action. If the remote host
>> is
>>>>> # down, messages are spooled to disk and sent when it is up again.
>>>>> $WorkDirectory /rsyslog/spool # where to place spool files
>>>>> $ActionQueueFileName uniqName # unique name prefix for spool files
>>>>> $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
>>>>> possible)
>>>>> $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
>>>>> $ActionQueueType LinkedList # run asynchronously
>>>>> $ActionResumeRetryCount -1 # infinite retries if host is down
>>>>> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
>>>>> optional
>>>>> *.* @10.57.106.140:514
>>>>>
>>>>> $ModLoad ommail
>>>>> $ActionMailSMTPServer <ip of smtp server>
>>>>> $ActionMailFrom <from address>
>>>>> $ActionMailTo <my email>
>>>>> $template mailSubject,"disk problem on %hostname%"
>>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
>>>>> $ActionMailSubject mailSubject
>>>>> # make sure we receive a mail only once in six
>>>>> # hours (21,600 seconds ;))
>>>>> $ActionExecOnlyOnceEveryInterval 21600
>>>>> # the if ... then ... mailBody mus be on one line!
>>>>> if $msg contains 'hard disk fatal failure' then :ommail:;mailBody
>>>>>
>>>>>
>>>>> # ######### Receiving Messages from Remote Hosts ##########
>>>>> # TCP Syslog Server:
>>>>> # provides TCP syslog reception and GSS-API (if compiled to
>>>>> support
>>> it)
>>>>> $ModLoad imtcp.so # load module
>>>>> $InputTCPServerRun 514 # start up TCP listener at port 514
>>>>>
>>>>> # UDP Syslog Server:
>>>>> $ModLoad imudp.so # provides UDP syslog reception
>>>>> $UDPServerRun 514 # start a UDP syslog server at standard port
>>>>> --------------------------------------------------------
>>>>> This e-mail, including any attachments, may be confidential,
>>> privileged or otherwise legally protected. If you have received this
>> e-
>>> mail in error, or from someone who was not authorized to send it to
>>> you, do not disseminate, copy or otherwise use this e-mail or its
>>> attachments. Please notify the sender immediately if you have
>>> received
>>> this e-mail by mistake, and delete it from your system.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

########################################################################
Ryo Fujita <rfujita at redhat.com>
Senior Solution Architect, RHCE
Red Hat K.K.
TEL +81-3-5798-8500
FAX +81-3-5798-8599
Ebisu Neonato 8F
4-1-18 Ebisu, Shibuya-ku,
Tokyo Japan 1500013
########################################################################

On Fri, 2008-07-25 at 10:33 -0400, (private) HKS wrote:
> Hi Rainer,
>
> Interesting post. The tendency to ignore errors/diagnostics is not
> limited to rsyslog, it's something that plagues pretty much every
> piece of software ever created anywhere.

I totally agree, but I think the situation is even worse for a syslogd.
Because it *is* the usual error reporting system. I think we are a bit
in the same situation (if you allow that analogy) like HIV which is
attacking the human immune system. We have a problem in the
troubleshooting system itself, and it may even prevent us from conveying
troubleshooting information :(

> Some users are ignorant of
> basic diagnostic tools like syslog, --help flags, tcpdump, and man
> pages. Others just need a bit of prodding before they slap themselves
> and realize that they forgot some basic troubleshooting. Some are lazy
> slobs.
>
> However, rsyslog lends itself to error reports that lack
> troubleshooting for one huge reason: errors are not reliably reported
> in the default configuration.


> It's happened a few times that I start
> rsyslogd at the command line, and yet there's nothing in the process
> table, and nothing dumped to my logs.

Could you elaborate a bit about above sentence - I have to admit I do
not fully get you...

> I've been able to track down the
> problem with -dn switches, but that involves weeding through a lot of
> irrelevant information.
>
> There are three modifications I'd love to see that would help resolve this:
>
> 1 - Configuration errors should be fatal
> 2 - Configuration and other runtime errors should be printed to STDERR

Does that really help? If so, that would be extremely easy to add. But
does somebody see them?

> 3 - By default, rsyslogd should log all of its own errors (fatal or
> not) to /var/log/rsyslog.log. This should also be user-configurable

This default comes into the way of some distros: they use different
pathes. I really don't like the idea of embedding any information about
the file system into the binary. Just think about embedded systems...

>
> My reasoning:
>
> 1 - If rsyslog doesn't understand your config file, obviously it won't
> be doing what you intend it to. There's little benefit in running with
> a horked configuration, but if you don't test thoroughly, the cost
> could be devastating. Some users won't know they're not logging things
> until they need them - then, if they keep their jobs, they probably
> won't keep rsyslog ("This software doesn't even log ssh attempts!").

That's quite controversal and and does not match rsyslog's root
philosophy. Let me explain:

The syslogd is an extremely important piece of software. If it is not
functioning,

a) vital data can be lost
b) the system as whole may hang (due to the fact that the log
socket gets filled up and so other processes block on it)

As such, rsyslogd is written in a sense that it tries to continue to
work under almost all circumstances. It even tries to survive a
system-wide low memory condition, though I am not 100% convinced it will
manage to do that in all cases. At least it is coded careful to survive
as much as possible.

Now let's assume that a configuration error happens that prevents
rsyslogd from executing an otherwise perfect configuration. Should we
actually stop it from running? And live with the results? Or should we
do whatever we can do and do that as good as possible? What, if it is
not a configuration issue but a hard disk failure or an admin error that
make part of the config unavailable (e.g. not able to load module, not
able to write file)? Assuming we can carry out other actions - should we
really abort? This may even come at the price that we can NOT warn users
(via wall) of a serious problem. And what with warnings? Things that
look wrong, but may be OK. Reason to abort? Or continue run? Tough
decision. So I still think the right thing to do is to run as long as
there is limited sense in it, because everything else would actually be
worse. Please note that rsyslogd even comes with a hardcoded emergency
configuration which is used in case the real configuration file can not
be obtained.

What I could add, however, is an option to make it stop on any config
error (not warning). However, I fear that, if turned on, the results can
be fatal in some cases...

>
> 2 - This will avoid a lot of silly questions without any supporting
> information. At least when you get a complaint like "Rsyslogd won't
> start!" it will generally include a "It just says 'configuration file
> invalid: broken syntax at line 135.'" For most users, this kind of
> error reporting will be enough to lead them to a solution. If nothing
> else, it removes the need to explain how to find the errors.
>
> 3 - For those more subtle problems, and things you might miss upon
> bootup. This also gives you a simple, easily accessible debugging
> source with minimal security implications and additional code and no
> required user configuration. I'm not sure how the code is written, but
> I'd prefer to see this happen even if the configuration file can't be
> read. Perhaps an $RsyslogErrors directive that, by default, references
> a different module that just dumps directly to a file? Users can
> configure it to put rsyslog errors through the actual logging engine
> and then manage it with typical syslog selector/action lines.

All internally-generated messages are run through a specific interface.
For the "diag plugin", I will define a hook, where a diag server can
register to receive these messages. Maybe a good compromise would be to
add this $RsyslogErrors directive and make it point to a file. So,
still, we do not necessarily have it, but package maintainers would
hopefully include it and point it to a specific file (what still means I
do not necessarily know where to look). A problem is source tarball
install, in which case the directive may not be set. But with a proper
sample rsyslog.conf...

> The HTTP server is an interesting idea, but not something I would make
> use of. The potential security problems are enormous, and it would
> introduce too many additional factors to the debugging process: is the
> firewall right? Was there already a process listening on that port? Is
> the user's configuration file correct? These are the kinds of things
> you already have to deal with on rsyslog itself - why force the
> debugging process to go through them again?

I see (and share) much of your argument. This is why I stayed away for
such a long time. But if you think a bit more about it, it has *very
interesting* capabilities. Eg. I may also use it to gather some
real-time view of internal state, emit commands like HUP and do a number
of other nice things. Again, security is problematic.

In any case, I'll give it a try. I have started to write a small plugin
which will dump the error messages within 10 minutes after startup. It
will not be a real http server but rather a listener that response to
anyone who connects to it (so that nc can be used as a client ;)). This
is very rough, but it has two side-effects:

a) it provides something to actually experiment with (and to extend if
we decide to keep that route)
b) it helps create the plumbing inside the internal interface that could
also be used by any other method, so the potential waste of time is
limited

So while I object some suggestions, I am very interested in a
discussion. The rsyslog "keep running" philosophy *is* a discussion
topic, just expect me to argue strongly in favor of it (pls think about
the HIV analogy to see why I think we have a unique situation in case of
a syslogd).

Thanks for the good post, and I guess I just wrote another long mail ;)

Rainer
>
> Hope that helps, and sorry for the long email.
> -HKS
>
>
>
>
> On Fri, Jul 25, 2008 at 6:31 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Thanks, HKS, for the good answers. There is one thing I would like to
> > bring to your attention: I often see that people do not check for
> > rsyslog error messages themselves. That complicates setup. I do not
> > blame anyone, but would like to make you aware of the problem.
> >
> > I have just blogged about a potential (partial) solution and will try to
> > implement it:
> >
> > http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.ht
> > ml
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of (private) HKS
> >> Sent: Thursday, July 24, 2008 11:36 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Help with Ommail Configuration
> >>
> >> Oh - one other possibility. rsyslogd has to have mail enabled at
> >> compile time to work with it at runtime. check your catchall logfile -
> >> if it has messages like this, you didn't compile it in:
> >>
> >> rsyslogd: could not load module '/usr/local/lib/rsyslog/ommail.so',
> >> dlopen: /usr/local/lib/rsyslog/ommail.so: cannot open shared object
> >> file: No such file or directory
> >>
> >> To fix this, you'll need to reinstall. This time, when you run
> >> ./configure be sure to include --enable-mail. make install clean and
> >> you should be good to go...
> >>
> >> -HKS
> >>
> >> On Thu, Jul 24, 2008 at 5:01 PM, (private) HKS <hks.private at gmail.com>
> >> wrote:
> >> > A few things to try:
> >> >
> >> > - You load ommail.so twice, once at the top and once right above
> >> your
> >> > $ActionMail... lines. I don't think this will break it, but it's
> >> > unnecessary - delete the second one.
> >> >
> >> > - $ActionExecOnlyOnceEveryInterval 21600 means that it's only going
> >> > to attempt sending a message once every 6 hours. For testing
> >> purposes,
> >> > this will be obnoxious. Until you're ready for production, change it
> >> > to:
> >> > $ActionExecOnlyOnceEveryInterval 30
> >> >
> >> > - Send everything to make sure you're not missing it based on
> >> > something in the property-replacer/templates/whatever. Replace "if
> >> > $msg contains 'hard disk fatal failure' then :ommail:;mailBody"
> > with:
> >> > *.* :ommail:;mailBody
> >> >
> >> > Try again. Try logging a few messages from the localhost first (with
> >> > RHEL you can just run "logger test" to log a user.notice message
> > with
> >> > content "test") and see if you get them.
> >> >
> >> > If you don't, check the mail logs on your mail server to see if it
> >> > ever received the message. If not, it's time to break out tcpdump
> > and
> >> > see if the packets are ever being generated.
> >> >
> >> > Hope that helps.
> >> >
> >> > -HKS
> >> >
> >> >
> >> >
> >> > On Thu, Jul 24, 2008 at 3:50 PM, Goutos, Kevin (DOB)
> >> > <Kevin.Goutos at budget.state.ny.us> wrote:
> >> >> Hello all,
> >> >>
> >> >> First off, I am not very Linux savvy. I don't have a lot of
> >> experience
> >> >> other then a basic course. This is probably way past my knowledge,
> >> but I
> >> >> really need to get it done. Appreciate any help you guys have to
> >> offer.
> >> >>
> >> >> I am working on a Red Hat Enterprise 4 box and I am running the
> >> latest
> >> >> edition of rsyslog. I currently have rsyslog configured to receive
> >> >> messages remotely via UDP. I am trying to set it up so that it will
> >> send
> >> >> out E-mail messages to the system Admin's based on the severity
> >> level of
> >> >> the log files I am receiving. I would like it so that any device
> >> that
> >> >> sends a log with a critical, alert, or emergency level facility
> > will
> >> >> send out an e-mail to a specific address.
> >> >>
> >> >> Here is my rsyslog.conf file. I used the sample code from Rainer
> >> >> Gerhards configuration page. I tried sending a test syslog with
> >> 'hard
> >> >> disk fatal failure' in it, but it is not sending out mail. Also
> >> tried
> >> >> without the templates below thinking it would just send me an email
> >> for
> >> >> every syslog that I received, but it doesn't appear to be sending
> >> mail.
> >> >> Any thoughts on what I am doing wrong. I'm sure there is a lot I
> >> need to
> >> >> do, so please let me know. Thanks!
> >> >>
> >> >> $template mailSubject,"disk problem on %hostname%"
> >> >> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"'
> >> >>
> >> >> I edited out the 3 lines below in the code for security reasons..
> >> >> $ActionMailSMTPServer <ip of smtp server>
> >> >> $ActionMailFrom <from address>
> >> >> $ActionMailTo <my email>
> >> >>
> >> >>
> >> >>
> >> >> Here is my code from rsyslog.conf below.
> >> >>
> >> >>
> >> >>
> >> >> # if you experience problems, check
> >> >> # http://www.rsyslog.com/troubleshoot for assistance
> >> >>
> >> >> # rsyslog v3: load input modules
> >> >> # If you do not load inputs, nothing happens!
> >> >> # You may need to set the module load path if modules are not
> > found.
> >> >>
> >> >> $ModLoad immark.so # provides --MARK-- message capability
> >> >> $ModLoad imuxsock.so # provides support for local system logging
> >> (e.g.
> >> >> via logger command)
> >> >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >> >> $ModLoad ommail
> >> >>
> >> >> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated%
> >> >> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
> >> >>
> >> >> # Log all kernel messages to the console.
> >> >> # Logging much else clutters up the screen.
> >> >> #kern.*
> > /dev/console
> >> >>
> >> >> # Log anything (except mail) of level info or higher.
> >> >> # Don't log private authentication messages!
> >> >> *.info;mail.none;authpriv.none;cron.none
> >> >> -/var/log/messages
> >> >>
> >> >> # The authpriv file has restricted access.
> >> >> authpriv.*
> >> /var/log/secure
> >> >>
> >> >> # Log all the mail messages in one place.
> >> >> mail.*
> >> >> -/var/log/maillog
> >> >>
> >> >> # Log cron stuff
> >> >> cron.* -
> >> /var/log/cron
> >> >>
> >> >> # Everybody gets emergency messages
> >> >> *.emerg *
> >> >>
> >> >> # Save news errors of level crit and higher in a special file.
> >> >> uucp,news.crit
> >> >> -/var/log/spooler
> >> >>
> >> >> # Save boot messages also to boot.log
> >> >> local7.*
> >> >> /var/log/boot.log
> >> >>
> >> >> #Catch all incoming syslog messages
> >> >> *.*
> >> >> /var/log/catchall;TraditionalFormatWithPRI
> >> >>
> >> >> # Remote Logging (we use TCP for reliable delivery)
> >> >> # An on-disk queue is created for this action. If the remote host
> > is
> >> >> # down, messages are spooled to disk and sent when it is up again.
> >> >> $WorkDirectory /rsyslog/spool # where to place spool files
> >> >> $ActionQueueFileName uniqName # unique name prefix for spool files
> >> >> $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> >> >> possible)
> >> >> $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> >> >> $ActionQueueType LinkedList # run asynchronously
> >> >> $ActionResumeRetryCount -1 # infinite retries if host is down
> >> >> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> >> >> *.* @10.57.106.140:514
> >> >>
> >> >> $ModLoad ommail
> >> >> $ActionMailSMTPServer <ip of smtp server>
> >> >> $ActionMailFrom <from address>
> >> >> $ActionMailTo <my email>
> >> >> $template mailSubject,"disk problem on %hostname%"
> >> >> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
> >> >> $ActionMailSubject mailSubject
> >> >> # make sure we receive a mail only once in six
> >> >> # hours (21,600 seconds ;))
> >> >> $ActionExecOnlyOnceEveryInterval 21600
> >> >> # the if ... then ... mailBody mus be on one line!
> >> >> if $msg contains 'hard disk fatal failure' then :ommail:;mailBody
> >> >>
> >> >>
> >> >> # ######### Receiving Messages from Remote Hosts ##########
> >> >> # TCP Syslog Server:
> >> >> # provides TCP syslog reception and GSS-API (if compiled to support
> >> it)
> >> >> $ModLoad imtcp.so # load module
> >> >> $InputTCPServerRun 514 # start up TCP listener at port 514
> >> >>
> >> >> # UDP Syslog Server:
> >> >> $ModLoad imudp.so # provides UDP syslog reception
> >> >> $UDPServerRun 514 # start a UDP syslog server at standard port
> >> >> --------------------------------------------------------
> >> >> This e-mail, including any attachments, may be confidential,
> >> privileged or otherwise legally protected. If you have received this
> > e-
> >> mail in error, or from someone who was not authorized to send it to
> >> you, do not disseminate, copy or otherwise use this e-mail or its
> >> attachments. Please notify the sender immediately if you have received
> >> this e-mail by mistake, and delete it from your system.
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

Hi Rio-san,

On Fri, 2008-07-25 at 23:56 +0900, Ryo Fujita wrote:
> Hi HKS-san,
>
> +1
>
> > 2 - This will avoid a lot of silly questions without any supporting
> > information. At least when you get a complaint like "Rsyslogd won't
> > start!" it will generally include a "It just says 'configuration file
> > invalid: broken syntax at line 135.'
>
> In addition,
> '... at line 135. See man 5 rsyslog.conf or $doc/rsyslog_conf.html.'
Same thought over here :)

The recent build even emits live links to troubleshooting information.
Log lines look like this:

2008-07-25T17:22:18.523900+02:00 rgf9dev rsyslogd-3003: invalid or
yet-unknown config file command - have you forgotten to load a module?
[try http://www.rsyslog.com/e/3003 ]

Note the part in the bracket at the end of the message. That repository
is far from being complete, but I hope more and more information will be
added.

I hope this is a useful addition. Together with the error number, we can
hopefully most often pinpoint the source of the problem (though the 3003
above is a good example of where it may not be possible to tell the
exact cause, but for sure give a good hint).

BTW: this is also one thing that makes me think about the scripting
engine. It shall provide better error reporting capabilties, but that's
harder than it initially sounds ;)

Rainer
>
> Any ideas?
>
> Best Rio.
>
> On 2008/07/25, at 23:33, (private) HKS wrote:
>
> > Hi Rainer,
> >
> > Interesting post. The tendency to ignore errors/diagnostics is not
> > limited to rsyslog, it's something that plagues pretty much every
> > piece of software ever created anywhere. Some users are ignorant of
> > basic diagnostic tools like syslog, --help flags, tcpdump, and man
> > pages. Others just need a bit of prodding before they slap themselves
> > and realize that they forgot some basic troubleshooting. Some are lazy
> > slobs.
> >
> > However, rsyslog lends itself to error reports that lack
> > troubleshooting for one huge reason: errors are not reliably reported
> > in the default configuration. It's happened a few times that I start
> > rsyslogd at the command line, and yet there's nothing in the process
> > table, and nothing dumped to my logs. I've been able to track down the
> > problem with -dn switches, but that involves weeding through a lot of
> > irrelevant information.
> >
> > There are three modifications I'd love to see that would help
> > resolve this:
> >
> > 1 - Configuration errors should be fatal
> > 2 - Configuration and other runtime errors should be printed to STDERR
> > 3 - By default, rsyslogd should log all of its own errors (fatal or
> > not) to /var/log/rsyslog.log. This should also be user-configurable
> >
> > My reasoning:
> >
> > 1 - If rsyslog doesn't understand your config file, obviously it won't
> > be doing what you intend it to. There's little benefit in running with
> > a horked configuration, but if you don't test thoroughly, the cost
> > could be devastating. Some users won't know they're not logging things
> > until they need them - then, if they keep their jobs, they probably
> > won't keep rsyslog ("This software doesn't even log ssh attempts!").
> >
> > 2 - This will avoid a lot of silly questions without any supporting
> > information. At least when you get a complaint like "Rsyslogd won't
> > start!" it will generally include a "It just says 'configuration file
> > invalid: broken syntax at line 135.'" For most users, this kind of
> > error reporting will be enough to lead them to a solution. If nothing
> > else, it removes the need to explain how to find the errors.
> >
> > 3 - For those more subtle problems, and things you might miss upon
> > bootup. This also gives you a simple, easily accessible debugging
> > source with minimal security implications and additional code and no
> > required user configuration. I'm not sure how the code is written, but
> > I'd prefer to see this happen even if the configuration file can't be
> > read. Perhaps an $RsyslogErrors directive that, by default, references
> > a different module that just dumps directly to a file? Users can
> > configure it to put rsyslog errors through the actual logging engine
> > and then manage it with typical syslog selector/action lines.
> >
> > The HTTP server is an interesting idea, but not something I would make
> > use of. The potential security problems are enormous, and it would
> > introduce too many additional factors to the debugging process: is the
> > firewall right? Was there already a process listening on that port? Is
> > the user's configuration file correct? These are the kinds of things
> > you already have to deal with on rsyslog itself - why force the
> > debugging process to go through them again?
> >
> > Hope that helps, and sorry for the long email.
> > -HKS
> >
> >
> >
> >
> > On Fri, Jul 25, 2008 at 6:31 AM, Rainer Gerhards
> > <rgerhards at hq.adiscon.com> wrote:
> >> Thanks, HKS, for the good answers. There is one thing I would like to
> >> bring to your attention: I often see that people do not check for
> >> rsyslog error messages themselves. That complicates setup. I do not
> >> blame anyone, but would like to make you aware of the problem.
> >>
> >> I have just blogged about a potential (partial) solution and will
> >> try to
> >> implement it:
> >>
> >> http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.ht
> >> ml
> >>
> >> Rainer
> >>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of (private) HKS
> >>> Sent: Thursday, July 24, 2008 11:36 PM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] Help with Ommail Configuration
> >>>
> >>> Oh - one other possibility. rsyslogd has to have mail enabled at
> >>> compile time to work with it at runtime. check your catchall
> >>> logfile -
> >>> if it has messages like this, you didn't compile it in:
> >>>
> >>> rsyslogd: could not load module '/usr/local/lib/rsyslog/ommail.so',
> >>> dlopen: /usr/local/lib/rsyslog/ommail.so: cannot open shared object
> >>> file: No such file or directory
> >>>
> >>> To fix this, you'll need to reinstall. This time, when you run
> >>> ./configure be sure to include --enable-mail. make install clean and
> >>> you should be good to go...
> >>>
> >>> -HKS
> >>>
> >>> On Thu, Jul 24, 2008 at 5:01 PM, (private) HKS <hks.private at gmail.com
> >>> >
> >>> wrote:
> >>>> A few things to try:
> >>>>
> >>>> - You load ommail.so twice, once at the top and once right above
> >>> your
> >>>> $ActionMail... lines. I don't think this will break it, but it's
> >>>> unnecessary - delete the second one.
> >>>>
> >>>> - $ActionExecOnlyOnceEveryInterval 21600 means that it's only going
> >>>> to attempt sending a message once every 6 hours. For testing
> >>> purposes,
> >>>> this will be obnoxious. Until you're ready for production, change
> >>>> it
> >>>> to:
> >>>> $ActionExecOnlyOnceEveryInterval 30
> >>>>
> >>>> - Send everything to make sure you're not missing it based on
> >>>> something in the property-replacer/templates/whatever. Replace "if
> >>>> $msg contains 'hard disk fatal failure' then :ommail:;mailBody"
> >> with:
> >>>> *.* :ommail:;mailBody
> >>>>
> >>>> Try again. Try logging a few messages from the localhost first
> >>>> (with
> >>>> RHEL you can just run "logger test" to log a user.notice message
> >> with
> >>>> content "test") and see if you get them.
> >>>>
> >>>> If you don't, check the mail logs on your mail server to see if it
> >>>> ever received the message. If not, it's time to break out tcpdump
> >> and
> >>>> see if the packets are ever being generated.
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> -HKS
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 24, 2008 at 3:50 PM, Goutos, Kevin (DOB)
> >>>> <Kevin.Goutos at budget.state.ny.us> wrote:
> >>>>> Hello all,
> >>>>>
> >>>>> First off, I am not very Linux savvy. I don't have a lot of
> >>> experience
> >>>>> other then a basic course. This is probably way past my knowledge,
> >>> but I
> >>>>> really need to get it done. Appreciate any help you guys have to
> >>> offer.
> >>>>>
> >>>>> I am working on a Red Hat Enterprise 4 box and I am running the
> >>> latest
> >>>>> edition of rsyslog. I currently have rsyslog configured to receive
> >>>>> messages remotely via UDP. I am trying to set it up so that it
> >>>>> will
> >>> send
> >>>>> out E-mail messages to the system Admin's based on the severity
> >>> level of
> >>>>> the log files I am receiving. I would like it so that any device
> >>> that
> >>>>> sends a log with a critical, alert, or emergency level facility
> >> will
> >>>>> send out an e-mail to a specific address.
> >>>>>
> >>>>> Here is my rsyslog.conf file. I used the sample code from Rainer
> >>>>> Gerhards configuration page. I tried sending a test syslog with
> >>> 'hard
> >>>>> disk fatal failure' in it, but it is not sending out mail. Also
> >>> tried
> >>>>> without the templates below thinking it would just send me an
> >>>>> email
> >>> for
> >>>>> every syslog that I received, but it doesn't appear to be sending
> >>> mail.
> >>>>> Any thoughts on what I am doing wrong. I'm sure there is a lot I
> >>> need to
> >>>>> do, so please let me know. Thanks!
> >>>>>
> >>>>> $template mailSubject,"disk problem on %hostname%"
> >>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"'
> >>>>>
> >>>>> I edited out the 3 lines below in the code for security reasons..
> >>>>> $ActionMailSMTPServer <ip of smtp server>
> >>>>> $ActionMailFrom <from address>
> >>>>> $ActionMailTo <my email>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Here is my code from rsyslog.conf below.
> >>>>>
> >>>>>
> >>>>>
> >>>>> # if you experience problems, check
> >>>>> # http://www.rsyslog.com/troubleshoot for assistance
> >>>>>
> >>>>> # rsyslog v3: load input modules
> >>>>> # If you do not load inputs, nothing happens!
> >>>>> # You may need to set the module load path if modules are not
> >> found.
> >>>>>
> >>>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>>> $ModLoad imuxsock.so # provides support for local system logging
> >>> (e.g.
> >>>>> via logger command)
> >>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>> $ModLoad ommail
> >>>>>
> >>>>> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated%
> >>>>> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
> >>>>>
> >>>>> # Log all kernel messages to the console.
> >>>>> # Logging much else clutters up the screen.
> >>>>> #kern.*
> >> /dev/console
> >>>>>
> >>>>> # Log anything (except mail) of level info or higher.
> >>>>> # Don't log private authentication messages!
> >>>>> *.info;mail.none;authpriv.none;cron.none
> >>>>> -/var/log/messages
> >>>>>
> >>>>> # The authpriv file has restricted access.
> >>>>> authpriv.*
> >>> /var/log/secure
> >>>>>
> >>>>> # Log all the mail messages in one place.
> >>>>> mail.*
> >>>>> -/var/log/maillog
> >>>>>
> >>>>> # Log cron stuff
> >>>>> cron.* -
> >>> /var/log/cron
> >>>>>
> >>>>> # Everybody gets emergency messages
> >>>>> *.emerg *
> >>>>>
> >>>>> # Save news errors of level crit and higher in a special file.
> >>>>> uucp,news.crit
> >>>>> -/var/log/spooler
> >>>>>
> >>>>> # Save boot messages also to boot.log
> >>>>> local7.*
> >>>>> /var/log/boot.log
> >>>>>
> >>>>> #Catch all incoming syslog messages
> >>>>> *.*
> >>>>> /var/log/catchall;TraditionalFormatWithPRI
> >>>>>
> >>>>> # Remote Logging (we use TCP for reliable delivery)
> >>>>> # An on-disk queue is created for this action. If the remote host
> >> is
> >>>>> # down, messages are spooled to disk and sent when it is up again.
> >>>>> $WorkDirectory /rsyslog/spool # where to place spool files
> >>>>> $ActionQueueFileName uniqName # unique name prefix for spool files
> >>>>> $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> >>>>> possible)
> >>>>> $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> >>>>> $ActionQueueType LinkedList # run asynchronously
> >>>>> $ActionResumeRetryCount -1 # infinite retries if host is down
> >>>>> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> >>>>> optional
> >>>>> *.* @10.57.106.140:514
> >>>>>
> >>>>> $ModLoad ommail
> >>>>> $ActionMailSMTPServer <ip of smtp server>
> >>>>> $ActionMailFrom <from address>
> >>>>> $ActionMailTo <my email>
> >>>>> $template mailSubject,"disk problem on %hostname%"
> >>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
> >>>>> $ActionMailSubject mailSubject
> >>>>> # make sure we receive a mail only once in six
> >>>>> # hours (21,600 seconds ;))
> >>>>> $ActionExecOnlyOnceEveryInterval 21600
> >>>>> # the if ... then ... mailBody mus be on one line!
> >>>>> if $msg contains 'hard disk fatal failure' then :ommail:;mailBody
> >>>>>
> >>>>>
> >>>>> # ######### Receiving Messages from Remote Hosts ##########
> >>>>> # TCP Syslog Server:
> >>>>> # provides TCP syslog reception and GSS-API (if compiled to
> >>>>> support
> >>> it)
> >>>>> $ModLoad imtcp.so # load module
> >>>>> $InputTCPServerRun 514 # start up TCP listener at port 514
> >>>>>
> >>>>> # UDP Syslog Server:
> >>>>> $ModLoad imudp.so # provides UDP syslog reception
> >>>>> $UDPServerRun 514 # start a UDP syslog server at standard port
> >>>>> --------------------------------------------------------
> >>>>> This e-mail, including any attachments, may be confidential,
> >>> privileged or otherwise legally protected. If you have received this
> >> e-
> >>> mail in error, or from someone who was not authorized to send it to
> >>> you, do not disseminate, copy or otherwise use this e-mail or its
> >>> attachments. Please notify the sender immediately if you have
> >>> received
> >>> this e-mail by mistake, and delete it from your system.
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> ########################################################################
> Ryo Fujita <rfujita at redhat.com>
> Senior Solution Architect, RHCE
> Red Hat K.K.
> TEL +81-3-5798-8500
> FAX +81-3-5798-8599
> Ebisu Neonato 8F
> 4-1-18 Ebisu, Shibuya-ku,
> Tokyo Japan 1500013
> ########################################################################
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

Responses inline.

> I totally agree, but I think the situation is even worse for a syslogd.
> Because it *is* the usual error reporting system. I think we are a bit
> in the same situation (if you allow that analogy) like HIV which is
> attacking the human immune system. We have a problem in the
> troubleshooting system itself, and it may even prevent us from conveying
> troubleshooting information :(

That's a good way of putting it. This is one of the reasons I think
it's so important to print errors to STDERR - if the logging
mechanisms are broken for some reason, then at least they're going
somewhere. And I sincerely hope that users have the sense to start it
from the command line the first time rather than sticking it into
their system startup scripts and rebooting.


>> It's happened a few times that I start
>> rsyslogd at the command line, and yet there's nothing in the process
>> table, and nothing dumped to my logs.
>
> Could you elaborate a bit about above sentence - I have to admit I do
> not fully get you...

I've messed up my configs bad enough a couple times that when I start
rsyslogd, nothing apparently happens. No errors to STDERR, no errors
in my logs, and the process isn't running. Rewriting the config or
running with -dn is the solution.



>> 2 - Configuration and other runtime errors should be printed to STDERR
>
> Does that really help? If so, that would be extremely easy to add. But
> does somebody see them?

Absolutely. The first time I configure any daemon, I do it in a test
environment and run it by hand until I have the configuration acting
exactly as I want. Most users take the "run-by-hand" precaution even
if they don't have the luxury of a test setup. These will also pop up
when a user tries to restart the daemon after making a config change.
I'm not sure if they'll be visible when a HUP signal is sent, but that
would make me awfully happy.


>> 3 - By default, rsyslogd should log all of its own errors (fatal or
>> not) to /var/log/rsyslog.log. This should also be user-configurable
>
> This default comes into the way of some distros: they use different
> pathes. I really don't like the idea of embedding any information about
> the file system into the binary. Just think about embedded systems...

Good point.


>> 1 - If rsyslog doesn't understand your config file, obviously it won't
>> be doing what you intend it to. There's little benefit in running with
>> a horked configuration, but if you don't test thoroughly, the cost
>> could be devastating. Some users won't know they're not logging things
>> until they need them - then, if they keep their jobs, they probably
>> won't keep rsyslog ("This software doesn't even log ssh attempts!").
>
> That's quite controversal and and does not match rsyslog's root
> philosophy. Let me explain:
>
> The syslogd is an extremely important piece of software. If it is not
> functioning,
>
> a) vital data can be lost
> b) the system as whole may hang (due to the fact that the log
> socket gets filled up and so other processes block on it)
>
> As such, rsyslogd is written in a sense that it tries to continue to
> work under almost all circumstances. It even tries to survive a
> system-wide low memory condition, though I am not 100% convinced it will
> manage to do that in all cases. At least it is coded careful to survive
> as much as possible.
>
> Now let's assume that a configuration error happens that prevents
> rsyslogd from executing an otherwise perfect configuration. Should we
> actually stop it from running? And live with the results? Or should we
> do whatever we can do and do that as good as possible? What, if it is
> not a configuration issue but a hard disk failure or an admin error that
> make part of the config unavailable (e.g. not able to load module, not
> able to write file)? Assuming we can carry out other actions - should we
> really abort? This may even come at the price that we can NOT warn users
> (via wall) of a serious problem. And what with warnings? Things that
> look wrong, but may be OK. Reason to abort? Or continue run? Tough
> decision. So I still think the right thing to do is to run as long as
> there is limited sense in it, because everything else would actually be
> worse. Please note that rsyslogd even comes with a hardcoded emergency
> configuration which is used in case the real configuration file can not
> be obtained.
>
> What I could add, however, is an option to make it stop on any config
> error (not warning). However, I fear that, if turned on, the results can
> be fatal in some cases...

You make a lot of good points here. Does the hardcoded emergency
configuration that rsyslogd uses actually log? I only ask because it's
easy (if you fail to read the documentation) to write a config file
that doesn't do anything. Would it make sense for rsyslog to determine
whether the passed configuration actually asks it to do anything, and
complaining if not?

Printing errors to STDERR would help with this. Another possibility
might be adding a flag (-n is typical for a lot of systems, but that's
already in use) that will tell rsyslog to parse the configuration,
spit out any problems, and quit. That way, administrators can modify a
config file on a live system and verify it's syntax without hosing the
actual rsyslogd process.



>> 3 - For those more subtle problems, and things you might miss upon
>> bootup. This also gives you a simple, easily accessible debugging
>> source with minimal security implications and additional code and no
>> required user configuration. I'm not sure how the code is written, but
>> I'd prefer to see this happen even if the configuration file can't be
>> read. Perhaps an $RsyslogErrors directive that, by default, references
>> a different module that just dumps directly to a file? Users can
>> configure it to put rsyslog errors through the actual logging engine
>> and then manage it with typical syslog selector/action lines.
>
> All internally-generated messages are run through a specific interface.
> For the "diag plugin", I will define a hook, where a diag server can
> register to receive these messages. Maybe a good compromise would be to
> add this $RsyslogErrors directive and make it point to a file. So,
> still, we do not necessarily have it, but package maintainers would
> hopefully include it and point it to a specific file (what still means I
> do not necessarily know where to look). A problem is source tarball
> install, in which case the directive may not be set. But with a proper
> sample rsyslog.conf...

I think a locally installed sample rsyslog.conf is a solid workaround
for the "different distros, different locations" problem.

>
>> The HTTP server is an interesting idea, but not something I would make
>> use of. The potential security problems are enormous, and it would
>> introduce too many additional factors to the debugging process: is the
>> firewall right? Was there already a process listening on that port? Is
>> the user's configuration file correct? These are the kinds of things
>> you already have to deal with on rsyslog itself - why force the
>> debugging process to go through them again?
>
> I see (and share) much of your argument. This is why I stayed away for
> such a long time. But if you think a bit more about it, it has *very
> interesting* capabilities. Eg. I may also use it to gather some
> real-time view of internal state, emit commands like HUP and do a number
> of other nice things. Again, security is problematic.
>
> In any case, I'll give it a try. I have started to write a small plugin
> which will dump the error messages within 10 minutes after startup. It
> will not be a real http server but rather a listener that response to
> anyone who connects to it (so that nc can be used as a client ;)). This
> is very rough, but it has two side-effects:
>
> a) it provides something to actually experiment with (and to extend if
> we decide to keep that route)
> b) it helps create the plumbing inside the internal interface that could
> also be used by any other method, so the potential waste of time is
> limited

It sounds interesting, and I agree that there's a lot of potential
there. Among them, the possibility of adding an XML-RPC API to allow
for greater interface flexibility...but frankly, that scares the hell
out of me. ;) I'll wait to see what you come up with.

-HKS

inline...

On Fri, 2008-07-25 at 11:44 -0400, (private) HKS wrote:
> Responses inline.
>
> > I totally agree, but I think the situation is even worse for a syslogd.
> > Because it *is* the usual error reporting system. I think we are a bit
> > in the same situation (if you allow that analogy) like HIV which is
> > attacking the human immune system. We have a problem in the
> > troubleshooting system itself, and it may even prevent us from conveying
> > troubleshooting information :(
>
> That's a good way of putting it. This is one of the reasons I think
> it's so important to print errors to STDERR - if the logging
> mechanisms are broken for some reason, then at least they're going
> somewhere. And I sincerely hope that users have the sense to start it
> from the command line the first time rather than sticking it into
> their system startup scripts and rebooting.

If I look at some of the support calls I see, I doubt that ;) The
problem is that there are a lot of novices who have never ever done any
real system administration. But as the rsyslog homepage says: "Its
advanced features make it suitable for enterprise-class, encryption
protected syslog relay chains while at the same time being very easy to
setup for the novice user." Look at the end of the sentence. In any
case, I see the utility of what you say, if not in the novice case, then
with more experienced users.

> >> It's happened a few times that I start
> >> rsyslogd at the command line, and yet there's nothing in the process
> >> table, and nothing dumped to my logs.
> >
> > Could you elaborate a bit about above sentence - I have to admit I do
> > not fully get you...
>
> I've messed up my configs bad enough a couple times that when I start
> rsyslogd, nothing apparently happens. No errors to STDERR, no errors
> in my logs, and the process isn't running. Rewriting the config or
> running with -dn is the solution.

Ah, ok, now I understand. Thanks.

>
>
>
> >> 2 - Configuration and other runtime errors should be printed to STDERR
> >
> > Does that really help? If so, that would be extremely easy to add. But
> > does somebody see them?
>
> Absolutely. The first time I configure any daemon, I do it in a test
> environment and run it by hand until I have the configuration acting
> exactly as I want.

I understand. That makes a lot of sense. As I said, it should be easy to
implement. I'll once again shuffle priorities a bit and make that the
top one. Should not take too long to implement. I'll probably have a
switch to disable it, but being enabled by default.

> Most users take the "run-by-hand" precaution even
> if they don't have the luxury of a test setup. These will also pop up
> when a user tries to restart the daemon after making a config change.
> I'm not sure if they'll be visible when a HUP signal is sent, but that
> would make me awfully happy.

So expect to be happy some time next week ;)

>
>
> >> 3 - By default, rsyslogd should log all of its own errors (fatal or
> >> not) to /var/log/rsyslog.log. This should also be user-configurable
> >
> > This default comes into the way of some distros: they use different
> > pathes. I really don't like the idea of embedding any information about
> > the file system into the binary. Just think about embedded systems...
>
> Good point.
>
>
> >> 1 - If rsyslog doesn't understand your config file, obviously it won't
> >> be doing what you intend it to. There's little benefit in running with
> >> a horked configuration, but if you don't test thoroughly, the cost
> >> could be devastating. Some users won't know they're not logging things
> >> until they need them - then, if they keep their jobs, they probably
> >> won't keep rsyslog ("This software doesn't even log ssh attempts!").
> >
> > That's quite controversal and and does not match rsyslog's root
> > philosophy. Let me explain:
> >
> > The syslogd is an extremely important piece of software. If it is not
> > functioning,
> >
> > a) vital data can be lost
> > b) the system as whole may hang (due to the fact that the log
> > socket gets filled up and so other processes block on it)
> >
> > As such, rsyslogd is written in a sense that it tries to continue to
> > work under almost all circumstances. It even tries to survive a
> > system-wide low memory condition, though I am not 100% convinced it will
> > manage to do that in all cases. At least it is coded careful to survive
> > as much as possible.
> >
> > Now let's assume that a configuration error happens that prevents
> > rsyslogd from executing an otherwise perfect configuration. Should we
> > actually stop it from running? And live with the results? Or should we
> > do whatever we can do and do that as good as possible? What, if it is
> > not a configuration issue but a hard disk failure or an admin error that
> > make part of the config unavailable (e.g. not able to load module, not
> > able to write file)? Assuming we can carry out other actions - should we
> > really abort? This may even come at the price that we can NOT warn users
> > (via wall) of a serious problem. And what with warnings? Things that
> > look wrong, but may be OK. Reason to abort? Or continue run? Tough
> > decision. So I still think the right thing to do is to run as long as
> > there is limited sense in it, because everything else would actually be
> > worse. Please note that rsyslogd even comes with a hardcoded emergency
> > configuration which is used in case the real configuration file can not
> > be obtained.
> >
> > What I could add, however, is an option to make it stop on any config
> > error (not warning). However, I fear that, if turned on, the results can
> > be fatal in some cases...
>
> You make a lot of good points here. Does the hardcoded emergency
> configuration that rsyslogd uses actually log?

Yes, but in a very limited sense. It puts *.err onto the console, does a
wall for *.panic and writes everything else to the controlling tty. I
have to admit that this code isn't awfully well tested, usually only
when something is done at the config code (and sometimes not even
then...).

> I only ask because it's
> easy (if you fail to read the documentation) to write a config file
> that doesn't do anything. Would it make sense for rsyslog to determine
> whether the passed configuration actually asks it to do anything, and
> complaining if not?

That's an interesting idea. I am not sure if this is already checked. I
can at least check if there is no action at all, which means there is no
useful work to be done. The question remains, as usual, where to
complain to. But that could be another situation where the hardcoded
config could be activated.

I have also thought about an implicit write user, with the user being
root by default (and being overwritable via command line and config
file).

>
> Printing errors to STDERR would help with this. Another possibility
> might be adding a flag (-n is typical for a lot of systems, but that's
> already in use) that will tell rsyslog to parse the configuration,
> spit out any problems, and quit. That way, administrators can modify a
> config file on a live system and verify it's syntax without hosing the
> actual rsyslogd process.

That's also a good idea. Rsyslog is quite modular (not yet as much as I
would like it to be, but well enough). So I could even generate a
dedicated config check tool. That would not touch any running
configuration at all. But maybe the switch is easier ;) I need to think
on how this affects the testbed I have begun to work on. Looks like this
could also drive some automated tests.

>
>
>
> >> 3 - For those more subtle problems, and things you might miss upon
> >> bootup. This also gives you a simple, easily accessible debugging
> >> source with minimal security implications and additional code and no
> >> required user configuration. I'm not sure how the code is written, but
> >> I'd prefer to see this happen even if the configuration file can't be
> >> read. Perhaps an $RsyslogErrors directive that, by default, references
> >> a different module that just dumps directly to a file? Users can
> >> configure it to put rsyslog errors through the actual logging engine
> >> and then manage it with typical syslog selector/action lines.
> >
> > All internally-generated messages are run through a specific interface.
> > For the "diag plugin", I will define a hook, where a diag server can
> > register to receive these messages. Maybe a good compromise would be to
> > add this $RsyslogErrors directive and make it point to a file. So,
> > still, we do not necessarily have it, but package maintainers would
> > hopefully include it and point it to a specific file (what still means I
> > do not necessarily know where to look). A problem is source tarball
> > install, in which case the directive may not be set. But with a proper
> > sample rsyslog.conf...
>
> I think a locally installed sample rsyslog.conf is a solid workaround
> for the "different distros, different locations" problem.
>
> >
> >> The HTTP server is an interesting idea, but not something I would make
> >> use of. The potential security problems are enormous, and it would
> >> introduce too many additional factors to the debugging process: is the
> >> firewall right? Was there already a process listening on that port? Is
> >> the user's configuration file correct? These are the kinds of things
> >> you already have to deal with on rsyslog itself - why force the
> >> debugging process to go through them again?
> >
> > I see (and share) much of your argument. This is why I stayed away for
> > such a long time. But if you think a bit more about it, it has *very
> > interesting* capabilities. Eg. I may also use it to gather some
> > real-time view of internal state, emit commands like HUP and do a number
> > of other nice things. Again, security is problematic.
> >
> > In any case, I'll give it a try. I have started to write a small plugin
> > which will dump the error messages within 10 minutes after startup. It
> > will not be a real http server but rather a listener that response to
> > anyone who connects to it (so that nc can be used as a client ;)). This
> > is very rough, but it has two side-effects:
> >
> > a) it provides something to actually experiment with (and to extend if
> > we decide to keep that route)
> > b) it helps create the plumbing inside the internal interface that could
> > also be used by any other method, so the potential waste of time is
> > limited
>
> It sounds interesting, and I agree that there's a lot of potential
> there. Among them, the possibility of adding an XML-RPC API to allow
> for greater interface flexibility...but frankly, that scares the hell
> out of me. ;) I'll wait to see what you come up with.

hehe... We'll see, but I'd say its faaar down the road. One interesting
aspect, though, could be a realtime view of syslog data as it is flowing
through the system. That could even be integrated into phpLogCon.

In any case, I do NOT want to add much (http) application logic /
presentation to such a system. But a few html (or xml ;)) pages
generated at appropriate times could be really useful. Just think about
things like a "granular HUP", like being able to tell to close some
files specifically etc etc. But that's too far advanced. For me
personally, it would be quite useful to get access to state variables
(like mem utilization, object instances, error count, frame rate) of the
running system - not under a debugger but in a real live system. From
the developers (and troubleshooter's point of view), that would be very
interesting. I could even hold the last 2000 or so messages of the debug
log in memory, plus the initial first 1000. Together, I guess, they can
solve over 90% of all things I have used debug logs for. But this can
only be done if I have dynamic runtime access to the live instance. And
this obviously also means there is a big security issue ;)

Please keep the thoughts flowing, we make good progess :)
Rainer
>
> -HKS
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

I don't have much more to add to this discussion - I think we're
pretty much on the same page. I'm looking forward to seeing the
results of your work.

-HKS


On Fri, Jul 25, 2008 at 12:07 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> <huge snip>
>
> Please keep the thoughts flowing, we make good progess :)
> Rainer

I have implemented much yesterday. It is available via git right now:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=d2feb7063e73938c05b
76862ea2e211cc09b30fe

I will create a testbed for it today and hopefully be able to release it
some time this afternoon. Unfortunately, it's a busy day...

Feedback is appreciated.
Rainer


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of (private) HKS
> Sent: Monday, July 28, 2008 5:54 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog error messages
>
> I don't have much more to add to this discussion - I think we're
> pretty much on the same page. I'm looking forward to seeing the
> results of your work.
>
> -HKS
>
>
> On Fri, Jul 25, 2008 at 12:07 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > <huge snip>
> >
> > Please keep the thoughts flowing, we make good progess :)
> > Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

OK, I have now a preview tarball available. I'd appreciate if you could
give it a try.

Use -N1 on the command line to make it do a config check, only.

It is available here:

http://download.rsyslog.com/download/rsyslog-3.21.1.tar.gz
md5sum:

I will replace that file once the "official" 3.21.1 is out.

I have one problem with autotools: a "make distcheck" fails, as far as I
can see because rsyslogd is not built inside the tools subdirectory. If
someone has a suggestion for a fix (or what could cause the problem), I
would appreciate that. Actually this "make distcheck" problem is what
made me hold back the official release.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, July 29, 2008 10:11 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog error messages
>
> I have implemented much yesterday. It is available via git right now:
>
>
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=d2feb7063e73938c05
> b
> 76862ea2e211cc09b30fe
>
> I will create a testbed for it today and hopefully be able to release
> it
> some time this afternoon. Unfortunately, it's a busy day...
>
> Feedback is appreciated.
> Rainer
>
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of (private) HKS
> > Sent: Monday, July 28, 2008 5:54 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog error messages
> >
> > I don't have much more to add to this discussion - I think we're
> > pretty much on the same page. I'm looking forward to seeing the
> > results of your work.
> >
> > -HKS
> >
> >
> > On Fri, Jul 25, 2008 at 12:07 PM, Rainer Gerhards
> > <rgerhards at hq.adiscon.com> wrote:
> > > <huge snip>
> > >
> > > Please keep the thoughts flowing, we make good progess :)
> > > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

It would have been better if I had actually included the md5sum ;). It
is b9ca041c00d093981e6c6e35d360e6e9

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, July 29, 2008 3:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog error messages
>
> OK, I have now a preview tarball available. I'd appreciate if you
could
> give it a try.
>
> Use -N1 on the command line to make it do a config check, only.
>
> It is available here:
>
> http://download.rsyslog.com/download/rsyslog-3.21.1.tar.gz
> md5sum:
>
> I will replace that file once the "official" 3.21.1 is out.
>
> I have one problem with autotools: a "make distcheck" fails, as far as
> I
> can see because rsyslogd is not built inside the tools subdirectory.
If
> someone has a suggestion for a fix (or what could cause the problem),
I
> would appreciate that. Actually this "make distcheck" problem is what
> made me hold back the official release.
>
> Thanks,
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: Tuesday, July 29, 2008 10:11 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog error messages
> >
> > I have implemented much yesterday. It is available via git right
now:
> >
> >
>
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=d2feb7063e73938c05
> > b
> > 76862ea2e211cc09b30fe
> >
> > I will create a testbed for it today and hopefully be able to
release
> > it
> > some time this afternoon. Unfortunately, it's a busy day...
> >
> > Feedback is appreciated.
> > Rainer
> >
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of (private) HKS
> > > Sent: Monday, July 28, 2008 5:54 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog error messages
> > >
> > > I don't have much more to add to this discussion - I think we're
> > > pretty much on the same page. I'm looking forward to seeing the
> > > results of your work.
> > >
> > > -HKS
> > >
> > >
> > > On Fri, Jul 25, 2008 at 12:07 PM, Rainer Gerhards
> > > <rgerhards at hq.adiscon.com> wrote:
> > > > <huge snip>
> > > >
> > > > Please keep the thoughts flowing, we make good progess :)
> > > > Rainer
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

I'll test this as soon as I can, but it may be a couple days...

-HKS

On Tue, Jul 29, 2008 at 4:10 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> I have implemented much yesterday. It is available via git right now:
>
> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=d2feb7063e73938c05b
> 76862ea2e211cc09b30fe
>
> I will create a testbed for it today and hopefully be able to release it
> some time this afternoon. Unfortunately, it's a busy day...
>
> Feedback is appreciated.
> Rainer
>
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of (private) HKS
>> Sent: Monday, July 28, 2008 5:54 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] rsyslog error messages
>>
>> I don't have much more to add to this discussion - I think we're
>> pretty much on the same page. I'm looking forward to seeing the
>> results of your work.
>>
>> -HKS
>>
>>
>> On Fri, Jul 25, 2008 at 12:07 PM, Rainer Gerhards
>> <rgerhards at hq.adiscon.com> wrote:
>> > <huge snip>
>> >
>> > Please keep the thoughts flowing, we make good progess :)
>> > Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>