Mailing List Archive

Seth,

I think you have some very good ideas and I know I would benefit from
those additional features in rsyslog. FWIW, I've been running rsyslog
for a couple of years and it works flawlessly.

Cheers,

-Dusty


>>> seth vidal <skvidal at fedoraproject.org> 07/16/07 5:41 PM >>>
Hi folks,
Fedora 8 will most likely be including rsyslog as the default syslog
daemon. As such I've been experimenting with it to see how well it works
to replace some of my former uses of syslog-ng. I like how small rsyslog
is and it looks pretty straightforward.

I have a couple of usage issues and some other random questions:


1. Can rsyslog automatically create subdirs for specific log outputs?
For example:
$template myvarname, "/var/log/hosts/%HOSTNAME%/messages"

right now I have to manually make /var/log/hosts/%HOSTNAME%/ as a dir
before any logs will be outputted? Can there be an option to do that
automatically?

2. related to 1 is there a way to set specific file ownerships not just
modes on the logs? I'd like to set the logs to be owned by
root:log_readers and to be 640, along with all directories that it
autocreates. That way no one else can look at the logs and I don't have
to do anything hurky like 'sg' before running rsyslog.

3. I understand the desire to remain config file format compatible with
sysklogd, however it seems like with any more logic being added into the
config you'll end up with progressively more difficult config files. *.*
@@hostname:port is pretty baroque looking as-is. Would it be reasonable
to have a newer config file format/language and do:
PSEUDOCODE
if [ -f /etc/$new-style-format.conf ]; then
conffile=/etc/$new-style-format.conf
else:
conffile=/etc/syslog.conf
fi
?

That way you retain backward compat to the old config file format from
sysklogd and you provide a migration path to a nicer file formats for
the future?

Thanks!
-sv


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Hi Sv,

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of seth vidal
> Sent: Tuesday, July 17, 2007 12:42 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] first time using rsyslog - some questions
>
> Hi folks,
> Fedora 8 will most likely be including rsyslog as the default syslog
> daemon. As such I've been experimenting with it to see how well it
> works
> to replace some of my former uses of syslog-ng. I like how small
> rsyslog
> is and it looks pretty straightforward.

Great to have someone experienced with syslog-ng on the list. I really
like to make rsyslog the best syslogd ever and it is most interesting to
learn what syslog-ng users find the most important features. In my point
of view, we have some things syslog-ng can not do, but others where
syslog-ng is ahead. I'd like to change that, but in order of importance
to the user ;)

>
> I have a couple of usage issues and some other random questions:
>
>
> 1. Can rsyslog automatically create subdirs for specific log outputs?
> For example:
> $template myvarname, "/var/log/hosts/%HOSTNAME%/messages"
>
> right now I have to manually make /var/log/hosts/%HOSTNAME%/ as a dir
> before any logs will be outputted? Can there be an option to do that
> automatically?

It's on the todo list, most probably to be implemented this week. All
interested please see:

https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_
id=123448&atid=696552

>
> 2. related to 1 is there a way to set specific file ownerships not
just
> modes on the logs? I'd like to set the logs to be owned by
> root:log_readers and to be 640, along with all directories that it
> autocreates. That way no one else can look at the logs and I don't
have
> to do anything hurky like 'sg' before running rsyslog.

Not yet on the feature list, but definitely interesting. I've filed a
new feature request:

https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_
id=123448&atid=696555


>
> 3. I understand the desire to remain config file format compatible
with
> sysklogd, however it seems like with any more logic being added into
> the
> config you'll end up with progressively more difficult config files.
> *.*
> @@hostname:port is pretty baroque looking as-is. Would it be
reasonable
> to have a newer config file format/language and do:
> PSEUDOCODE
> if [ -f /etc/$new-style-format.conf ]; then
> conffile=/etc/$new-style-format.conf
> else:
> conffile=/etc/syslog.conf
> fi
> ?
>
> That way you retain backward compat to the old config file format from
> sysklogd and you provide a migration path to a nicer file formats for
> the future?

I've already some ideas for a new log file format. I won't elaborate
right now, because I'd like to iron out 1.16.1 this morning and that's a
priority. However, the log file will still be quite different from
syslog-ng, following a more simple concept (at least I think so). I
searched the web a lot and found numerous questions on the syslog-ng log
file format. So I think for the next version of rsyslogs format, it
should be very easy to grasp yet powerful. And, yes, I agree the current
way of doing things is not optimal. OK, already written more than I
wanted: my plan is to get feature-complete, then restructure the code to
be finally modular (I inherited a non-modular design form syslogd) and
then define output and input plugins. In the course of that, you'll see
config file changes. I am thinking along these lines:

(for a listener)

$protocol tcp
$port 514
$RepeatedMsgReduction on # already in 1.16.1
$EscapeControlCharactersOnReceive off # already in 1.16.1
$listen 127.0.0.1

Similar things for the output side...

Looking forward to more comments.

Rainer
>
> Thanks!
> -sv
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote:

> Great to have someone experienced with syslog-ng on the list. I really
> like to make rsyslog the best syslogd ever and it is most interesting to
> learn what syslog-ng users find the most important features. In my point
> of view, we have some things syslog-ng can not do, but others where
> syslog-ng is ahead. I'd like to change that, but in order of importance
> to the user ;)

Great! Featurewise, right now, rsyslog has a lot of the pieces in the
right place to be a great syslog server. My primary uses of syslog-ng
was:
1. large scale central file server (500ish systems logging everything
centrally)
2. ssl-encrypting/tcp syslog server for communicating over wans.
3. invoking sec for specialized event behavior



> It's on the todo list, most probably to be implemented this week. All
> interested please see:
>
> https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_
> id=123448&atid=696552
>

Great to hear! Thanks.

One of the use cases for me is:

/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages
/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log
/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log
etc

so that for log cleanup I can simply do:
tmpwatch -f 720 /var/log/hosts/

and that removes all logs older than 30 days.

No need to run logrotate, no need to hup the server. It also makes logs
rsyncable b/c the filenames/path change as they progress. This lets me
rsync my logs to another box, process them for results/reports and not
disrupt or bog down my primary syslog server.

So if automatically-creating-dirs is made possible then I can do the
above on my central syslog server.

> > 2. related to 1 is there a way to set specific file ownerships not
> just
> > modes on the logs? I'd like to set the logs to be owned by
> > root:log_readers and to be 640, along with all directories that it
> > autocreates. That way no one else can look at the logs and I don't
> have
> > to do anything hurky like 'sg' before running rsyslog.
>
> Not yet on the feature list, but definitely interesting. I've filed a
> new feature request:
>
> https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_
> id=123448&atid=696555

great! monitoring that rfe now, thanks.


> I've already some ideas for a new log file format. I won't elaborate
> right now, because I'd like to iron out 1.16.1 this morning and that's a
> priority. However, the log file will still be quite different from
> syslog-ng, following a more simple concept (at least I think so). I
> searched the web a lot and found numerous questions on the syslog-ng log
> file format. So I think for the next version of rsyslogs format, it
> should be very easy to grasp yet powerful. And, yes, I agree the current
> way of doing things is not optimal. OK, already written more than I
> wanted: my plan is to get feature-complete, then restructure the code to
> be finally modular (I inherited a non-modular design form syslogd) and
> then define output and input plugins. In the course of that, you'll see
> config file changes. I am thinking along these lines:
>
> (for a listener)
>
> $protocol tcp
> $port 514
> $RepeatedMsgReduction on # already in 1.16.1
> $EscapeControlCharactersOnReceive off # already in 1.16.1
> $listen 127.0.0.1
>
> Similar things for the output side...
>
> Looking forward to more comments.

What would you think of a config file format that's a bit closer to a
language? So you could do more case-like statements for what to get
logged where? for example:

my_emerg_output = /var/log/mail/emergency
my_generic_output = |/usr/bin/keep_count_of_messages
my_normal_logs = /var/log/mail/normal

if facility == 'mail':
if priority == 'emerg':
output == my_emerg_output
elif priority < 'emerg' and priority > 'debug':
filter_out(host='localhost')
output == my_generic_output, my_normal_logs
else:
output == /dev/null

So this would mean:
if the facility is mail and priority emerg then send it to my mail
emergency file
if the facility is mail and the priority is between emerg and debug
then remove all logs where the sending host is 'localhost' and then send
it to my generic output AND to my normal logs
otherwise, dump the output to /dev/null.


Would something like that be reasonable? Alternatively, can you tell me
a better way to do something like the above with what you have in mind
for the config file? I'm really not wed to any one format - I just like
it when config file examples are easy to put together and parse.


For additional feature sets:
Something that syslog-ng cannot do but I've always wanted a syslog
daemon to do is store-and-forward remote logging and/or failover remote
logging.

ie:
If you're using tcp-based logging from client to server you can
determine that the server is no longer accepting your messages from the
client. In that event, have the client start packing away all of the log
entries it would normally send to the server into a tmpfile some place.
When the server comes back push all of the logs over to it from earliest
to latest. Yes, the logs will be out of order but that's the least of
the worries, at least you have the data remotely. Setting a limit on the
size of the tmpfile makes sense to me and when that limit is hit start
discarding the EARLIER log entries for the later ones.

As an option related to this. Allow a remote log server configuration
option to include multiple hostnames/destinations in decreasing
priority.
So as long as the first one is up, you log to it, when it stops being up
you log to the next one in the list and so on and so on.

Thanks,
-sv

On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote:

> Great to have someone experienced with syslog-ng on the list. I really
> like to make rsyslog the best syslogd ever and it is most interesting to
> learn what syslog-ng users find the most important features. In my point
> of view, we have some things syslog-ng can not do, but others where
> syslog-ng is ahead. I'd like to change that, but in order of importance
> to the user ;)

Great! Featurewise, right now, rsyslog has a lot of the pieces in the
right place to be a great syslog server. My primary uses of syslog-ng
was:
1. large scale central file server (500ish systems logging everything
centrally)
2. ssl-encrypting/tcp syslog server for communicating over wans.
3. invoking sec for specialized event behavior



> It's on the todo list, most probably to be implemented this week. All
> interested please see:
>
> https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_
> id=123448&atid=696552
>

Great to hear! Thanks.

One of the use cases for me is:

/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages
/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log
/var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log
etc

so that for log cleanup I can simply do:
tmpwatch -f 720 /var/log/hosts/

and that removes all logs older than 30 days.

No need to run logrotate, no need to hup the server. It also makes logs
rsyncable b/c the filenames/path change as they progress. This lets me
rsync my logs to another box, process them for results/reports and not
disrupt or bog down my primary syslog server.

So if automatically-creating-dirs is made possible then I can do the
above on my central syslog server.

> > 2. related to 1 is there a way to set specific file ownerships not
> just
> > modes on the logs? I'd like to set the logs to be owned by
> > root:log_readers and to be 640, along with all directories that it
> > autocreates. That way no one else can look at the logs and I don't
> have
> > to do anything hurky like 'sg' before running rsyslog.
>
> Not yet on the feature list, but definitely interesting. I've filed a
> new feature request:
>
> https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_
> id=123448&atid=696555

great! monitoring that rfe now, thanks.


> I've already some ideas for a new log file format. I won't elaborate
> right now, because I'd like to iron out 1.16.1 this morning and that's a
> priority. However, the log file will still be quite different from
> syslog-ng, following a more simple concept (at least I think so). I
> searched the web a lot and found numerous questions on the syslog-ng log
> file format. So I think for the next version of rsyslogs format, it
> should be very easy to grasp yet powerful. And, yes, I agree the current
> way of doing things is not optimal. OK, already written more than I
> wanted: my plan is to get feature-complete, then restructure the code to
> be finally modular (I inherited a non-modular design form syslogd) and
> then define output and input plugins. In the course of that, you'll see
> config file changes. I am thinking along these lines:
>
> (for a listener)
>
> $protocol tcp
> $port 514
> $RepeatedMsgReduction on # already in 1.16.1
> $EscapeControlCharactersOnReceive off # already in 1.16.1
> $listen 127.0.0.1
>
> Similar things for the output side...
>
> Looking forward to more comments.

What would you think of a config file format that's a bit closer to a
language? So you could do more case-like statements for what to get
logged where? for example:

my_emerg_output = /var/log/mail/emergency
my_generic_output = |/usr/bin/keep_count_of_messages
my_normal_logs = /var/log/mail/normal

if facility == 'mail':
if priority == 'emerg':
output == my_emerg_output
elif priority < 'emerg' and priority > 'debug':
filter_out(host='localhost')
output == my_generic_output, my_normal_logs
else:
output == /dev/null

So this would mean:
if the facility is mail and priority emerg then send it to my mail
emergency file
if the facility is mail and the priority is between emerg and debug
then remove all logs where the sending host is 'localhost' and then send
it to my generic output AND to my normal logs
otherwise, dump the output to /dev/null.


Would something like that be reasonable? Alternatively, can you tell me
a better way to do something like the above with what you have in mind
for the config file? I'm really not wed to any one format - I just like
it when config file examples are easy to put together and parse.


For additional feature sets:
Something that syslog-ng cannot do but I've always wanted a syslog
daemon to do is store-and-forward remote logging and/or failover remote
logging.

ie:
If you're using tcp-based logging from client to server you can
determine that the server is no longer accepting your messages from the
client. In that event, have the client start packing away all of the log
entries it would normally send to the server into a tmpfile some place.
When the server comes back push all of the logs over to it from earliest
to latest. Yes, the logs will be out of order but that's the least of
the worries, at least you have the data remotely. Setting a limit on the
size of the tmpfile makes sense to me and when that limit is hit start
discarding the EARLIER log entries for the later ones.

As an option related to this. Allow a remote log server configuration
option to include multiple hostnames/destinations in decreasing
priority.
So as long as the first one is up, you log to it, when it stops being up
you log to the next one in the list and so on and so on.

Thanks,
-sv

Hi,

I finished 1.17.0 (too much went in it for 1.16.1 ;)), so I finally have
time for an intelligent answer ;) See inline below...

On Tue, 2007-07-17 at 08:28 -0400, seth vidal wrote:
> On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote:
>
> > Great to have someone experienced with syslog-ng on the list. I really
> > like to make rsyslog the best syslogd ever and it is most interesting to
> > learn what syslog-ng users find the most important features. In my point
> > of view, we have some things syslog-ng can not do, but others where
> > syslog-ng is ahead. I'd like to change that, but in order of importance
> > to the user ;)
>
> Great! Featurewise, right now, rsyslog has a lot of the pieces in the
> right place to be a great syslog server. My primary uses of syslog-ng
> was:
> 1. large scale central file server (500ish systems logging everything
> centrally)
> 2. ssl-encrypting/tcp syslog server for communicating over wans.
> 3. invoking sec for specialized event behavior
>
That sounds great. I would be really interested in your experience with
1.17.0 in this environment (now is the time to try to break it,
feature-wise its more or less ready for fedora 8 ;)).

>
>
> > It's on the todo list, most probably to be implemented this week. All
> > interested please see:
> >
> > https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_
> > id=123448&atid=696552
> >
>
> Great to hear! Thanks.

I had a good time today, its in.

>
> One of the use cases for me is:
>
> /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages
> /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log
> /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log
> etc
>
This is a sample of what you can do now:

$umask 0000 # let default umask not get into our way
$CreateDirs on
$DynaCacheFileSize 50 # depending on number of hosts
$FileOwner whoever
$FileGroup loggroup
$FileCreateMode 0600
$DirOwner whoever
$DirGroup loggroup
$DirCreateMode 0600
$FailOnChownFailure off # its more important for us that logs be written
$template file-msgs,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY
%/messages
$RepeatedMessageReduction on # or off?
*.* -?file-msgs

It is important to keep in mind that all parameters keep in effect until
they are either overwritten (by a new directive) or
$ResetConfigVariables is specified.

> "
$
> so that for log cleanup I can simply do:
> tmpwatch -f 720 /var/log/hosts/
>
> and that removes all logs older than 30 days.
>
> No need to run logrotate, no need to hup the server. It also makes logs
> rsyncable b/c the filenames/path change as they progress. This lets me
> rsync my logs to another box, process them for results/reports and not
> disrupt or bog down my primary syslog server.
>
> So if automatically-creating-dirs is made possible then I can do the
> above on my central syslog server.

Please check out - it's brand-new code, so probably some things go wrong
at some place (of course, not in lab ;)).

[snip]

> > I've already some ideas for a new log file format. I won't elaborate
> > right now, because I'd like to iron out 1.16.1 this morning and that's a
> > priority. However, the log file will still be quite different from
> > syslog-ng, following a more simple concept (at least I think so). I
> > searched the web a lot and found numerous questions on the syslog-ng log
> > file format. So I think for the next version of rsyslogs format, it
> > should be very easy to grasp yet powerful. And, yes, I agree the current
> > way of doing things is not optimal. OK, already written more than I
> > wanted: my plan is to get feature-complete, then restructure the code to
> > be finally modular (I inherited a non-modular design form syslogd) and
> > then define output and input plugins. In the course of that, you'll see
> > config file changes. I am thinking along these lines:
> >
[snip]

> What would you think of a config file format that's a bit closer to a
> language? So you could do more case-like statements for what to get
> logged where? for example:
>
> my_emerg_output = /var/log/mail/emergency
> my_generic_output = |/usr/bin/keep_count_of_messages
> my_normal_logs = /var/log/mail/normal
>
> if facility == 'mail':
> if priority == 'emerg':
> output == my_emerg_output
> elif priority < 'emerg' and priority > 'debug':
> filter_out(host='localhost')
> output == my_generic_output, my_normal_logs
> else:
> output == /dev/null
>
> So this would mean:
> if the facility is mail and priority emerg then send it to my mail
> emergency file
> if the facility is mail and the priority is between emerg and debug
> then remove all logs where the sending host is 'localhost' and then send
> it to my generic output AND to my normal logs
> otherwise, dump the output to /dev/null.
>
>
> Would something like that be reasonable? Alternatively, can you tell me
> a better way to do something like the above with what you have in mind
> for the config file? I'm really not wed to any one format - I just like
> it when config file examples are easy to put together and parse.
>

I have to admit I have not yet thought about the filtering side. The
modules are input - filter - output. Whatever it is, it won't happen for
2.0.0, which shall be the result of the current development branch in
the not so distant future (let's say 3 to 5 weeks from now). I'll
consider your recommendation when I think about a new way to handle
filters. But no promise ;)

>
> For additional feature sets:
> Something that syslog-ng cannot do but I've always wanted a syslog
> daemon to do is store-and-forward remote logging and/or failover remote
> logging.
>
> ie:
> If you're using tcp-based logging from client to server you can
> determine that the server is no longer accepting your messages from the
> client. In that event, have the client start packing away all of the log
> entries it would normally send to the server into a tmpfile some place.
> When the server comes back push all of the logs over to it from earliest
> to latest. Yes, the logs will be out of order but that's the least of
> the worries, at least you have the data remotely. Setting a limit on the
> size of the tmpfile makes sense to me and when that limit is hit start
> discarding the EARLIER log entries for the later ones.

This *is* on the upcoming feature list. But to implement it, a number of
core changes must be applied first, including a change of the threading
model. I already have a rough idea on most of these things, so I
wouldn't outrule that the whole thing materializes this year (but again,
no promise ;)).
>
> As an option related to this. Allow a remote log server configuration
> option to include multiple hostnames/destinations in decreasing
> priority.
> So as long as the first one is up, you log to it, when it stops being up
> you log to the next one in the list and so on and so on.

That sounds fairly easy. Let's see if I can fit that in without touching
too much code.

I've added a tracker:

http://sourceforge.net/tracker/index.php?func=detail&aid=1755530&group_id=123448&atid=696555

Thanks for the comments,
Rainer

On Tue, 2007-07-17 at 17:38 +0200, Rainer Gerhards wrote:
> Hi,
>
> That sounds great. I would be really interested in your experience with
> 1.17.0 in this environment (now is the time to try to break it,
> feature-wise its more or less ready for fedora 8 ;)).

right now it is doing what it should be doing, except for one thing. if
I have a log defined like this:

$template hostsecure, "/var/log/hosts/%HOSTNAME%/secure/%$NOW%"
authpriv.* -?hostsecure


I'm getting logs from all of the hosts in this path not just ones of %
HOSTNAME%

It seemed to be working before 1.17.0.

> I had a good time today, its in.
>

Thank you for taking care of this so quickly. It is quite helpful.



-sv

> right now it is doing what it should be doing, except for one thing.
if
> I have a log defined like this:
>
> $template hostsecure, "/var/log/hosts/%HOSTNAME%/secure/%$NOW%"
> authpriv.* -?hostsecure
>
>
> I'm getting logs from all of the hosts in this path not just ones of %
> HOSTNAME%
>
> It seemed to be working before 1.17.0.

I have to admit I do not fully understand. There is no hostname
restriction (at least not one I can see from the excerpt), so it should
be processed for all hosts. I am sure I overlook something, please
provide a bit more detail.

>
> > I had a good time today, its in.
> >
>
> Thank you for taking care of this so quickly. It is quite helpful.

The question came at the right time, everything was already set. I can't
guarantee this will be the case the next time ;)

Rainer

Hi sv,

long, long time ago, you wrote:
> As an option related to this. Allow a remote log server configuration
> option to include multiple hostnames/destinations in decreasing
> priority.
> So as long as the first one is up, you log to it, when it stops being
> up
> you log to the next one in the list and so on and so on.

I know you will not like the config file syntax (nor do I, but that's a
different topic ;-]), but I have finally implemented that feature. I
personally think it is quite powerful. Please read my blog for details:

http://rgerhards.blogspot.com/2007/08/finally-ability-to-automatically-s
witch.html

I now go ahead and upgrade a few output drivers. Then it'll work with
MySQL, too. ;)

Thanks again for the suggestion,
Rainer

On Thu, 2007-08-02 at 18:54 +0200, Rainer Gerhards wrote:
> Hi sv,
>
> long, long time ago, you wrote:
> > As an option related to this. Allow a remote log server configuration
> > option to include multiple hostnames/destinations in decreasing
> > priority.
> > So as long as the first one is up, you log to it, when it stops being
> > up
> > you log to the next one in the list and so on and so on.
>
> I know you will not like the config file syntax (nor do I, but that's a
> different topic ;-]), but I have finally implemented that feature. I
> personally think it is quite powerful. Please read my blog for details:
>
> http://rgerhards.blogspot.com/2007/08/finally-ability-to-automatically-s
> witch.html
>
> I now go ahead and upgrade a few output drivers. Then it'll work with
> MySQL, too. ;)

That's great, thanks!

-sv