Mailing List Archive

Separating logs by host?
Is it possible with rsyslog to separate logs based on IP address and
not logging facility?

I have about 10 pieces of equipment that I'd like to get syslog
fromt, but I've run out of log facilities (local1 through local7).
It'd be nice if each of those pieces of equipment could have it's
own log. I can't find anything in the documentation that covers this.

Am I missing it?

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
Looks like doc can need some more guides ;)

Does this help: http://www.rsyslog.com/Article60.phtml

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:30 PM
> To: rsyslog-users
> Subject: [rsyslog] Separating logs by host?
>
> Is it possible with rsyslog to separate logs based on IP address and
> not logging facility?
>
> I have about 10 pieces of equipment that I'd like to get syslog
> fromt, but I've run out of log facilities (local1 through local7).
> It'd be nice if each of those pieces of equipment could have it's
> own log. I can't find anything in the documentation that covers this.
>
> Am I missing it?
>
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
Separating logs by host? [ In reply to ]
Rainer Gerhards wrote:
> Looks like doc can need some more guides ;)
>
> Does this help: http://www.rsyslog.com/Article60.phtml

Excellent... doing this causes all messages to be logged in that
format. Even local messages from the box (cron, mail, authpriv, etc)
to go into that format.

Can I apply that logging statement to a group of IPs? I have a bunch
of DSL equipment that logs. It would be nice if I could break all my
equipment out like:

Routers: /var/log/routers-%$NOW%.log
DSL: /var/log/dsl-%$NOW%.log
Dial-up: /var/log/dial--%$NOW%.log

Each/Any of those groups could have one or many sources. We have
about 13 unique DSL devices that should all probably go in the same
log for simplicity. It would be REALLY cool to be able to say
something like:

192.168.1.1/24:*.* /var/log/dsl-%$NOW%.log
192.168.3.1/24,10.1.1.1/24:*.* /var/log/routers-%$NOW%.log

Am I just dreaming or is that possible?

Also why is the next version of rsyslog (after reading your blog)
going to be version 3 and not 2.0?

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
I need to think about the real question a bit ;) Not an immediate
solution ... but not hopeless either ;)

On v2/v3 - v2 will be the next stable release:
http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:47 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
>
> Rainer Gerhards wrote:
> > Looks like doc can need some more guides ;)
> >
> > Does this help: http://www.rsyslog.com/Article60.phtml
>
> Excellent... doing this causes all messages to be logged in that
> format. Even local messages from the box (cron, mail, authpriv, etc)
> to go into that format.
>
> Can I apply that logging statement to a group of IPs? I have a bunch
> of DSL equipment that logs. It would be nice if I could break all my
> equipment out like:
>
> Routers: /var/log/routers-%$NOW%.log
> DSL: /var/log/dsl-%$NOW%.log
> Dial-up: /var/log/dial--%$NOW%.log
>
> Each/Any of those groups could have one or many sources. We have
> about 13 unique DSL devices that should all probably go in the same
> log for simplicity. It would be REALLY cool to be able to say
> something like:
>
> 192.168.1.1/24:*.* /var/log/dsl-%$NOW%.log
> 192.168.3.1/24,10.1.1.1/24:*.* /var/log/routers-%$NOW%.log
>
> Am I just dreaming or is that possible?
>
> Also why is the next version of rsyslog (after reading your blog)
> going to be version 3 and not 2.0?
>
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
Separating logs by host? [ In reply to ]
Rainer Gerhards wrote:
> I need to think about the real question a bit ;) Not an immediate
> solution ... but not hopeless either ;)
>
> On v2/v3 - v2 will be the next stable release:
> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html

Another good example is that we have several linux mail servers. I
like to aggregate all of their logging to one rsyslog server. They
all log to mail.* on their local machines, and then forward also to
my rsyslog server.

The problem is that shows up in my mail.* on the syslog server, and
thus gets mixed in with the local mail.* from the syslog server
itself. It would be nice to be able to separate those, by IP would
be the easiest.

mail.* /var/log/maillog
1.2.3.4/32:mail.* /var/log/mail-server.log

Or something like that...

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
Scott Baker wrote:
> Rainer Gerhards wrote:
>> I need to think about the real question a bit ;) Not an immediate
>> solution ... but not hopeless either ;)
>>
>> On v2/v3 - v2 will be the next stable release:
>> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html
>
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
>
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
>
> mail.* /var/log/maillog
> 1.2.3.4/32:mail.* /var/log/mail-server.log
>
> Or something like that...

might logging to an (my|pg)sql server solve all your issues?

cheers,
raoul
--
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at
Barawitzkagasse 10/2/2/11 email. office at ipax.at
1190 Wien tel. +43 1 3670030
FN 277995t HG Wien fax. +43 1 3670030 15
____________________________________________________________________
Separating logs by host? [ In reply to ]
Mmhhh... once we have expression support, that will be an easy thing to
do. Currently, I think there are a number of clumpsy work-arounds. We
are around two month away from expressions, at least if all goes well.

I think you can achive something with BDS-style filter blocks
(!progname/-host/+host), something along these lines:

-host
mail.* /var/log/maillog
+hostname
mail.* /var/log/mail-server.log

Of course, this is hostname specific (and that again is not the IP but
what is in the message...)

To get to the IP, I think you can get at least some way with class A,B,C
addresses, but NOT with any other masks. Here is the idea:

$template dsl, /var/log/dsl-%$NOW%.log
$template routers, /var/log/routers-%$NOW%.log
:FROMHOST, startswith, "192.168.1." -?dsl
:FROMHOST, startswith, "192.168.3." -?routers
:FROMHOST, startswith, "10.1.1." -?routers

Common pitfall: be sure to include the trailing dot in the condition to
match. If the rule were

:FROMHOST, startswith, "192.168.1" -?dsl

It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!

With the current engine, there are unfortunately no logical operations
available. So you cannot check for mail facility or whatever else. A
somewhat crude work-around would be to include the facility in the file
name, e.g. by specifying it as follows:

$template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log

That, of course, will result in a file written for each facility, even
those that you are not interested in. A unfortunately do not have a
solution for this now.

With v3 expressions, I envision something along these lines:

If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
"mail" then
writefile "/var/log/dslmail-%$NOW%"

But that is not possible yet. And this is only an idea, not the actual
config file format we will have at that time. In my blog, there are a
number of posts about it, but the bottom line is that it is not yet
designed:

http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html

Doc about current capabilities is here:

http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml

search for "Filter Conditions" on that page.

The properties you need are documented here:

http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
phtml

Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
idea... ;))

As always, feedback is appreciated. I would be most interested to learn
at which final config you arrived.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 1:23 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
>
> Rainer Gerhards wrote:
> > I need to think about the real question a bit ;) Not an immediate
> > solution ... but not hopeless either ;)
> >
> > On v2/v3 - v2 will be the next stable release:
> > http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-
> v3.html
>
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
>
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
>
> mail.* /var/log/maillog
> 1.2.3.4/32:mail.* /var/log/mail-server.log
>
> Or something like that...
>
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
Separating logs by host? [ In reply to ]
Rainer Gerhards wrote:
> Mmhhh... once we have expression support, that will be an easy thing to
> do. Currently, I think there are a number of clumpsy work-arounds. We
> are around two month away from expressions, at least if all goes well.
>
> I think you can achive something with BDS-style filter blocks
> (!progname/-host/+host), something along these lines:
>
> -host
> mail.* /var/log/maillog
> +hostname
> mail.* /var/log/mail-server.log
>
> Of course, this is hostname specific (and that again is not the IP but
> what is in the message...)
>
> To get to the IP, I think you can get at least some way with class A,B,C
> addresses, but NOT with any other masks. Here is the idea:
>
> $template dsl, /var/log/dsl-%$NOW%.log
> $template routers, /var/log/routers-%$NOW%.log
> :FROMHOST, startswith, "192.168.1." -?dsl
> :FROMHOST, startswith, "192.168.3." -?routers
> :FROMHOST, startswith, "10.1.1." -?routers
>
> Common pitfall: be sure to include the trailing dot in the condition to
> match. If the rule were
>
> :FROMHOST, startswith, "192.168.1" -?dsl
>
> It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!
>
> With the current engine, there are unfortunately no logical operations
> available. So you cannot check for mail facility or whatever else. A
> somewhat crude work-around would be to include the facility in the file
> name, e.g. by specifying it as follows:
>
> $template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log
>
> That, of course, will result in a file written for each facility, even
> those that you are not interested in. A unfortunately do not have a
> solution for this now.
>
> With v3 expressions, I envision something along these lines:
>
> If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
> "mail" then
> writefile "/var/log/dslmail-%$NOW%"
>
> But that is not possible yet. And this is only an idea, not the actual
> config file format we will have at that time. In my blog, there are a
> number of posts about it, but the bottom line is that it is not yet
> designed:
>
> http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html
>
> Doc about current capabilities is here:
>
> http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml
>
> search for "Filter Conditions" on that page.
>
> The properties you need are documented here:
>
> http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
> phtml
>
> Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
> idea... ;))
>
> As always, feedback is appreciated. I would be most interested to learn
> at which final config you arrived.

Awesome! This is TOTALLY workable for what we're doing. I'm trying
to set it up and getting a weird message:

------------------------------------------------------------------

Dec 21 08:43:17 green rsyslogd:error: extra characters in config
line ignored: '/var/log/cisco-dsl-%$NOW%.log'
Dec 21 08:43:17 green rsyslogd: Could not find template 'dsl' -
action disabled
Dec 21 08:43:17 green rsyslogd:the last error occured in
/etc/rsyslog.conf, line 37

------------------------------------------------------------------

The config I used it almost exactly what you provided...

$template dsl, /var/log/cisco-dsl-%$NOW%.log
:FROMHOST, startswith, "10.3." -?dsl

Am I missing something basic?

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
[sniiiip]
> The config I used it almost exactly what you provided...
>
> $template dsl, /var/log/cisco-dsl-%$NOW%.log
> :FROMHOST, startswith, "10.3." -?dsl
>
> Am I missing something basic?

I forgot the quotes around the template string ;) That should work (not
tested, though):

$template dsl, "/var/log/cisco-dsl-%$NOW%.log"
:FROMHOST, startswith, "10.3." -?dsl

Rainer
Separating logs by host? [ In reply to ]
Rainer Gerhards wrote:
> I forgot the quotes around the template string ;) That should work (not
> tested, though):
>
> $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> :FROMHOST, startswith, "10.3." -?dsl

That was it... now rsyslog starts without errors.

Howeever it's not logging to the file I have set to though, instead
it's going to /var/log/messages. I've verified with tcpdump that
it's sending from the correct IP address. I've tried both startswith
and isequal and neither seem to work.

I don't know how to troubleshoot any further? Is there some debug
mode or something that will give me more information about why it's
not logging the way I think it should?

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
Debug mode is the keyword here ;)

Add -d -n to the command line and run it interactively with stdout
redirected to a file. Maybe I've made a mistake with the directives,
maybe we have another issue. Debug output can get quite large, please do
not send it to the list (I think mailman will bounce it anyhow). You can
email me privately (rgerhards at adiscon.com) if you like.

Just be warned: its 7p right now here and I can not promise to fix a
larger problem before xmas ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 6:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
>
> Rainer Gerhards wrote:
> > I forgot the quotes around the template string ;) That should work
> (not
> > tested, though):
> >
> > $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> > :FROMHOST, startswith, "10.3." -?dsl
>
> That was it... now rsyslog starts without errors.
>
> Howeever it's not logging to the file I have set to though, instead
> it's going to /var/log/messages. I've verified with tcpdump that
> it's sending from the correct IP address. I've tried both startswith
> and isequal and neither seem to work.
>
> I don't know how to troubleshoot any further? Is there some debug
> mode or something that will give me more information about why it's
> not logging the way I think it should?
>
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
Separating logs by host? [ In reply to ]
Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
>
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
>
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;)

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
Separating logs by host? [ In reply to ]
Just out of my head and from the pda while being out of office: try the -x option. Will check mail later again :)

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 21.12.07 19:13
Betreff: Re: [rsyslog] Separating logs by host?

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
>
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
>
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;)

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

--
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog