Mailing List Archive: Troubleshooting missing log entries

Troubleshooting missing log entries

Dec 18, 2008, 11:59 AM

Post #1 of 5 (2007 views)

I have the following entry in my rsyslog conf, to match entries based on IP
address. Somehow it's not matching any entries.

# Switches
$FileCreateMode 0644
:FROMHOST, isequal, "65.182.224.13" -?switches # Necalea
:FROMHOST, isequal, "65.182.224.202" -?switches
:FROMHOST, isequal, "66.206.80.60" -?switches

If I do a tcpdump I see syslog hitting the box, it's just rsyslog isn't
handling it right.

11:58:20.722867 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
local4.info, length: 121
11:58:23.962613 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
local4.info, length: 130
11:58:41.242621 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
local4.info, length: 108
11:58:45.874064 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
local4.info, length: 130

This box gets about 500 lines of syslog a minute so I can't really turn on
debug. How else can I troubleshoot this? This is a Fedora 8 box running:
rsyslog-2.0.2-3.fc8

- Scott
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Re: Troubleshooting missing log entries [ In reply to ]

rgerhards at hq

Dec 18, 2008, 11:19 AM

Post #2 of 5 (1956 views)

Permalink

On Thu, 2008-12-18 at 11:59 -0800, Scott Baker wrote:
> I have the following entry in my rsyslog conf, to match entries based on IP
> address. Somehow it's not matching any entries.
>
> # Switches
> $FileCreateMode 0644
> :FROMHOST, isequal, "65.182.224.13" -?switches # Necalea
> :FROMHOST, isequal, "65.182.224.202" -?switches
> :FROMHOST, isequal, "66.206.80.60" -?switches
>
> If I do a tcpdump I see syslog hitting the box, it's just rsyslog isn't
> handling it right.
>
> 11:58:20.722867 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
> local4.info, length: 121
> 11:58:23.962613 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
> local4.info, length: 130
> 11:58:41.242621 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
> local4.info, length: 108
> 11:58:45.874064 IP 65.182.224.13.8888 > 65.182.224.26.514: SYSLOG
> local4.info, length: 130
>
> This box gets about 500 lines of syslog a minute so I can't really turn on
> debug. How else can I troubleshoot this? This is a Fedora 8 box running:
> rsyslog-2.0.2-3.fc8

I'd still go for debug mode. You don't need to run it very long. We just
need to see how a few of these messages are fully processed. A proper
test setup would be to start up in debug mode with the network cable
pulled, then plug it in for a second or two, then unplug it again. Once
rsyslogd is finished processing, stop it. That should lead to useful
info in the debug log.

Oh - and are you sure that fromhost has the proper IP addresses? If not
100% sure, verify it by putting something like '%FROMHOST%' into a debug
template (note that there is also FROMHOST-IP, which will have the IP
address no matter if names are resolved or not).

HTH,
Rainer
>
> - Scott
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Re: Troubleshooting missing log entries [ In reply to ]

david at lang

Dec 19, 2008, 4:07 AM

Post #3 of 5 (1960 views)

Permalink

On Thu, 18 Dec 2008, Rainer Gerhards wrote:

> On Thu, 2008-12-18 at 11:59 -0800, Scott Baker wrote:
>> I have the following entry in my rsyslog conf, to match entries based on IP
>> address. Somehow it's not matching any entries.
>>
>> # Switches
>> $FileCreateMode 0644
>> :FROMHOST, isequal, "65.182.224.13" -?switches # Necalea
>> :FROMHOST, isequal, "65.182.224.202" -?switches
>> :FROMHOST, isequal, "66.206.80.60" -?switches
>
> Oh - and are you sure that fromhost has the proper IP addresses? If not
> 100% sure, verify it by putting something like '%FROMHOST%' into a debug
> template (note that there is also FROMHOST-IP, which will have the IP
> address no matter if names are resolved or not).

I was seeing some issues where the fromhost was not getting set properly,
I'll have to go back and dig up the details, but I think I was seeing it
use the localhost as the fromhost and putting the real fromhost
information in the message.

I found it by creating an output format that I could tweak and playing
with it to see what was actually showing up in the various parameters.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Re: Troubleshooting missing log entries [ In reply to ]

bakers at web-ster

Dec 19, 2008, 8:38 AM

Post #4 of 5 (1962 views)

Permalink

Rainer Gerhards wrote:
> I'd still go for debug mode. You don't need to run it very long. We just
> need to see how a few of these messages are fully processed. A proper
> test setup would be to start up in debug mode with the network cable
> pulled, then plug it in for a second or two, then unplug it again. Once
> rsyslogd is finished processing, stop it. That should lead to useful
> info in the debug log.
>
> Oh - and are you sure that fromhost has the proper IP addresses? If not
> 100% sure, verify it by putting something like '%FROMHOST%' into a debug
> template (note that there is also FROMHOST-IP, which will have the IP
> address no matter if names are resolved or not).

I like the debug template idea, that's genius. Is there a way to have a
bunch of filters to catch assorted things, and then an "everything
leftover" filter?

------------------------------------------------------------------------

# Mail servers log to their special section
$FileCreateMode 0644
:FROMHOST, isequal, "magenta" -?magic-mail
:FROMHOST, isequal, "cyan" -?magic-mail
:FROMHOST, isequal, "orange" -?magic-mail

# Firewalls
:FROMHOST, isequal, "yin" -?firewall
:FROMHOST, isequal, "yang" -?firewall

# Everything that didn't get caught by one of the above filters
(I have no idea what the syntax would be)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Re: Troubleshooting missing log entries [ In reply to ]

rgerhards at hq

Dec 19, 2008, 8:40 AM

Post #5 of 5 (1950 views)

Permalink

Without verification, but should work:

# Mail servers log to their special section
$FileCreateMode 0644
:FROMHOST, isequal, "magenta" -?magic-mail
& ~
:FROMHOST, isequal, "cyan" -?magic-mail
& ~
:FROMHOST, isequal, "orange" -?magic-mail
& ~

# Firewalls
:FROMHOST, isequal, "yin" -?firewall
& ~
:FROMHOST, isequal, "yang" -?firewall
& ~

*.* /var/log/catchrest

& ~ discards the message after it is written to the file in question.

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 19, 2008 5:39 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Troubleshooting missing log entries
>
> Rainer Gerhards wrote:
> > I'd still go for debug mode. You don't need to run it very long. We
> just
> > need to see how a few of these messages are fully processed. A
proper
> > test setup would be to start up in debug mode with the network cable
> > pulled, then plug it in for a second or two, then unplug it again.
> Once
> > rsyslogd is finished processing, stop it. That should lead to useful
> > info in the debug log.
> >
> > Oh - and are you sure that fromhost has the proper IP addresses? If
> not
> > 100% sure, verify it by putting something like '%FROMHOST%' into a
> debug
> > template (note that there is also FROMHOST-IP, which will have the
IP
> > address no matter if names are resolved or not).
>
>
> I like the debug template idea, that's genius. Is there a way to have
a
> bunch of filters to catch assorted things, and then an "everything
> leftover" filter?
>
>
-----------------------------------------------------------------------
> -
>
> # Mail servers log to their special section
> $FileCreateMode 0644
> :FROMHOST, isequal, "magenta" -?magic-mail
> :FROMHOST, isequal, "cyan" -?magic-mail
> :FROMHOST, isequal, "orange" -?magic-mail
>
> # Firewalls
> :FROMHOST, isequal, "yin" -?firewall
> :FROMHOST, isequal, "yang" -?firewall
>
> # Everything that didn't get caught by one of the above filters
> (I have no idea what the syntax would be)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com