Mailing List Archive

I'm not familiar with syslog-ng, but I suspect this is just a
difference in the formatting/interpretation of the raw packet. Try
capturing a message or two from syslog-ng with tcpdump and compare to
section 4 of RFC 3164.

You can see how rsyslog interprets it with the following template:
$template test, "TIME: %timestamp% HOST: %hostname% TAG: %syslogtag%
PROGRAM: %programname% MSG: %rawmsg%\n"

This should give you some idea of how to create a template that will
log in the format you need.

-HKS



On Thu, Aug 21, 2008 at 1:33 PM, Jeff Schroeder <jeffschroed at gmail.com> wrote:
> Rsyslog seems to be mangling messages sent from our in-house applications.
> We are trying to get the same format as we did with syslog-ng previously and
> not having a lot of luck.
>
> The logs are in the format something like this:
> TAG APPLICATION: MSG
>
> In syslog-ng, it looks like this locally and when sent to a remote
> syslog-ng server:
> Aug 21 00:00:00 ops051.nyc03.int ADMIN JAVA-EVENT:
> ops-192.168.101.251:65261 offset changed 0.0004593 seconds!
>
> rsyslog with the TraditionalFileFormat makes it look like this
> locally, stripping the hostname:
> Aug 21 00:00:00 JAVA-EVENT: ops-192.168.101.251:65261 offset changed
> 0.0004593 seconds!
>
> and like this when sent to a remote syslog-ng server. Notice how the
> ADMIN tag was removed:
> Aug 21 00:00:00 ops051.nyc03.int JAVA-EVENT: ops-192.168.101.251:65261
> offset changed 0.0004593 seconds!
>
> We have realtime logscraping software that looks at all message tags
> and does custom reporting on them.
> It keys off the TAG property that rsyslog is stripping off. What can
> we do to enable this? We've been playing
> with custom $templates in the rsyslog.conf to no real avail.
>
> Thanks!
>
> --
> Jeff Schroeder
>
> Don't drink and derive, alcohol and analysis don't mix.
> http://www.digitalprognosis.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

I have only been able to have a brief look, but it looks like the message is incorrectly formatted. rsyslog is smart enough to detect that the hostname is missing if the tag is followed by a character not valid in hostnames. But if the tag even looks like a hostname, it has no chance of detecting that it isn't one. As suggested, see RFC 3164 for what the format should look like. I think the -x option (or some other) enables to strip hostname detection, but I am not sure. You can "solve" this by misusing some fields. E.g. FROMHOST probably has what actually is the tag. HKS suggestion will help you find a suitable format.

Rainer


-----Original Message-----
From: rsyslog-bounces@lists.adiscon.com on behalf of (private) HKS
Sent: Thu 8/21/2008 7:46 PM
To: rsyslog-users
Subject: Re: [rsyslog] Problems migrating from syslog-ng

I'm not familiar with syslog-ng, but I suspect this is just a
difference in the formatting/interpretation of the raw packet. Try
capturing a message or two from syslog-ng with tcpdump and compare to
section 4 of RFC 3164.

You can see how rsyslog interprets it with the following template:
$template test, "TIME: %timestamp% HOST: %hostname% TAG: %syslogtag%
PROGRAM: %programname% MSG: %rawmsg%\n"

This should give you some idea of how to create a template that will
log in the format you need.

-HKS



On Thu, Aug 21, 2008 at 1:33 PM, Jeff Schroeder <jeffschroed at gmail.com> wrote:
> Rsyslog seems to be mangling messages sent from our in-house applications.
> We are trying to get the same format as we did with syslog-ng previously and
> not having a lot of luck.
>
> The logs are in the format something like this:
> TAG APPLICATION: MSG
>
> In syslog-ng, it looks like this locally and when sent to a remote
> syslog-ng server:
> Aug 21 00:00:00 ops051.nyc03.int ADMIN JAVA-EVENT:
> ops-192.168.101.251:65261 offset changed 0.0004593 seconds!
>
> rsyslog with the TraditionalFileFormat makes it look like this
> locally, stripping the hostname:
> Aug 21 00:00:00 JAVA-EVENT: ops-192.168.101.251:65261 offset changed
> 0.0004593 seconds!
>
> and like this when sent to a remote syslog-ng server. Notice how the
> ADMIN tag was removed:
> Aug 21 00:00:00 ops051.nyc03.int JAVA-EVENT: ops-192.168.101.251:65261
> offset changed 0.0004593 seconds!
>
> We have realtime logscraping software that looks at all message tags
> and does custom reporting on them.
> It keys off the TAG property that rsyslog is stripping off. What can
> we do to enable this? We've been playing
> with custom $templates in the rsyslog.conf to no real avail.
>
> Thanks!
>
> --
> Jeff Schroeder
>
> Don't drink and derive, alcohol and analysis don't mix.
> http://www.digitalprognosis.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

On Thu, Aug 21, 2008 at 10:53 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> I have only been able to have a brief look, but it looks like the message is incorrectly formatted. rsyslog is smart enough to detect that the hostname is missing if the tag is followed by a character not valid in hostnames. But if the tag even looks like a hostname, it has no chance of detecting that it isn't one. As suggested, see RFC 3164 for what the format should look like. I think the -x option (or some other) enables to strip hostname detection, but I am not sure. You can "solve" this by misusing some fields. E.g. FROMHOST probably has what actually is the tag. HKS suggestion will help you find a suitable format.

Is there an equivalent of "-x" with "-c 3" enabled? It doesn't seem to
work with -c3 and I'd
rather not run in compatibility mode.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com on behalf of (private) HKS
> Sent: Thu 8/21/2008 7:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems migrating from syslog-ng
>
> I'm not familiar with syslog-ng, but I suspect this is just a
> difference in the formatting/interpretation of the raw packet. Try
> capturing a message or two from syslog-ng with tcpdump and compare to
> section 4 of RFC 3164.
>
> You can see how rsyslog interprets it with the following template:
> $template test, "TIME: %timestamp% HOST: %hostname% TAG: %syslogtag%
> PROGRAM: %programname% MSG: %rawmsg%\n"
>
> This should give you some idea of how to create a template that will
> log in the format you need.
>
> -HKS
>
>
>
> On Thu, Aug 21, 2008 at 1:33 PM, Jeff Schroeder <jeffschroed at gmail.com> wrote:
>> Rsyslog seems to be mangling messages sent from our in-house applications.
>> We are trying to get the same format as we did with syslog-ng previously and
>> not having a lot of luck.
>>
>> The logs are in the format something like this:
>> TAG APPLICATION: MSG
>>
>> In syslog-ng, it looks like this locally and when sent to a remote
>> syslog-ng server:
>> Aug 21 00:00:00 ops051.nyc03.int ADMIN JAVA-EVENT:
>> ops-192.168.101.251:65261 offset changed 0.0004593 seconds!
>>
>> rsyslog with the TraditionalFileFormat makes it look like this
>> locally, stripping the hostname:
>> Aug 21 00:00:00 JAVA-EVENT: ops-192.168.101.251:65261 offset changed
>> 0.0004593 seconds!
>>
>> and like this when sent to a remote syslog-ng server. Notice how the
>> ADMIN tag was removed:
>> Aug 21 00:00:00 ops051.nyc03.int JAVA-EVENT: ops-192.168.101.251:65261
>> offset changed 0.0004593 seconds!
>>
>> We have realtime logscraping software that looks at all message tags
>> and does custom reporting on them.
>> It keys off the TAG property that rsyslog is stripping off. What can
>> we do to enable this? We've been playing
>> with custom $templates in the rsyslog.conf to no real avail.
>>
>> Thanks!

--
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

On Thu, Aug 21, 2008 at 4:44 PM, Jeff Schroeder <jeffschroed at gmail.com> wrote:
> On Thu, Aug 21, 2008 at 10:53 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> I have only been able to have a brief look, but it looks like the message is incorrectly formatted. rsyslog is smart enough to detect that the hostname is missing if the tag is followed by a character not valid in hostnames. But if the tag even looks like a hostname, it has no chance of detecting that it isn't one. As suggested, see RFC 3164 for what the format should look like. I think the -x option (or some other) enables to strip hostname detection, but I am not sure. You can "solve" this by misusing some fields. E.g. FROMHOST probably has what actually is the tag. HKS suggestion will help you find a suitable format.
>
> Is there an equivalent of "-x" with "-c 3" enabled? It doesn't seem to
> work with -c3 and I'd
> rather not run in compatibility mode.


I don't think so.

-HKS

On Thu, Aug 21, 2008 at 2:00 PM, (private) HKS <hks.private at gmail.com> wrote:
> On Thu, Aug 21, 2008 at 4:44 PM, Jeff Schroeder <jeffschroed at gmail.com> wrote:
>> On Thu, Aug 21, 2008 at 10:53 AM, Rainer Gerhards
>> <rgerhards at hq.adiscon.com> wrote:
>>> I have only been able to have a brief look, but it looks like the message is incorrectly formatted. rsyslog is smart enough to detect that the hostname is missing if the tag is followed by a character not valid in hostnames. But if the tag even looks like a hostname, it has no chance of detecting that it isn't one. As suggested, see RFC 3164 for what the format should look like. I think the -x option (or some other) enables to strip hostname detection, but I am not sure. You can "solve" this by misusing some fields. E.g. FROMHOST probably has what actually is the tag. HKS suggestion will help you find a suitable format.

You were right Rainer. It looks like the java code which injects the
message is sending malformed
syslog requests. syslog-ng still sends it through and does the correct
things. Is there a way to make
rsyslog a bit less strict about it? Running rsyslog with -c0 defeats
the purpose of using rsyslog.

Until our application has been fixed and rolled out accross our
clusters worldwide, we rolled back to syslog-ng.


>>
>> Is there an equivalent of "-x" with "-c 3" enabled? It doesn't seem to
>> work with -c3 and I'd
>> rather not run in compatibility mode.
>
>
> I don't think so.
>
> -HKS
>



--
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

Being back...

(inline)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jeff Schroeder
> Sent: Friday, August 22, 2008 3:21 PM
> To: (private) HKS
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Problems migrating from syslog-ng
>
> On Thu, Aug 21, 2008 at 2:00 PM, (private) HKS <hks.private at gmail.com>
> wrote:
> > On Thu, Aug 21, 2008 at 4:44 PM, Jeff Schroeder
> <jeffschroed at gmail.com> wrote:
> >> On Thu, Aug 21, 2008 at 10:53 AM, Rainer Gerhards
> >> <rgerhards at hq.adiscon.com> wrote:
> >>> I have only been able to have a brief look, but it looks like the
> message is incorrectly formatted. rsyslog is smart enough to detect
> that the hostname is missing if the tag is followed by a character not
> valid in hostnames. But if the tag even looks like a hostname, it has
> no chance of detecting that it isn't one. As suggested, see RFC 3164
> for what the format should look like. I think the -x option (or some
> other) enables to strip hostname detection, but I am not sure. You can
> "solve" this by misusing some fields. E.g. FROMHOST probably has what
> actually is the tag. HKS suggestion will help you find a suitable
> format.
>
> You were right Rainer. It looks like the java code which injects the
> message is sending malformed
> syslog requests.

Please provide samples of the raw messages, what syslog-ng does to them
and what rsyslog does (and what you would ideally like to see, if that's
different in any aspect ;)). [.I know you have sent most of it - except
the source message, but I'd like to have a consistent set to look at.]

> syslog-ng still sends it through and does the correct
> things. Is there a way to make
> rsyslog a bit less strict about it?

It depends on the above things. The problem is that when we cannot
detect whether it is a tag or a hostname, there is no way to do it
automatically. I can, of course, add a switch that tells the parser that
there never is a hostname inside the message. I suspect this is what
syslog-ng is doing. This prevents relay chains from properly conveying
the hostname, but I guess it would work in your case. It needs to be a
user option, because obviously most users will never want to use this
handling.

> Running rsyslog with -c0 defeats
> the purpose of using rsyslog.

Well... not really. The -cX switches change some aspects of behavior,
but do not change the core itself. However, I do not think that -c0
would change anything. Does it? If so, my analysis would obviously be
wrong...

> Until our application has been fixed and rolled out accross our
> clusters worldwide, we rolled back to syslog-ng.

Of course, I'd like to support the format as-is (under above
constraints;)).

Rainer
>
>
> >>
> >> Is there an equivalent of "-x" with "-c 3" enabled? It doesn't seem
> to
> >> work with -c3 and I'd
> >> rather not run in compatibility mode.
> >
> >
> > I don't think so.
> >
> > -HKS
> >
>
>
>
> --
> Jeff Schroeder
>
> Don't drink and derive, alcohol and analysis don't mix.
> http://www.digitalprognosis.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog