Mailing List Archive: Logformat problems

Logformat problems

Jul 26, 2001, 9:04 AM

Post #1 of 6 (3196 views)

Can anyone help with this? I have about 16 log files which were saved by a
colleague as text files, and then forwarded to me. All of them give corrupt
line errors. Example:

142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET / HTTP/1.0" 200 4250 "
http://www.coolcanine.com/cgi-bin/search/index.pl?/Recreation/Pets/Dogs/Trai
ning/
" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)"
142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET
/images/buttons/video.gif HTTP/1.0" 200 3449 "-" "Mozilla/3.01
(compatible;)"

All the files which were emailed directly to me from the server, and also
saved as text files, are analysed without problem. Example:

136.167.201.215 - - [15/Jul/2001:17:12:49 +0000] "GET /lads.txt HTTP/1.0"
200 7495 "-" "Mozilla/4.61 [en] (Win98; I)"
136.167.201.215 - - [15/Jul/2001:17:12:49 +0000] "GET /reading_like/
HTTP/1.0" 200 11861
"http://www.google.com/search?q=reward+marker+dog&hl=en&safe=off&start=20&sa
=N" "Mozilla/4.61 [en] (Win98; I)"

The original log files in both cases were emailed from the same server, and
saved as text files, and I can't see any difference. Any suggestions
welcome!

Thanks

Margaret Reed

+------------------------------------------------------------------------
| This is the analog-help mailing list. To unsubscribe from this
| mailing list, go to
| http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
| List archives are available at
| http://www.mail-archive.com/analog-help@lists.isite.net/
| http://lists.isite.net/listgate/analog-help/archives/
| http://www.tallylist.com/archives/index.cfm/mlist.7
+------------------------------------------------------------------------

Logformat problems [ In reply to ]

S.R.E.Turner at statslab

Jul 27, 2001, 11:02 AM

Post #2 of 6 (3063 views)

Permalink

On Thu, 26 Jul 2001, Margaret Reed wrote:

> Can anyone help with this? I have about 16 log files which were saved by a
> colleague as text files, and then forwarded to me. All of them give corrupt
> line errors. Example:
>
> 142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET / HTTP/1.0" 200 4250 "
> http://www.coolcanine.com/cgi-bin/search/index.pl?/Recreation/Pets/Dogs/Trai
> ning/
> " "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)"
> 142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET
> /images/buttons/video.gif HTTP/1.0" 200 3449 "-" "Mozilla/3.01
> (compatible;)"
>

These seem to work fine, as long as they're all on one line. But many mail
programs put extra line breaks in your text, so you might want to check that
each entry is one line. That's the only thing I can think of immediately.

--
Stephen Turner http://www.statslab.cam.ac.uk/~sret1/
Statistical Laboratory, Wilberforce Road, Cambridge, CB3 0WB, England
"This is Henman's 8th Wimbledon, and he's only lost 7 matches." BBC, 2/Jul/01

+------------------------------------------------------------------------
| This is the analog-help mailing list. To unsubscribe from this
| mailing list, go to
| http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
| List archives are available at
| http://www.mail-archive.com/analog-help@lists.isite.net/
| http://lists.isite.net/listgate/analog-help/archives/
| http://www.tallylist.com/archives/index.cfm/mlist.7
+------------------------------------------------------------------------

Logformat problems [ In reply to ]

margaret at the-reeds

Jul 30, 2001, 6:47 AM

Post #3 of 6 (3063 views)

Permalink

Many thanks - I'll make different arrangements next time I'm on holiday to
avoid this problem!

Margaret

> -----Original Message-----
> From: owner-analog-help@lists.isite.net
> [mailto:owner-analog-help@lists.isite.net]On Behalf Of Stephen Turner
> Sent: 27 July 2001 19:02
> To: analog-help@lists.isite.net
> Subject: Re: [analog-help] Logformat problems
>
>
> On Thu, 26 Jul 2001, Margaret Reed wrote:
>
> > Can anyone help with this? I have about 16 log files which were
> saved by a
> > colleague as text files, and then forwarded to me. All of them
> give corrupt
> > line errors. Example:
> >
> > 142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET / HTTP/1.0"
> 200 4250 "
> >
> http://www.coolcanine.com/cgi-bin/search/index.pl?/Recreation/Pets
/Dogs/Trai
> ning/
> " "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)"
> 142.165.70.20 - - [01/Jul/2001:21:22:34 +0000] "GET
> /images/buttons/video.gif HTTP/1.0" 200 3449 "-" "Mozilla/3.01
> (compatible;)"
>

These seem to work fine, as long as they're all on one line. But many mail
programs put extra line breaks in your text, so you might want to check that
each entry is one line. That's the only thing I can think of immediately.

--
Stephen Turner http://www.statslab.cam.ac.uk/~sret1/
Statistical Laboratory, Wilberforce Road, Cambridge, CB3 0WB, England
"This is Henman's 8th Wimbledon, and he's only lost 7 matches." BBC,
2/Jul/01

+------------------------------------------------------------------------
| This is the analog-help mailing list. To unsubscribe from this
| mailing list, go to
| http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
| List archives are available at
| http://www.mail-archive.com/analog-help@lists.isite.net/
| http://lists.isite.net/listgate/analog-help/archives/
| http://www.tallylist.com/archives/index.cfm/mlist.7
+------------------------------------------------------------------------

+------------------------------------------------------------------------
| This is the analog-help mailing list. To unsubscribe from this
| mailing list, go to
| http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
| List archives are available at
| http://www.mail-archive.com/analog-help@lists.isite.net/
| http://lists.isite.net/listgate/analog-help/archives/
| http://www.tallylist.com/archives/index.cfm/mlist.7
+------------------------------------------------------------------------

Re: Logformat Problems [ In reply to ]

analog07 at eircom

Feb 29, 2008, 6:38 PM

Post #4 of 6 (3065 views)

Permalink

At Friday, February 29, 2008 6:20 PM, The Wolf <tsawolf@gmail.com> wrote:

>> Hello everyone!
>>
>> I've been messing around with LOGFORMAT, trying to get it working
>> with our non-standard log setup.
>>
>> Unfortunately, there seem to be some rather weird errors going on.
>>
>> LOGFORMAT (%v\t%j\t%t\t%s\t%u\t[%d/%M/%Y:%h:%n:%j
>> %j]\t"%r"\t%c\t%b\t"%f"\t"%B") matches our real setup. All fields
>> are tab delimited, except for the date which is ISO compliant.
>>
>> servername, gzip Ratio, processing time, client IP, username,
>> [DD/MMM/YYYY:HH:NN:SS -TZTZ], "Request", status code, size,
>> "Referrer", "Useragent"
>>
>> However, analog thinks that all lines are corrupt, and the error
>> seems to point towards a different place in lines.
>>
>> A sample (sanitized for our users protection):
>> C: server - - 1.2.3.4 -
>> [27/Feb/2008:18:54:24 -0500] "GET /images/image.png HTTP/1.1"
>> 200 4373 " www.somesite.com" "Mozilla/5.0 (Windows; U;
>> Windows NT 6.0; en-US;) Gecko/0000 Firefox/0000"
>> C: *
>>
>> If we put a space where the first tab is (some of the tabs appear a
>> space long, but cat -e confirms they are actually tabs), it changes
>> to this:
>> C: server - - 1.2.3.4 -
>> [27/Feb/2008:18:54:24 -0500] "GET /images/image.png HTTP/1.1"
>> 200 4373 " www.somesite.com" "Mozilla/5.0 (Windows; U;
>> Windows NT 6.0; en-US;) Gecko/0000 Firefox/0000"
>> C:
>> *
>>
>> I'm not sure what to do next, so I turn to you.

If I change your %t to %j, Analog parses the line - it appears that "-"
isn't a valid %t.

(Tab delimited log files are almost impossible to debug - you have to
change the tabs to spaces in a sample line, and the \t to %w in the
LOGFORMAT to have any chance of figuring out what Analog doesn't like).

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: Logformat Problems [ In reply to ]

tsawolf at gmail

Feb 29, 2008, 6:52 PM

Post #5 of 6 (3064 views)

Permalink

That's unfortunate. Still, I guess I can make a awk script to ignore that
part.

Still. That's a ENH bug for sure. ;-)

Thanks!

tsawolf

On Fri, Feb 29, 2008 at 9:38 PM, Aengus <analog07@eircom.net> wrote:

> If I change your %t to %j, Analog parses the line - it appears that "-"
> isn't a valid %t.
>
> (Tab delimited log files are almost impossible to debug - you have to
> change the tabs to spaces in a sample line, and the \t to %w in the
> LOGFORMAT to have any chance of figuring out what Analog doesn't like).
>
> Aengus
>

Re: Logformat Problems [ In reply to ]

analog07 at eircom

Mar 1, 2008, 5:49 AM

Post #6 of 6 (3060 views)

Permalink

At Friday, February 29, 2008 9:52 PM, The Wolf <tsawolf@gmail.com> wrote:

>> That's unfortunate. Still, I guess I can make a awk script to ignore
>> that part.

No need to do that - just specify %j in the LOGFORMAt - that's what I did!

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------