Mailing List Archive: Very complex apache log format

Very complex apache log format

Nov 16, 2007, 6:08 PM

Post #1 of 5 (3775 views)

Hi,

I've read the Analog docs pretty closely. I've used Analog on and off over
the last 10 years and am always pleased to come back...

Here are the facts:

Httpd.conf logformat line:

LogFormat "%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" \
\"(client %{User-agent}i)\" \"(elapsed %D)\"" mainserver

What I put in my analog.cfg file:

APACHELOGFORMAT (%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" \"(client
%{User-agent}i)\" \"(elapsed %D)\")

Which, as far as I can read, *should* work.. However, when I run analog:

$ analog -G +g./analog.cfg -v
analog: analog version 6.0/Unix
analog: Warning C: Too many arguments for configuration command: ignoring
end of line starting:
APACHELOGFORMAT (%h %l %u %t \"(%r) \"
(For help on all errors and warnings, see docs/errors.html)
analog: Warning C: Ignoring long configuration line starting
# cuscon11608.tstt.net.tt - - [04/Nov/2007:00:00:06 -0400] "(GET /imag
analog: Warning M: Logfile /u/2/j/jf2412/logfiles/*.20071110 contains lines
with no bytes: byte counts may be low

I do see that the hosts.txt file I set up is getting bigger and bigger so
SOMETHING's happening..

Of course I won't know until the file is done being parsed, etc.. But I'm
concerned about this error... Is there ANYTHING you can suggest I do to make
analog happier with the apache logformat?

TIA,

Joshua

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: Very complex apache log format [ In reply to ]

analog07 at eircom

Nov 17, 2007, 4:59 AM

Post #2 of 5 (3723 views)

Permalink

At Friday, November 16, 2007 9:08 PM, Joshua S. Freeman
<jf2412@columbia.edu> wrote:

> Hi,
>
> I've read the Analog docs pretty closely. I've used Analog on and
> off over the last 10 years and am always pleased to come back...
>
> Here are the facts:
>
> Httpd.conf logformat line:
>
> LogFormat "%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" \
> \"(client %{User-agent}i)\" \"(elapsed %D)\"" mainserver
>
>
> What I put in my analog.cfg file:
>
> APACHELOGFORMAT (%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\"
> \"(client %{User-agent}i)\" \"(elapsed %D)\")
>
>
> Which, as far as I can read, *should* work.. However, when I run
> analog:
>
> $ analog -G +g./analog.cfg -v
> analog: analog version 6.0/Unix
> analog: Warning C: Too many arguments for configuration command:
> ignoring end of line starting:
> APACHELOGFORMAT (%h %l %u %t \"(%r) \"
> (For help on all errors and warnings, see docs/errors.html)

You're using () as a delimiter for your LOGFORMAT, even though you have ()
characters in the format string. Use a different delimiter, such as [].

APACHELOGFORMAT [.%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\"
\"(client %{User-agent}i)\" \"(elapsed %D)\"]

> analog: Warning C: Ignoring long configuration line starting
> # cuscon11608.tstt.net.tt - - [04/Nov/2007:00:00:06 -0400] "(GET
> /imag

You probably put a line from your logfile into the .cfg so you could
compare it to your LOGFORMAT.

> analog: Warning M: Logfile /u/2/j/jf2412/logfiles/*.20071110
> contains lines with no bytes: byte counts may be low

Many of the reports will report on the bytes transferred - if you're not
recording that information, Analog can't report on it.

> I do see that the hosts.txt file I set up is getting bigger and
> bigger so SOMETHING's happening..
>
> Of course I won't know until the file is done being parsed, etc.. But
> I'm concerned about this error... Is there ANYTHING you can suggest I
> do to make analog happier with the apache logformat?

That sounds like you're having Analog do it's own DNS lookups. You really
need to look into some of the DNS Helper apps that will create the
"hosts.txt" file for you much faster than Analog can do on it's own.

http://www.analog.cx/helpers/#dns

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: Very complex apache log format [ In reply to ]

jf2412 at columbia

Nov 17, 2007, 7:10 AM

Post #3 of 5 (3662 views)

Permalink

Hi Aengus,

This thing with the [] doesn't really seem to be working... I'm guessing I
need to change to LOGFORMAT and figure out how to translate that
APACHELOGFORMAT string into an analog string?

J.

On 11/17/07 7:59 AM, "Aengus" <analog07@eircom.net> wrote:

> At Friday, November 16, 2007 9:08 PM, Joshua S. Freeman
> <jf2412@columbia.edu> wrote:
>
>> Hi,
>>
>> I've read the Analog docs pretty closely. I've used Analog on and
>> off over the last 10 years and am always pleased to come back...
>>
>> Here are the facts:
>>
>> Httpd.conf logformat line:
>>
>> LogFormat "%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" \
>> \"(client %{User-agent}i)\" \"(elapsed %D)\"" mainserver
>>
>>
>> What I put in my analog.cfg file:
>>
>> APACHELOGFORMAT (%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\"
>> \"(client %{User-agent}i)\" \"(elapsed %D)\")
>>
>>
>> Which, as far as I can read, *should* work.. However, when I run
>> analog:
>>
>> $ analog -G +g./analog.cfg -v
>> analog: analog version 6.0/Unix
>> analog: Warning C: Too many arguments for configuration command:
>> ignoring end of line starting:
>> APACHELOGFORMAT (%h %l %u %t \"(%r) \"
>> (For help on all errors and warnings, see docs/errors.html)
>
> You're using () as a delimiter for your LOGFORMAT, even though you have ()
> characters in the format string. Use a different delimiter, such as [].
>
> APACHELOGFORMAT [.%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\"
> \"(client %{User-agent}i)\" \"(elapsed %D)\"]
>
>> analog: Warning C: Ignoring long configuration line starting
>> # cuscon11608.tstt.net.tt - - [04/Nov/2007:00:00:06 -0400] "(GET
>> /imag
>
> You probably put a line from your logfile into the .cfg so you could
> compare it to your LOGFORMAT.
>
>> analog: Warning M: Logfile /u/2/j/jf2412/logfiles/*.20071110
>> contains lines with no bytes: byte counts may be low
>
> Many of the reports will report on the bytes transferred - if you're not
> recording that information, Analog can't report on it.
>
>> I do see that the hosts.txt file I set up is getting bigger and
>> bigger so SOMETHING's happening..
>>
>> Of course I won't know until the file is done being parsed, etc.. But
>> I'm concerned about this error... Is there ANYTHING you can suggest I
>> do to make analog happier with the apache logformat?
>
> That sounds like you're having Analog do it's own DNS lookups. You really
> need to look into some of the DNS Helper apps that will create the
> "hosts.txt" file for you much faster than Analog can do on it's own.
>
> http://www.analog.cx/helpers/#dns
>
> Aengus
>
> +------------------------------------------------------------------------
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html
> | List archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +------------------------------------------------------------------------

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: Very complex apache log format [ In reply to ]

analog-author at lists

Nov 17, 2007, 7:48 AM

Post #4 of 5 (3694 views)

Permalink

On 17/11/2007, Joshua S. Freeman <jf2412@columbia.edu> wrote:
> Hi Aengus,
>
> This thing with the [] doesn't really seem to be working... I'm guessing I
> need to change to LOGFORMAT and figure out how to translate that
> APACHELOGFORMAT string into an analog string?
>

Square brackets can't be used as delimiters. Single quotes will work, however.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: Very complex apache log format [ In reply to ]

jf2412 at columbia

Nov 17, 2007, 10:46 AM

Post #5 of 5 (3706 views)

Permalink

Thanks Stephen,

I replaced the square bracket with single quotes. I have tried running
analog with the double quotes escaped and un-escaped.

In both cases, Analog takes about 2 minutes to run and produces and empty
report.

In the case of the un-escaped double quoted version:

$ analog: analog version 6.0/Unix
analog: Warning L: Large number of corrupt lines in logfile
/u/2/j/jf2412/logfiles/httpd.access.20071110: turn debugging on or try
different LOGFORMAT
(For help on all errors and warnings, see docs/errors.html)
Current logfile format:
%S %j %j [%d/%M/%Y:%h:%n:%j] "(%j%w%r%wHTTP%j)" %c %b "(ref %f)"
"(client %B)" "(elapsed %D)"\n

The line in my analog.cfg is:

APACHELOGFORMAT '%h %l %u %t "(%r)" %>s %b "(ref %{Referer}i)" "(client
%{User-agent}i)" "(elapsed %D)"'

So, even though I am trying to use Analog's 'APACHELOGFORMAT' and our apache
log format is understood by our apache server, I'm not sure I'm able to
communicate its nuances successfully to Analog...

Thanks for any further advice.

Joshua

On 11/17/07 10:48 AM, "Stephen Turner" <analog-author@lists.meer.net> wrote:

> On 17/11/2007, Joshua S. Freeman <jf2412@columbia.edu> wrote:
>> Hi Aengus,
>>
>> This thing with the [] doesn't really seem to be working... I'm guessing I
>> need to change to LOGFORMAT and figure out how to translate that
>> APACHELOGFORMAT string into an analog string?
>>
>
> Square brackets can't be used as delimiters. Single quotes will work, however.

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------