Mailing List Archive: APACHELOGFORMAT and hosts report

APACHELOGFORMAT and hosts report

Oct 24, 2008, 7:20 AM

Post #1 of 3 (2414 views)

Hello Analog gurus,

I've been using Analog on-and-off for a while, and I'm a big fan.

I'm trying to get Analog to give me a "hosts" report. The problem I seem
to have is that the logs are writing an X-Forwarded-For header which is
the only way I have of knowing what the actual browser IP address was.
(lots of network topology in the way....)

So based on the following log format in Apache httpd.conf:
(I'm pretty sure this is current, but I will double-check)

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"\"%{Cookie}i\" %D" webtrends

So in analog.cfg, I have:

APACHELOGFORMAT (%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
\"%{Referer}i\" \"%{User-Agent}i\"\"%{Cookie}i\" %D)

And here's a sample line from the Apache access log:

10.235.166.27 - - [22/Oct/2008:09:22:49 -0500] "GET /wps/portal/xxx
HTTP/1.1" 400 65536 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)""WT_FPC=id=10.234.239.40-2330051872.29954568:lv=1224706655084:ss=1224706491290;
JSESSIONID=0000HDRNq7GzVKH0HRzrmcAv123:139i273in;
erU47MFBA6M2SE7HASZ6CLAGK3341=PWD=&CLX=EnhancedRTE&HMS=ppdapz0131&LGN=MJSW43TFNJZDC;
__utma=101953745.1997367580080200200.1221591400.1221591400.1221591400.1;
__utmz=101953745.1221591400.1.1.utmcsr=<hostname>.com|utmccn=(referral)|utmcmd=referral|utmcct=/wps/portal/!ut/p/c1/04_sb8k8xllm9msszpy8xbz9cp0os3gdfwnvj29dm2mxazmj91avl08jawjq9_piz03vl8h2vaqavxwhdw!!/dl2/d1/l2djqsevuut3qs9zqnb3lzzfme8ws0jlmtyzrda2mkdvskwxmjawmdawmda!/"
576318

Finally I get to my question: how can I get a "hosts" report from this?

I tried making the APACHELOGFORMAT use %S as the first token, but that
didn't work.

Thanks in advance!

Don Jones

Life is not tested or documented to be fair. Thinking life is fair is not
supported.

Re: APACHELOGFORMAT and hosts report [ In reply to ]

analog07 at eircom

Oct 24, 2008, 7:57 AM

Post #2 of 3 (2314 views)

Permalink

On 10/24/2008 10:20 AM, Don Jones wrote:
>
> Hello Analog gurus,
>
> I've been using Analog on-and-off for a while, and I'm a big fan.
>
> I'm trying to get Analog to give me a "hosts" report. The problem I
> seem to have is that the logs are writing an X-Forwarded-For header
> which is the only way I have of knowing what the actual browser IP
> address was. (lots of network topology in the way....)
>
> So based on the following log format in Apache httpd.conf:
> (I'm pretty sure this is current, but I will double-check)
>
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
> \"%{User-Agent}i\"\"%{Cookie}i\" %D" webtrends
>
> So in analog.cfg, I have:
>
> APACHELOGFORMAT (%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
> \"%{Referer}i\" \"%{User-Agent}i\"\"%{Cookie}i\" %D)
>
> And here's a sample line from the Apache access log:
>
> 10.235.166.27 - - [22/Oct/2008:09:22:49 -0500] "GET /wps/portal/xxx
> HTTP/1.1" 400 65536 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
> 5.1; SV1; .NET CLR 1.1.4322; .NET CLR
> 2.0.50727)""WT_FPC=id=10.234.239.40-2330051872.29954568:lv=1224706655084:ss=1224706491290;
> JSESSIONID=0000HDRNq7GzVKH0HRzrmcAv123:139i273in;
> erU47MFBA6M2SE7HASZ6CLAGK3341=PWD=&CLX=EnhancedRTE&HMS=ppdapz0131&LGN=MJSW43TFNJZDC;
> __utma=101953745.1997367580080200200.1221591400.1221591400.1221591400.1;
> __utmz=101953745.1221591400.1.1.utmcsr=<hostname>.com|utmccn=(referral)|utmcmd=referral|utmcct=/wps/portal/!ut/p/c1/04_sb8k8xllm9msszpy8xbz9cp0os3gdfwnvj29dm2mxazmj91avl08jawjq9_piz03vl8h2vaqavxwhdw!!/dl2/d1/l2djqsevuut3qs9zqnb3lzzfme8ws0jlmtyzrda2mkdvskwxmjawmdawmda!/"
> 576318
>
> Finally I get to my question: how can I get a "hosts" report from this?
>
> I tried making the APACHELOGFORMAT use %S as the first token, but that
> didn't work.

APACHELOGFORMAT is simply a mechanism for translating the line from the
Apache configuration file into "native" Analog format. Whenever your
Apache logformat string gets a bit complex, you're going to have to give
up on the convenience of this automatic translation mechanism, and tell
Analog exactly how it should interpret the logfile, by writing an Analog
LOGFORMAT string, rather than relying on Analog to do the translation
for you.

Try this LOGFORMAT
(%S %j %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b "%f" "%B""%j" %D)

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Re: APACHELOGFORMAT and hosts report [ In reply to ]

donjones at us

Oct 24, 2008, 10:35 AM

Post #3 of 3 (2308 views)

Permalink

That worked beautifully! Thank you!

Don Jones
Life is not tested or documented to be fair. Thinking life is fair is not
supported.

From:
Aengus <analog07@eircom.net>
To:
Support for analog web log analyzer <analog-help@lists.meer.net>
Date:
10/24/2008 11:03 AM
Subject:
Re: [analog-help] APACHELOGFORMAT and hosts report

On 10/24/2008 10:20 AM, Don Jones wrote:
>
> Hello Analog gurus,
>
> I've been using Analog on-and-off for a while, and I'm a big fan.
>
> I'm trying to get Analog to give me a "hosts" report. The problem I
> seem to have is that the logs are writing an X-Forwarded-For header
> which is the only way I have of knowing what the actual browser IP
> address was. (lots of network topology in the way....)
>
> So based on the following log format in Apache httpd.conf:
> (I'm pretty sure this is current, but I will double-check)
>
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
> \"%{User-Agent}i\"\"%{Cookie}i\" %D" webtrends
>
> So in analog.cfg, I have:
>
> APACHELOGFORMAT (%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
> \"%{Referer}i\" \"%{User-Agent}i\"\"%{Cookie}i\" %D)
>
> And here's a sample line from the Apache access log:
>
> 10.235.166.27 - - [22/Oct/2008:09:22:49 -0500] "GET /wps/portal/xxx
> HTTP/1.1" 400 65536 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
> 5.1; SV1; .NET CLR 1.1.4322; .NET CLR
>
2.0.50727)""WT_FPC=id=10.234.239.40-2330051872.29954568:lv=1224706655084:ss=1224706491290;

> JSESSIONID=0000HDRNq7GzVKH0HRzrmcAv123:139i273in;
>
erU47MFBA6M2SE7HASZ6CLAGK3341=PWD=&CLX=EnhancedRTE&HMS=ppdapz0131&LGN=MJSW43TFNJZDC;

> __utma=101953745.1997367580080200200.1221591400.1221591400.1221591400.1;

>
__utmz=101953745.1221591400.1.1.utmcsr=<hostname>.com|utmccn=(referral)|utmcmd=referral|utmcct=/wps/portal/!ut/p/c1/04_sb8k8xllm9msszpy8xbz9cp0os3gdfwnvj29dm2mxazmj91avl08jawjq9_piz03vl8h2vaqavxwhdw!!/dl2/d1/l2djqsevuut3qs9zqnb3lzzfme8ws0jlmtyzrda2mkdvskwxmjawmdawmda!/"

> 576318
>
> Finally I get to my question: how can I get a "hosts" report from this?
>
> I tried making the APACHELOGFORMAT use %S as the first token, but that
> didn't work.

APACHELOGFORMAT is simply a mechanism for translating the line from the
Apache configuration file into "native" Analog format. Whenever your
Apache logformat string gets a bit complex, you're going to have to give
up on the convenience of this automatic translation mechanism, and tell
Analog exactly how it should interpret the logfile, by writing an Analog
LOGFORMAT string, rather than relying on Analog to do the translation
for you.

Try this LOGFORMAT
(%S %j %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b "%f" "%B""%j" %D)

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------