Mailing List Archive

Corrupt log entries
I have submitted this question to the analog forum as well as going through
the documentation for about a week now and I am no closer to finding a
resolution. Any help you may be able to give would be much appreciated.

I am running an Ezrproxy server (Fedora 8) and I want Analog (Same Fedora 8
Server) to provide stats based on the Ezproxy logs.

The designated Ezproxy log format is:
LogSPU spu.log %{ezproxy-spuaccess}i %h %l %u %t "%r" %s %b

Here are a few logs in the "spu.log" file:
./analog: analog version 6.0/Unix
F: Closing configuration file /usr/local/analog-6.0.3/analog.cfg
F: Opening /usr/local/analog-6.0.3/lang/uk.lng as language file
F: Closing language file /usr/local/analog-6.0.3/lang/uk.lng
F: Opening /usr/local/analog-6.0.3/lang/ukdom.tab as domains file
F: Closing domains file /usr/local/analog-6.0.3/lang/ukdom.tab
F: Opening /usr/local/analog-6.0.3/lang/ukdesc.txt as report descriptions
file
F: Closing report descriptions file /usr/local/analog-6.0.3/lang/ukdesc.txt
F: Opening /usr/local/analog-6.0.3/logfile.log as logfile
C: proxy 205.188.116.15 - gNDs4AexHJMk3B9 [15/Jul/2008:21:04:33 -0600] "GET
http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.c
om?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.205 - gNDs4AexHJMk3B9 [15/Jul/2008:21:04:33 -0600] "GET
http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 192.168.1.130 - E37HRQKiv0dANot [18/Jul/2008:14:53:48 -0600] "GET
http://infoweb.newsbank.com?db=OBIT HTTP/1.1" 200 0
C: *
C: proxy 66.158.45.2 - N5IpSRYjj6PC0q4 [18/Jul/2008:14:54:16 -0600] "GET
http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelist
HTTP/1.1" 200 0
C: *
C: proxy 66.158.45.2 - N5IpSRYjj6PC0q4 [18/Jul/2008:14:59:29 -0600] "GET
http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelist
HTTP/1.1" 200 0
C: *
C: proxy 66.158.45.2 - 6sjKkVq74O8W7qZ [18/Jul/2008:15:21:22 -0600] "GET
http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelistk8
HTTP/1.1" 200 0
C: *
C: proxy 66.158.45.2 - deRkpGPltRsoYFP [18/Jul/2008:15:48:30 -0600] "GET
http://proquest.umi.com/login HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.196 - RmvHR9lpYXIJyDT [19/Jul/2008:09:03:05 -0600]
"GET
http://proquest.umi.com/pqdweb?SQ=%22Nick+Nitti%22&DBId=15108&onDate=&before
Date=&fromDate=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax
=PQ&searchInterface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&
clientId=20908&JSEnabled=1 HTTP/1.1" 200 0
C: *
C: proxy 205.188.117.15 - RmvHR9lpYXIJyDT [19/Jul/2008:09:05:57 -0600]
"POST
http://proquest.umi.com/pqdweb?SQ=%22Nick+Nitti%22&DBId=15108&onDate=&before
Date=&fromDate=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax
=PQ&searchInterface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&
clientId=20908&JSEnabled=1 HTTP/1.1" 200 0
C: *
C: proxy 205.188.117.139 - RmvHR9lpYXIJyDT [19/Jul/2008:09:06:25 -0600]
"GET
http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.c
om?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.205 - RmvHR9lpYXIJyDT [19/Jul/2008:09:06:25 -0600]
"GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.13 - Cjm6To7u638jzSX [19/Jul/2008:17:28:44 -0600]
"GET
http://proquest.umi.com/pqdweb?SQ=Yonkee&DBId=15108&onDate=&beforeDate=&from
Date=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax=PQ&search
Interface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&clientId=2
0908&JSEnabled=1 HTTP/1.1" 200 0
C: *
C: proxy 205.188.117.139 - Cjm6To7u638jzSX [19/Jul/2008:17:29:44 -0600]
"GET
http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.c
om?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.205 - Cjm6To7u638jzSX [19/Jul/2008:17:29:45 -0600]
"GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.117.134 - SIB7ja516kjANPa [21/Jul/2008:11:21:21 -0600]
"GET
http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.c
om?db=CTRB HTTP/1.1" 200 0
C: *
C: proxy 205.188.116.205 - SIB7ja516kjANPa [21/Jul/2008:11:21:21 -0600]
"GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0
C: *
F: Closing logfile /usr/local/analog-6.0.3/logfile.log
S: Successful requests: 52
S: Redirected requests: 21
S: Failed requests: 0
S: Requests returning informational status code: 0
S: Status code not given: 0
S: Unwanted lines: 0
S: Corrupt lines: 16
./analog: Warning L: Large number of corrupt lines in logfile
/usr/local/analog-6.0.3/logfile.log: turn debugging on or try different
LOGFORMAT
(For help on all errors and warnings, see docs/errors.html)
Current logfile format:
%j %S %j %j [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b\n
S: Earliest entry in logfile: 16/Jul/08:1147
S: Latest entry in logfile: 23/Jul/08:0919
F: Opening /usr/local/apache2/htdocs/ob_proxyStats.html as output file
F: Opening /usr/local/apache2/htdocs/org.png as pie chart file
F: Closing /usr/local/apache2/htdocs/org.png
./analog: Warning R: In Request Report, turning off pie chart with no wedges
F: Closing /usr/local/apache2/htdocs/ob_proxyStats.html

To me, it looks like the logfiles marked as "corrupt" are in the same format
as the logfiles that are not corrupt:

Marked as corrupted:
C: proxy 192.168.1.130 - E37HRQKiv0dANot [18/Jul/2008:14:53:48 -0600] "GET
http://infoweb.newsbank.com?db=OBIT HTTP/1.1" 200 0

Not marked as corrupted:
proxy 64.12.117.9 - CGATZNblYc4mtcZ [17/Jul/2008:00:09:34 -0600] "GET
http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

I've attached the test logfile that I have been using to generate these
errors.


Again, any help whatsoever would be appreciated.
RE: Corrupt log entries [ In reply to ]
I don't know if it got modified in the email, but it looks like there are extra spaces around the IP address in the corrupt line. In your LOGFORMAT you might try using %w instead of a space as a separator.

--

Jeremy Wadsack
Seven Simple Machines

________________________________
From: analog-help-bounces@lists.meer.net [mailto:analog-help-bounces@lists.meer.net] On Behalf Of Brian Clanton
Sent: Friday, July 25, 2008 12:29 PM
To: analog-author@lists.meer.net
Cc: analog-help@lists.meer.net
Subject: [analog-help] Corrupt log entries


I have submitted this question to the analog forum as well as going through the documentation for about a week now and I am no closer to finding a resolution. Any help you may be able to give would be much appreciated.

I am running an Ezrproxy server (Fedora 8) and I want Analog (Same Fedora 8 Server) to provide stats based on the Ezproxy logs.

The designated Ezproxy log format is:
LogSPU spu.log %{ezproxy-spuaccess}i %h %l %u %t "%r" %s %b

Here are a few logs in the "spu.log" file:
./analog: analog version 6.0/Unix
F: Closing configuration file /usr/local/analog-6.0.3/analog.cfg
F: Opening /usr/local/analog-6.0.3/lang/uk.lng as language file
F: Closing language file /usr/local/analog-6.0.3/lang/uk.lng
F: Opening /usr/local/analog-6.0.3/lang/ukdom.tab as domains file
F: Closing domains file /usr/local/analog-6.0.3/lang/ukdom.tab
F: Opening /usr/local/analog-6.0.3/lang/ukdesc.txt as report descriptions file
F: Closing report descriptions file /usr/local/analog-6.0.3/lang/ukdesc.txt
F: Opening /usr/local/analog-6.0.3/logfile.log as logfile
C: proxy 205.188.116.15 - gNDs4AexHJMk3B9 [15/Jul/2008:21:04:33 -0600] "GET http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.205 - gNDs4AexHJMk3B9 [15/Jul/2008:21:04:33 -0600] "GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 192.168.1.130 - E37HRQKiv0dANot [18/Jul/2008:14:53:48 -0600] "GET http://infoweb.newsbank.com?db=OBIT HTTP/1.1" 200 0

C: *
C: proxy 66.158.45.2 - N5IpSRYjj6PC0q4 [18/Jul/2008:14:54:16 -0600] "GET http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelist HTTP/1.1" 200 0

C: *
C: proxy 66.158.45.2 - N5IpSRYjj6PC0q4 [18/Jul/2008:14:59:29 -0600] "GET http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelist HTTP/1.1" 200 0

C: *
C: proxy 66.158.45.2 - 6sjKkVq74O8W7qZ [18/Jul/2008:15:21:22 -0600] "GET http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelistk8 HTTP/1.1" 200 0

C: *
C: proxy 66.158.45.2 - deRkpGPltRsoYFP [18/Jul/2008:15:48:30 -0600] "GET http://proquest.umi.com/login HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.196 - RmvHR9lpYXIJyDT [19/Jul/2008:09:03:05 -0600] "GET http://proquest.umi.com/pqdweb?SQ=%22Nick+Nitti%22&DBId=15108&onDate=&beforeDate=&fromDate=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax=PQ&searchInterface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&clientId=20908&JSEnabled=1 HTTP/1.1" 200 0

C: *
C: proxy 205.188.117.15 - RmvHR9lpYXIJyDT [19/Jul/2008:09:05:57 -0600] "POST http://proquest.umi.com/pqdweb?SQ=%22Nick+Nitti%22&DBId=15108&onDate=&beforeDate=&fromDate=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax=PQ&searchInterface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&clientId=20908&JSEnabled=1 HTTP/1.1" 200 0

C: *
C: proxy 205.188.117.139 - RmvHR9lpYXIJyDT [19/Jul/2008:09:06:25 -0600] "GET http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.205 - RmvHR9lpYXIJyDT [19/Jul/2008:09:06:25 -0600] "GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.13 - Cjm6To7u638jzSX [19/Jul/2008:17:28:44 -0600] "GET http://proquest.umi.com/pqdweb?SQ=Yonkee&DBId=15108&onDate=&beforeDate=&fromDate=&toDate=&FT=1&AT=any&author=&sortby=CHRON&RQT=305&querySyntax=PQ&searchInterface=1&moreOptState=CLOSED&TS=1215778558&h_pubtitle=&h_pmid=&clientId=20908&JSEnabled=1 HTTP/1.1" 200 0

C: *
C: proxy 205.188.117.139 - Cjm6To7u638jzSX [19/Jul/2008:17:29:44 -0600] "GET http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.205 - Cjm6To7u638jzSX [19/Jul/2008:17:29:45 -0600] "GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.117.134 - SIB7ja516kjANPa [21/Jul/2008:11:21:21 -0600] "GET http://proxy-ob.dupagels.lib.il.us:18457/login?url=http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
C: proxy 205.188.116.205 - SIB7ja516kjANPa [21/Jul/2008:11:21:21 -0600] "GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

C: *
F: Closing logfile /usr/local/analog-6.0.3/logfile.log
S: Successful requests: 52
S: Redirected requests: 21
S: Failed requests: 0
S: Requests returning informational status code: 0
S: Status code not given: 0
S: Unwanted lines: 0
S: Corrupt lines: 16
./analog: Warning L: Large number of corrupt lines in logfile
/usr/local/analog-6.0.3/logfile.log: turn debugging on or try different
LOGFORMAT
(For help on all errors and warnings, see docs/errors.html)
Current logfile format:
%j %S %j %j [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b\n
S: Earliest entry in logfile: 16/Jul/08:1147
S: Latest entry in logfile: 23/Jul/08:0919
F: Opening /usr/local/apache2/htdocs/ob_proxyStats.html as output file
F: Opening /usr/local/apache2/htdocs/org.png as pie chart file
F: Closing /usr/local/apache2/htdocs/org.png
./analog: Warning R: In Request Report, turning off pie chart with no wedges
F: Closing /usr/local/apache2/htdocs/ob_proxyStats.html

To me, it looks like the logfiles marked as "corrupt" are in the same format as the logfiles that are not corrupt:

Marked as corrupted:
C: proxy 192.168.1.130 - E37HRQKiv0dANot [18/Jul/2008:14:53:48 -0600] "GET http://infoweb.newsbank.com?db=OBIT HTTP/1.1" 200 0

Not marked as corrupted:
proxy 64.12.117.9 - CGATZNblYc4mtcZ [17/Jul/2008:00:09:34 -0600] "GET http://infoweb.newsbank.com?db=CTRB HTTP/1.1" 200 0

I've attached the test logfile that I have been using to generate these errors.
<<...>>

Again, any help whatsoever would be appreciated.