Mailing List Archive

Analog and EZproxy
Hello Edward,

I saw your posting on the EZproxy mailing list and was wondering if you
could help me out.

I took a look at your stats below and this is similar to the types of
reports I'd like to create from our EZproxy log files. Log files and SPU
logs are generated on a monthly basis on our EZproxy server (which is a
Windows server). I downloaded Analog 6.0 on a PC running Windows XP for
the purposing of generating usage statistics; I'm new to both
applications, especially Analog.

I'm writing because I'm wondering how you were able to get Analog to
work. I tried running it on a couple of smaller log files to test it out
by running the following command: analog ezp200801.log

But got the following error message:

analog: analog version 6.0/Win32
analog: Warning E: Redirecting future diagnostic messages to errors1.txt
(For help on all errors and warnings, see docs/errors.html)

I have the following entered into the EZproxy configuration file:

## Logfile directives added 2006-04-04, LogSPU added 2006-05-18

LogFile -strftime ezp%Y%m.log

LogFormat %h %{ezproxy-session}i %u %t "%r" %s %b %{referer}i

LogSPU -strftime spu%Y%m.log %h %{ezproxy-session}i %u %t "%r" %s %b
%{ezproxy-groups}i %v %{referer}i



I attached my error report but I'm starting to thinking the analog
configuration file (analog.cfg) might need to be adjusted; actually, I'm
not sure where to begin. Can you give me some pointers?



Thank you so much in advance,



Tammy


Tammy K. Baker, Technology Librarian
Automated Library Services
District Services and Training Center
North Harris Montgomery Community College District
5000 Research Forest Drive
The Woodlands, Texas 77381-4356 USA

Telephone: (832) 813-6652
Facsimile: (832) 813-6614
Electronic mail: tammy.k.baker@nhmccd.edu
<mailto:tammy.k.baker@nhmccd.edu>

NHMCCD is becoming
Lone Star College System

-----Original Message-----

From: Edward Spodick [mailto:lbspodic@ust.hk <mailto:lbspodic@ust.hk> ]

Sent: Thursday, January 10, 2008 7:31 PM

To: EZProxy discussion list

Subject: Re: [ezproxy] Logs and Reporting Tools



We also use Analog, especially because of its convenient FILEALIAS
command, which helps me in grouping database information. Our log files
are about 300MB on average and they process fast with no problems on a
Linux box - about 2-3 minutes.

The cumulative report processes 1.1 GB of gzipped log files - many GB of
ungzipped data, but analog processes them from GZ format directly. The
most recent cumulative run took just under 70 minutes on a dual-CPU
Linux server w/2GB of RAM.

You can see some of our stats at:

http://library.ust.hk/usage/ezp/monthly/
<http://library.ust.hk/usage/ezp/monthly/>

http://library.ust.hk/usage/ezp/cumulative/
<http://library.ust.hk/usage/ezp/cumulative/>

We have several other reports generated, but those are not available
outside of out IT section as most include usernames or
departmental/usergropu breakdowns, etc.

-Edward

At 6:19 PM -0500 1/10/08, Ian.Hertz@esc.edu wrote:

>I use Analog, a free, customizable reporting tool to create html
reports on ezproxy.log files. Each month the log is approximately .5GB
in size. It takes around 2 minutes for this to run. I want to run them
on a quarterly basis (so I used some fancy DOS commands to combine
files). Is there some other, better, easier or ezproxy way to do this?

>So now, obviously, I have a file size of approximately 1.5GB. When I
run the report, 16 hours later, I still have no results and of course it
kills my machine (not a server). Can some give feedback on their
approximate file sizes? I'm sure after some of the more recent posts,
I'm going to be advised to choose a different tool, however, I was going
with free. Any feedback is appreciated.

--

Edward F Spodick, Information Technology Manager Hong Kong University of
Science & Technology Library lbspodic@ust.hk tel:852-2358-6743
fax:852-2358-1043
Re: Analog and EZproxy [ In reply to ]
Baker, Tammy K <Tammy.K.Baker@nhmccd.edu> wrote:
>
> I'm writing because I'm wondering how you were able to get Analog to
> work. I tried running it on a couple of smaller log files to test it
> out by running the following command: analog ezp200801.log
>
> But got the following error message:
>
> analog: analog version 6.0/Win32
> analog: Warning E: Redirecting future diagnostic messages to
> errors1.txt (For help on all errors and warnings, see
> docs/errors.html)

That's a warning, not an error - it just says that it's going to write any error messages to a file instead of writing them to the screen. If you want them to write to the screen, comment the ERRFILE command out of the analog.cfg file.

The output that was in the error file is

Warning L: Large number of corrupt lines in logfile ezp200801.log:
turn debugging on or try different LOGFORMAT

That basically means that Analog doesn't recognize the data in your logfile - it can't tell what each field means.

> I have the following entered into the EZproxy configuration file:
>
> ## Logfile directives added 2006-04-04, LogSPU added 2006-05-18
>
> LogFile -strftime ezp%Y%m.log
>
> LogFormat %h %{ezproxy-session}i %u %t "%r" %s %b %{referer}i
>
> LogSPU -strftime spu%Y%m.log %h %{ezproxy-session}i %u %t "%r" %s %b
> %{ezproxy-groups}i %v %{referer}i

This is the Analog list, not the EZProxy list, so I have no idea what LOGSPU means, but that LogFormat line wou;d probably correspond to something like this in the analog.cfg file:

LOGFORMAT (%S %c %u [%d/%M/%Y:%h:%n:%j] "%j %r HTTP%j" %c %b %f)

(Note that Analog wants the date/time field spelled out - I'm just guessing at what the %t in EZProxy might actually generate).

> I attached my error report but I'm starting to thinking the analog
> configuration file (analog.cfg) might need to be adjusted; actually,
> I'm not sure where to begin. Can you give me some pointers?

Add
LOGFORMAT (%S %c %u [%d/%M/%Y:%h:%n:%j] "%j %r HTTP%j" %c %b %f)
LOGFILE ezp200801.log

When Analog completes, open the file Report.html that it generates to see the result. If you're still not getting any output from Analog, post 2 or 3 lines from your logfile, so that we can tweak the LOGFORMAT command that you need to put in your Analog.cfg file.

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Analog and EZproxy [ In reply to ]
At 1:35 PM -0600 1/11/08, Baker, Tammy K wrote:
>I'm writing because I'm wondering how you were able to get Analog to work. I tried running it on a couple of smaller log files to test it out by running the following command: analog ezp200801.log

I think Aengus replied well on that and the logfile issue.


>## Logfile directives added 2006-04-04, LogSPU added 2006-05-18
>LogFile -strftime ezp%Y%m.log
>LogFormat %h %{ezproxy-session}i %u %t "%r" %s %b %{referer}i
>LogSPU -strftime spu%Y%m.log %h %{ezproxy-session}i %u %t "%r" %s %b %{ezproxy-groups}i %v %{referer}i


Well, we use EZProxy's default log format, which is handled well by Analog. I have considered using the LogSPU to create a second log file with a different subset/modset of data, but have not done so yet. Since you are modifying the logformat for the default file, you will need to modify the logformat command for Analog to match, as Aengus pointed out. You can also refer to
http://analog.cx/docs/logfmt.html
for more details on that.


>I attached my error report but I'm starting to thinking the analog configuration file (analog.cfg) might need to be adjusted; actually, I'm not sure where to begin. Can you give me some pointers?

Start exploring the logformat command options, and aengus' example. Then see what information Analog's output adds to your understanding or confusion. :) This list is full of very helpful people who can assist in working through the final stages of cobwebs if needed.

Oh, and once you start getting functional information from Analog, you might be interested in a few of the specific settings I used for our Analog-generated EZProxy stats reports - I will be glad to share config files with you..

-Edward

--
Edward F Spodick, Information Technology Manager
Hong Kong University of Science & Technology Library
lbspodic@ust.hk tel:852-2358-6743 fax:852-2358-1043
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
RE: Analog and Ezproxy [ In reply to ]
Analog can match the four-digit year using the %Y code:

LOGFILE ezp%Y%M%D

See http://analog.cx/docs/logfile.html for details on the command options.

--

Jeremy Wadsack
Seven Simple Machines

-----Original Message-----
From: analog-help-bounces@lists.meer.net [mailto:analog-help-bounces@lists.meer.net] On Behalf Of Shannon E. Fox
Sent: Thursday, July 10, 2008 11:06 AM
To: analog-help@lists.meer.net
Cc: ezproxy@ls.suny.edu
Subject: [analog-help] Analog and Ezproxy

I want to use Analog to analyze my Ezproxy stats. I have entered the
following command into the ezproxy.cfg file to set up daily logs that
can then be analyzed monthly:

LogFile -strftime log/ezp%Y%m%d.log

The problem is that this command is generating file names that include
the entire year, rather then the appended '08, and Analog doesn't like
that.

Does anyone know how I should amend the command line above so that my
log file name will look like

ezp080701

instead of

ezp20080701

?

Thank you.

Shannon Fox
Coordinator of Electronic Services
Abell Library, Suite 6L
Austin College
900 N. Grand Ave.
Sherman, TX 75090-4400
(903) 813-2559
sfox@austincollege.edu


+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Analog and Ezproxy [ In reply to ]
Shannon E. Fox <sfox@austincollege.edu> wrote:
> I want to use Analog to analyze my Ezproxy stats. I have entered the
> following command into the ezproxy.cfg file to set up daily logs that
> can then be analyzed monthly:
>
> LogFile -strftime log/ezp%Y%m%d.log
>
> The problem is that this command is generating file names that include
> the entire year, rather then the appended '08, and Analog doesn't like
> that.
>
> Does anyone know how I should amend the command line above so that my
> log file name will look like
>
> ezp080701
>
> instead of
>
> ezp20080701

????

Analog doesn't care what your logfiles are called.

If you're using codes in your Analog LOGFILE command, there's a code for a 4 digit year as well as a 2 digit year. Assuming you're using

LOGFILE log/ezp%y%M%D.log

in Analog now, just change it to

LOGFILE log/ezp%Y%M%D.log

(uppercase Y for 4-digit year).

http://analog.cx/docs/logfile.html#datecodes

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
RE: RE: Analog and Ezproxy [ In reply to ]
Jeremy,

Thank you for your response. I appreciate it.
Shannon

-----Original Message-----
From: analog-help-bounces@lists.meer.net [mailto:analog-help-bounces@lists.meer.net] On Behalf Of Jeremy Wadsack
Sent: Thursday, July 10, 2008 2:41 PM
To: Support for analog web log analyzer
Subject: [analog-help] RE: Analog and Ezproxy

Analog can match the four-digit year using the %Y code:

LOGFILE ezp%Y%M%D

See http://analog.cx/docs/logfile.html for details on the command options.

--

Jeremy Wadsack
Seven Simple Machines

-----Original Message-----
From: analog-help-bounces@lists.meer.net [mailto:analog-help-bounces@lists.meer.net] On Behalf Of Shannon E. Fox
Sent: Thursday, July 10, 2008 11:06 AM
To: analog-help@lists.meer.net
Cc: ezproxy@ls.suny.edu
Subject: [analog-help] Analog and Ezproxy

I want to use Analog to analyze my Ezproxy stats. I have entered the
following command into the ezproxy.cfg file to set up daily logs that
can then be analyzed monthly:

LogFile -strftime log/ezp%Y%m%d.log

The problem is that this command is generating file names that include
the entire year, rather then the appended '08, and Analog doesn't like
that.

Does anyone know how I should amend the command line above so that my
log file name will look like

ezp080701

instead of

ezp20080701

?

Thank you.

Shannon Fox
Coordinator of Electronic Services
Abell Library, Suite 6L
Austin College
900 N. Grand Ave.
Sherman, TX 75090-4400
(903) 813-2559
sfox@austincollege.edu


+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Analog and Ezproxy [ In reply to ]
Shannon E. Fox <sfox@austincollege.edu> wrote:
>
> Try as I might, I cannot get Analog to analyze the files. I'm not sure
> what I am doing wrong. I've tried variations on this configuration. I
> have looked at the logfiles and they seem to be okay. I've looked at
> documentation for Analog and for Ezproxy.
>
> Can anyone help me?

What error messages does Analog display when you run it?

Aengus

-----------------------------------------------------------------
Find the home of your dreams with eircom net property
Sign up for email alerts now http://www.eircom.net/propertyalerts


+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
RE: Analog and Ezproxy [ In reply to ]
It doesn't display any error message. It just stays blank.

Thanks,
Shannon

-----Original Message-----
From: analog-help-bounces@lists.meer.net
[mailto:analog-help-bounces@lists.meer.net] On Behalf Of
analog07@eircom.net
Sent: Wednesday, February 18, 2009 3:53 PM
To: Support for analog web log analyzer
Subject: Re: [analog-help] Analog and Ezproxy

Shannon E. Fox <sfox@austincollege.edu> wrote:
>
> Try as I might, I cannot get Analog to analyze the files. I'm not sure
> what I am doing wrong. I've tried variations on this configuration. I
> have looked at the logfiles and they seem to be okay. I've looked at
> documentation for Analog and for Ezproxy.
>
> Can anyone help me?

What error messages does Analog display when you run it?

Aengus

-----------------------------------------------------------------
Find the home of your dreams with eircom net property
Sign up for email alerts now http://www.eircom.net/propertyalerts


+-----------------------------------------------------------------------
-
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+-----------------------------------------------------------------------
-

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
RE: Analog and Ezproxy [ In reply to ]
Support for analog web log analyzer <analog-help@lists.meer.net> wrote:

< It doesn't display any error message. It just stays blank.

Analog is a command line application. If there are any warnings or errors, they will be displayed. But if you're running Analog on Windows and just double-clicking on the Analog icon, the window with the warnings will close before you can read it.

The simplest solution is to open a command prompt (Start/Run CMD), CD to the Analog directory, and run analog at the command line.

You can also add the ERRFILE directive to your analog.cfg, and specify a file that Analog should write these erros or warnings to.

http://analog.cx/docs/debug.html#ERRFILE

Aengus

-----------------------------------------------------------------
Find the home of your dreams with eircom net property
Sign up for email alerts now http://www.eircom.net/propertyalerts


+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------