Mailing List Archive

Processing Past Log Files
I have log files that have not been processed being that they're too large...4GB per day. Is there a standard operation procedure to processing past log files stored on the server?
Thanks,
KUSH
Re: Processing Past Log Files [ In reply to ]
As I was about to send this reply, I noticed that you had addressed your
e-mail directly to my address, and that of some other contributors. I
consider that to be an abuse of the voluntary support process that the
analog-help mailing list facilitates. Nobody is entitled to any response
or support from any subscriber to this list, and if someone has not
responded because they are too busy, or are away, or because they have
decided to unsubscribe from the list, that's their business, and sending
direct e-mail to individual responders on this list as well as to the
list is essentially spam.


On 7/28/2008 11:36 PM, Kush wrote:
>
> I have log files that have not been processed being that they're too
> large...4GB per day. Is there a standard operation procedure to
> processing past log files stored on the server?

Analogs "Cache" mode essentially summarises the information in a log
file, so that you can generate reports from these summaries. This allows
you to generate reports from sets of log files that would require
vastly more memory than Analog would be able to handle.

It's important to understand that the caching mode works by discarding a
great deal of "linking" information, so that you while you can tell that
X Hosts visted your sites, and that there were Y requests for a
particular page on your site, you can't can't use a cache file to tell
how many hosts requested just that particular page - you'd have to go
back to the original log files to find that out.

The other important thing to bear in mind about cache files is that they
can only generate reliable reports for the data that they were designed
to capture. That means that you must generate your report design (the
.cfg files that control which reports are turned on, what the FLOORs and
SUBFLOORs are, what is INCLUDEd and EXCLUDEd, etc) before generating
your cache files. If you subsequently decide that you need to change the
.cfg files, you may need to recreate the cache files for all the logs.

Analog Cache files can help you generate reports from very large log
files, but they can be complex, and you can make mistakes if you're not
careful.
http://analog.cx/docs/cache.html

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Processing Past Log Files [ In reply to ]
OK, my apologies. Can you assist me with the issue?



----- Original Message ----
From: Aengus <analog07@eircom.net>
To: Support for analog web log analyzer <analog-help@lists.meer.net>
Sent: Tuesday, July 29, 2008 7:27:14 AM
Subject: Re: [analog-help] Processing Past Log Files

As I was about to send this reply, I noticed that you had addressed your
e-mail directly to my address, and that of some other contributors. I
consider that to be an abuse of the voluntary support process that the
analog-help mailing list facilitates. Nobody is entitled to any response
or support from any subscriber to this list, and if someone has not
responded because they are too busy, or are away, or because they have
decided to unsubscribe from the list, that's their business, and sending
direct e-mail to individual responders on this list as well as to the
list is essentially spam.


On 7/28/2008 11:36 PM, Kush wrote:
>
> I have log files that have not been processed being that they're too
> large...4GB per day. Is there a standard operation procedure to
> processing past log files stored on the server?

Analogs "Cache" mode essentially summarises the information in a log
file, so that you can generate reports from these summaries. This allows
  you to generate reports from sets of log files that would require
vastly more memory than Analog would be able to handle.

It's important to understand that the caching mode works by discarding a
great deal of "linking" information, so that you while you can tell that
X Hosts visted your sites, and that there were Y requests for a
particular page on your site, you can't can't use a cache file to tell
how many hosts requested just that particular page - you'd have to go
back to the original log files to find that out.

The other important thing to bear in mind about cache files is that they
can only generate reliable reports for the data that they were designed
to capture. That means that you must generate your report design (the
.cfg files that control which reports are turned on, what the FLOORs and
SUBFLOORs are, what is INCLUDEd and EXCLUDEd, etc) before generating
your cache files. If you subsequently decide that you need to change the
  .cfg files, you may need to recreate the cache files for all the logs.

Analog Cache files can help you generate reports from very large log
files, but they can be complex, and you can make mistakes if you're not
careful.
http://analog.cx/docs/cache.html

Aengus
+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.meer.net/mailman/listinfo/analog-help
|
|  Analog Documentation: http://analog.cx/docs/Readme.html
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------