Mailing List Archive

janet ward wrote:
> I want to exclude a part of the day from my day calculations (this seems
> to be the only way to remove a unidentifiable robot that went through
> during the time to be excluded).
>
> I want to exclude from 11:45 to 11:51.
>
> The FROM and TO parameters appear only to allow me to exclude *before*
> 11:45 and *after* 11:51, whereas I want to exclude *after* 11:45 and
> *before* 11:51.
The only solution I can think of without modifying the code is filtering
the requests before analog processes the log file, either by reading
from stdin and filtering using grep, or by using a filter as a preprocessor.

cat log | grep -v -f file-with-date-time-patterns-to-exclude | analog ...

--
Klaus Johannes Rusch
KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Thank you Klaus Johannes Rusch for your reply. As I said, I am a
programming/coding klutz.

So your reply, which is no doubt very helpful if I could understand it,

<The only solution I can think of without modifying the code is
<filtering the requests before analog processes the log file, either by
<reading from stdin and filtering using grep, or by using a filter as a
<preprocessor.

<cat log | grep -v -f file-with-date-time-patterns-to-exclude | analog <...

means nothing much to me.

Where is stdin, how do I read from it and how do I do a filter?
How would I use a filter (what would that look like?) "as a preprocessor"?

I'm afraid that I am 'drowning' in this strange talk and need a very
large and buoyant life-jacket to help me keep afloat and navigate
through this verbage. Sigh.

Sorry.

JW
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Thank you for your reply. As I said, I am a programming/coding klutz.

So your reply, which is no doubt very helpful if I could understand it,
means nothing much to me.

Where is stdin, how do I read from it and how do I do a filter?
How would I use a filter (what would that look like?) "as a preprocessor"?

I'm afraid that I am 'drowning' in this strange talk and need a very
large and buoyant life-jacket to help me keep afloat and navigate
through this verbage. Sigh.

Sorry.

JW



Klaus Johannes Rusch wrote:
> janet ward wrote:
>> I want to exclude a part of the day from my day calculations (this seems
>> to be the only way to remove a unidentifiable robot that went through
>> during the time to be excluded).
>>
>> I want to exclude from 11:45 to 11:51.
>>
>> The FROM and TO parameters appear only to allow me to exclude *before*
>> 11:45 and *after* 11:51, whereas I want to exclude *after* 11:45 and
>> *before* 11:51.
> The only solution I can think of without modifying the code is filtering
> the requests before analog processes the log file, either by reading
> from stdin and filtering using grep, or by using a filter as a
> preprocessor.
>
> cat log | grep -v -f file-with-date-time-patterns-to-exclude | analog ...
>

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

janet ward <janet-w@wanadoo.fr> wrote:

> I want to exclude a part of the day from my day calculations (this
> seems to be the only way to remove a unidentifiable robot that went
> through during the time to be excluded).

If it's unidentifiable, what makes you think it's a robot? Does it's IP address not identify it?

> I want to exclude from 11:45 to 11:51.
>
> The FROM and TO parameters appear only to allow me to exclude *before*
> 11:45 and *after* 11:51, whereas I want to exclude *after* 11:45 and
> *before* 11:51.
>
> Please would you advise?

You can only have one set of TO/FROM, and the FROM time has to be before the TO time.

If it was for more than a 6 minute window, you could try createing cache files for the 2 periods and combining them, but it seems that simply editing out the 6 minutes would be the simplest way to do what you're trying to do.

Personally, I'd take a much closer look at the data in that 6 minute window. Turn on the Host report (HOST ON), and see if there's a single IP address generating the anomalous data. Then look for that address in the whole log (HOST INCLUDE w.x.y.z) and see if it only occurs in your 6 minute window. If it does, then exclude it (HOSTEXCLUDE w.x.y.z)

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

> You can only have one set of TO/FROM, and the FROM time has to be before the TO time.
>
> If it was for more than a 6 minute window, you could try creating cache files for the 2 periods and combining them, but it seems that simply editing out the 6 minutes would be the simplest way to do what you're trying to do.
>
> Personally, I'd take a much closer look at the data in that 6 minute window. Turn on the Host report (HOST ON), and see if there's a single IP address generating the anomalous data. Then look for that address in the whole log (HOST INCLUDE w.x.y.z) and see if it only occurs in your 6 minute window. If it does, then exclude it (HOSTEXCLUDE w.x.y.z)

This is where I started from - the HOST report showed up *nothing* out
of the ordinary - entries in the region of between 2 and 18 per HOST,
which I why I wanted to exclude what looked like the time span that
included the culprit/s.

In fact, while waiting for (understandable) advice, I have deleted two
sections of entries from my raw log files [.3800 lines from 11:46:01 to
11:50:58, and 670 lines from 14:49:01 to 14:51:49]. This has made the
resulting analog look much more reasonable.

JW
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

janet ward wrote:
> In fact, while waiting for (understandable) advice, I have deleted two
> sections of entries from my raw log files [.3800 lines from 11:46:01 to
> 11:50:58, and 670 lines from 14:49:01 to 14:51:49]. This has made the
> resulting analog look much more reasonable.
... which has the same effect as using grep. Definitely the best choice
as long as the log files are small enough that modifying them in an
editor is an option.

--
Klaus Johannes Rusch
KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------