Mailing List Archive

On Wednesday, November 28, 2007 1:56 AM [EDT],
Ridwan <ridwan@chemwatch.net> wrote:

> I would like to create a report from an IIS web server log files for
> only the lines which contain a certain pattern of strings in them,
> like sydneyports. Help please?

Without a bit more details, the best suggestion I can make would be to try
FILEINCLUDE *sydneyports*

http://analog.cx/docs/include.html

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Hi Anegus,

Thanks for your response. Sorry I should have been more specific. In my
IIS log file, among other logging properties I also log the "URI Query
(cs-uri-query)" which basically is the web request URLs sent back from
the clients to the web server. For example, in the IIS log, URI Query
logged for a line is the string:
"operation=checkpass&user=mnphmf&pwd=blahblah&passdirect=y" and for my
Analog report I want to take into account only the lines from that IIS
log file which has the word "mnphmf" in them or may be "user=mnphmf" in
them. This way I would know exactly how many times they logged into my
server. By the way, we run CGI based web applications on our web server
and hence most of the activities are not pulling up just static files,
they are rather web requests as URLs from the clients sent to the server
which executes a .exe CGI program with parameters at the server and it
sends back a dynamic HTML page as streams to the client. Its (CGI) a
very old technology but works well for our heavily database driven
applications.

regards
Ridwan

Aengus wrote:
> On Wednesday, November 28, 2007 1:56 AM [EDT],
> Ridwan <ridwan@chemwatch.net> wrote:
>
>> I would like to create a report from an IIS web server log files for
>> only the lines which contain a certain pattern of strings in them,
>> like sydneyports. Help please?
>
> Without a bit more details, the best suggestion I can make would be to
> try FILEINCLUDE *sydneyports*
>
> http://analog.cx/docs/include.html
>
> Aengus
>
> _____________________________________________________________________
>
> This message has been checked for all known viruses by the MessageLabs
> Virus Scanning Service. For further information visit
> http://www.Hi-Speed.net.au
> ________________________________________________________________________


________________________________________________________________________

This email has been scanned for all viruses by the MessageLabs SkyScan
service. For more information on a proactive anti-virus service working
around the clock, around the globe, visit http://www.hi-speed.net.au
________________________________________________________________________
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

On Wednesday, November 28, 2007 6:43 PM [EDT],
Ridwan <ridwan@chemwatch.net> wrote:

> Hi Anegus,
>
> Thanks for your response. Sorry I should have been more specific. In
> my IIS log file, among other logging properties I also log the "URI
> Query (cs-uri-query)" which basically is the web request URLs sent
> back from the clients to the web server. For example, in the IIS log,
> URI Query logged for a line is the string:
> "operation=checkpass&user=mnphmf&pwd=blahblah&passdirect=y" and for my
> Analog report I want to take into account only the lines from that IIS
> log file which has the word "mnphmf" in them or may be "user=mnphmf"
> in them. This way I would know exactly how many times they logged
> into my server. By the way, we run CGI based web applications on our
> web server and hence most of the activities are not pulling up just
> static files, they are rather web requests as URLs from the clients
> sent to the server which executes a .exe CGI program with parameters
> at the server and it sends back a dynamic HTML page as streams to the
> client. Its (CGI) a very old technology but works well for our
> heavily database driven applications.

I'm sure that there's a better way, but the first thing that coes to mind is
a slightly convoluted process using a FILEALIAS.

FILEALIAS *.cgi?*mnphmf* $1.abc?$2mnphmf$3

will take all calls to your script and convert the script extension to .abc,
if and only if that particular line includes the string you care about. Then
if you use

FILEINCLUDE *.abc*

analog will report on just those lines.

(By the way, I'm subscribed to the list. Just reply to the list, if you cc
me, I end up with 2 copies of your message, and I only need 1).

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

On 28/11/2007, Aengus <analog07@eircom.net> wrote:
> On Wednesday, November 28, 2007 6:43 PM [EDT],
> Ridwan <ridwan@chemwatch.net> wrote:
>

> I'm sure that there's a better way, but the first thing that coes to mind is
> a slightly convoluted process using a FILEALIAS.
>
> FILEALIAS *.cgi?*mnphmf* $1.abc?$2mnphmf$3
>
> will take all calls to your script and convert the script extension to .abc,
> if and only if that particular line includes the string you care about. Then
> if you use
>
> FILEINCLUDE *.abc*
>
> analog will report on just those lines.
>

I think you're interpreting the question differently from me, Aengus.
For finding a specific user,
FILEINCLUDE *mnphmf*
should work fine.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Stephen Turner <analog-author@lists.meer.net> wrote:
> On 28/11/2007, Aengus <analog07@eircom.net> wrote:
>> On Wednesday, November 28, 2007 6:43 PM [EDT],
>> Ridwan <ridwan@chemwatch.net> wrote:
>>
>
>> I'm sure that there's a better way, but the first thing that coes to
>> mind is a slightly convoluted process using a FILEALIAS.
>>
>> FILEALIAS *.cgi?*mnphmf* $1.abc?$2mnphmf$3
>>
>> will take all calls to your script and convert the script extension
>> to .abc, if and only if that particular line includes the string you
>> care about. Then if you use
>>
>> FILEINCLUDE *.abc*
>>
>> analog will report on just those lines.
>>
>
> I think you're interpreting the question differently from me, Aengus.
> For finding a specific user,
> FILEINCLUDE *mnphmf*
> should work fine.

That's what I thought - but it didn't work that way for me.

The documenation for REQINCLUDE in http://analog.cx/docs/args.html says "in the Request Report files with arguments are only included if their parent file is included". Even though this shouldn't directly apply to FILEINCLUDE (because the stem of a query is always displayed in the Request Report, even if it only occurs with a query string), it looks as though this is coming into play here.

Here's a one line log file:
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-09-19 00:00:19
#Fields: date time c-ip cs-method cs-uri-stem cs-uri-query sc-status
2007-09-19 00:16:20 127.0.0.1 GET /test/test.asp PageNo=2&Order=DESC&Display=0 200

If I call analog with these parameters, I get a Request Report that shows the Request, with the args listed on a 2nd line:
analog test.log +C"reqfloor 1r" +C"reqargsfloor 1r"

If I call analog with this additional parameter, I get an empty report:
analog test.log +C"reqfloor 1r" +C"reqargsfloor 1r" +C"FILEINCLUDE *DESC*"

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

On 29/11/2007, Aengus <analog07@eircom.net> wrote:
>
> Here's a one line log file:
> #Software: Microsoft Internet Information Services 6.0
> #Version: 1.0
> #Date: 2007-09-19 00:00:19
> #Fields: date time c-ip cs-method cs-uri-stem cs-uri-query sc-status
> 2007-09-19 00:16:20 127.0.0.1 GET /test/test.asp PageNo=2&Order=DESC&Display=0 200
>
> If I call analog with these parameters, I get a Request Report that shows the Request, with the args listed on a 2nd line:
> analog test.log +C"reqfloor 1r" +C"reqargsfloor 1r"
>
> If I call analog with this additional parameter, I get an empty report:
> analog test.log +C"reqfloor 1r" +C"reqargsfloor 1r" +C"FILEINCLUDE *DESC*"
>

It turns out this is a problem with case insensitivity, though it's
less clear what to do about it. If you're on a case insensitive
filesystem (such as Windows), all the filenames are lower-cased, and
so all the FILEINCLUDEs are lower-cased too to match. That way *test*
and *TEST* will both match.

The problem is that the lower casing only applies to the stem of the
URL, not the query string; but analog doesn't know that the
FILEINCLUDE is targetted at the query string, so it's looking for
"desc" and not finding it.

A workaround is to use
FILEINCLUDE REGEXP:DESC
That still does a case insensitive match, but the case insensitivity
is handled by the regexp engine not by folding the specification to
lower case, so it will match.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

Stephen Turner <analog-author@lists.meer.net> wrote:
> It turns out this is a problem with case insensitivity, though it's
> less clear what to do about it. If you're on a case insensitive
> filesystem (such as Windows), all the filenames are lower-cased, and
> so all the FILEINCLUDEs are lower-cased too to match. That way *test*
> and *TEST* will both match.
>
> The problem is that the lower casing only applies to the stem of the
> URL, not the query string; but analog doesn't know that the
> FILEINCLUDE is targetted at the query string, so it's looking for
> "desc" and not finding it.
>
> A workaround is to use
> FILEINCLUDE REGEXP:DESC
> That still does a case insensitive match, but the case insensitivity
> is handled by the regexp engine not by folding the specification to
> lower case, so it will match.

Wow!

:-)

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------