Mailing List Archive

Corrupt logfile lines
Hi!

I'm a newbie to Apache and Linux (Suse), a new server.

The anlog display:

uccessful requests: 93
Successful requests for pages: 21
Distinct files requested: 62
Distinct hosts served: 7
Corrupt logfile lines: 19
Unwanted logfile entries: 121
Data transferred: 3.08 megabytes

What are corrupt lines?

I found that in the /etc/appache2/mod-log_config.com is the line:

LogFormat "%h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\"" combined

What ca be wrong?
How to fix it?

TIA

Nanu

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Corrupt logfile lines [ In reply to ]
On 2/22/2009 10:28 AM, Nanu Kalmanovitz wrote:
> Hi!
>
> I'm a newbie to Apache and Linux (Suse), a new server.
>
> The anlog display:
>
> uccessful requests: 93
> Successful requests for pages: 21
> Distinct files requested: 62
> Distinct hosts served: 7
> Corrupt logfile lines: 19
> Unwanted logfile entries: 121
> Data transferred: 3.08 megabytes
>
> What are corrupt lines?

Corrupt lines are lines that don't match the format that Analog is using
to parse the logfile.

If you add DEBUG ON to your analog.cfg, analog will output the lines
that it can't parse, with an asterisk under the part that doesn't match.
You might want to use the ERRFILE command to send the output to a file,
rather than to the screen.

http://analog.cx/docs/debug.html#ERRFILE

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Corrupt logfile lines [ In reply to ]
Thanks!

I added the DEBUG ON & ERRFILE.

Following are few segments of the ERRFILE:

D: Looking up 74.6.18.236:
D: resolved to llf520158.crawl.yahoo.net
C: 66.249.71.69 - - [22/Feb/2009:21:10:55 +0200] "GET /K_comm/english/Roots_tree/Pictures/Stu511L.htm HTTP/1.1" 403 590
C: *
D: Looking up 65.55.105.218:
...
D: Looking up 85.65.16.75:
D: resolved to 85.65.16.75.dynamic.barak-online.net
C: ::1 - - [22/Feb/2009:21:59:27 +0200] "GET /" 400 991
C: *
D: Looking up 85.65.25.160:
...
D: resolved to cpe-72-185-234-199.tampabay.res.rr.com
C: 66.249.71.70 - - [22/Feb/2009:23:52:39 +0200] "GET /K_comm/english/Roots_tree/Pictures/Zelma2211Pic.htm HTTP/1.1" 403 590
C: *
D: Looking up 83.103.156.249:
...
D: Looking up 24.148.9.96:
D: resolved to 24-148-9-96.arm-bsr1.chi-arm.il.cable.rcn.com
C: 66.249.71.69 - - [23/Feb/2009:05:24:25 +0200] "GET /robots.txt HTTP/1.1" 403 590
C: *
C: 66.249.71.69 - - [23/Feb/2009:05:24:25 +0200] "GET /K_comm/english/Roots_tree/Branch_0000241.htm HTTP/1.1" 403 590
C: *
D: Looking up 65.55.108.145:
...

I found that in the /etc/appache2/mod-log_config.com is the line:

LogFormat "%h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\"" combined

Please help me to configure and resolve the problem.

TIA

Nanu

>>> Aengus<analog07@eircom.net> 22/02/2009 17:35:11 >>>
On 2/22/2009 10:28 AM, Nanu Kalmanovitz wrote:
> Hi!
>
> I'm a newbie to Apache and Linux (Suse), a new server.
>
> The anlog display:
>
> uccessful requests: 93
> Successful requests for pages: 21
> Distinct files requested: 62
> Distinct hosts served: 7
> Corrupt logfile lines: 19
> Unwanted logfile entries: 121
> Data transferred: 3.08 megabytes
>
> What are corrupt lines?

Corrupt lines are lines that don't match the format that Analog is using
to parse the logfile.

If you add DEBUG ON to your analog.cfg, analog will output the lines
that it can't parse, with an asterisk under the part that doesn't match.
You might want to use the ERRFILE command to send the output to a file,
rather than to the screen.

http://analog.cx/docs/debug.html#ERRFILE

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
The stars are at the end of the line. That's probably because these
lines don't have referrer and browser on the end, and the other lines
in your logfile do? If so, you could write two LOGFORMAT lines to
capture both types of lines, but it would be better to fix your
logfile. At a minimum, make sure you don't have each line being
written twice, once with referrer and browser and once without.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
I'm cpmpletely new and without anyexperience...can u halp me with a step by step instructions?

TIA

Nanu


>>> Stephen Turner<analog-author@lists.meer.net> 23/02/2009 10:22:35 >>>
The stars are at the end of the line. That's probably because these
lines don't have referrer and browser on the end, and the other lines
in your logfile do? If so, you could write two LOGFORMAT lines to
capture both types of lines, but it would be better to fix your
logfile. At a minimum, make sure you don't have each line being
written twice, once with referrer and browser and once without.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
On 2/23/2009 1:20 AM, Nanu Kalmanovitz wrote:

> I found that in the /etc/appache2/mod-log_config.com is the line:
>
> LogFormat "%h %l %u %t \"%r\" %>s %b \
> \"%{Referer}i\" \"%{User-Agent}i\"" combined

Is there a LOGFORMAT (or APACHELOGFORMAT) command in your analog.cfg
file? Analog can recognize many common logfile formats without
specifying a LOGFORMAT command, but it assumes that all the lines in a
logfile match a single format. The "corrupt" lines that you post don't
match the format string from your apache setup. (Note that Analog
doesn't know or care what is set up in your apache configuration files -
it will only use the LOGFORMAT or APACHELOGFORMAT commands in
analog.cfg, or autodetection, if you don't have a LOGFORMAT).

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
In the analog.cfg i have the line LOGFORMAT COMBINE

TIA

Nanu


>>> Aengus<analog07@eircom.net> 23/02/2009 17:14:35 >>>
On 2/23/2009 1:20 AM, Nanu Kalmanovitz wrote:

> I found that in the /etc/appache2/mod-log_config.com is the line:
>
> LogFormat "%h %l %u %t \"%r\" %>s %b \
> \"%{Referer}i\" \"%{User-Agent}i\"" combined

Is there a LOGFORMAT (or APACHELOGFORMAT) command in your analog.cfg
file? Analog can recognize many common logfile formats without
specifying a LOGFORMAT command, but it assumes that all the lines in a
logfile match a single format. The "corrupt" lines that you post don't
match the format string from your apache setup. (Note that Analog
doesn't know or care what is set up in your apache configuration files -
it will only use the LOGFORMAT or APACHELOGFORMAT commands in
analog.cfg, or autodetection, if you don't have a LOGFORMAT).

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Corrupt logfile lines [ In reply to ]
2009/2/24 Nanu Kalmanovitz <nanu@kalmanovitz.co.il>:
> In the analog.cfg i have the line LOGFORMAT COMBINE
>

But the lines you posted were not in COMBINED format because they
don't have referrer and browser on the end. Have you got logfile lines
in more than one format?

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Corrupt logfile lines [ In reply to ]
Hi!

I'm realy confused, since I'm new to Suse, Apache and of course not experienced with Analog terminology.

Maybe the next information can make it easier for you to help me.

I found the followings in the Apache files hierarchy:

1. File: /etc/sysconfig/apache2 contain the line:
APACHE_ACCESS_LOG="/var/log/apache2/access_log combined"
-------------------------------------------------------------

2. In the file /etx/apache2/syscinfig.d/global.conf are the following lines, last line is about access_log:
#
# Files in this directory are created at apache start time by /usr/sbin/rcapache2
# Do not edit them!
#
# see /etc/sysconfig/apache2
Timeout 300
ServerSignature on
UseCanonicalName off
ServerTokens OS
LogLevel warn
CustomLog /var/log/apache2/access_log combined
----------------------------------------------------------------

3. File: mod_log_config.conf are the following lines:
#
# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# http://httpd.apache.org/docs-2.2/mod/mod_log_config.html
#

#
# Format string: Nickname:
#
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
LogFormat "%h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined

# To use %I and %O, you need to enable mod_logio
<IfModule mod_logio.c>
LogFormat "%h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>

# Use one of these when you want a compact non-error SSL logfile on a virtual
# host basis:
<IfModule mod_ssl.c>
Logformat "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \
\"%r\" %b" ssl_common
Logformat "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \
\"%r\" %b \"%{Referer}i\" \"%{User-Agent}i\"" ssl_combined
</IfModule>
-------------------------------------------------------------------------------

4. In the Analog_Error file appears:
#
F: Closing logfile /var/log/apache2/access_log
S: Successful requests: 989
S: Redirected requests: 8
S: Failed requests: 48
S: Requests returning informational status code: 0
S: Status code not given: 0
S: Unwanted lines: 3649
S: Corrupt lines: 563
analog: Warning L: Large number of corrupt lines in logfile
/var/log/apache2/access_log: turn debugging on or try different LOGFORMAT
Current logfile format:
%S %j %j [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b "%f" "%B"\n
%S %j %j [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b "%f" "%B"\n
%S %j %j [%d/%M/%Y:%h:%n:%j] "%r" %c %b "%f" "%B"\n
---------------------------------------------------------------------------------

5. File /etc/analog.cfg contain:
#
DEBUG ON
# LOGFORMAT COMBINED
APACHELOGFORMAT COMBINED

ERRFILE /Applic/HTDocs/Analog_Errors.txt
LOGFILE /var/log/apache2/access_log
OUTFILE /Applic/HTDocs/Analog_Report.html
HOSTNAME "Kalmanovitz Bros. Ltd."
HOSTURL http://www.kalmanovitz.co.il/
-----------------------------------------------------------------------------------

6. Few lines from the /var/log/apache2/access_log file:
C: ::1 - - [24/Feb/2009:21:54:44 +0200] "GET /" 400 991
C: *
C: 66.249.71.70 - - [25/Feb/2009:00:15:32 +0200] "GET /CZ/2006-26/Gac27a.pdf HTTP/1.1" 403 590
C: *
C: 66.249.71.70 - - [25/Feb/2009:05:22:43 +0200] "GET /robots.txt HTTP/1.1" 403 590
C: *
C: 66.249.71.70 - - [25/Feb/2009:05:22:43 +0200] "GET /K_comm/english/Roots_tree/Pictures/rt0051a.jpg HTTP/1.1" 403 590
C: *
C: ::1 - - [25/Feb/2009:05:59:14 +0200] "GET /" 400 991
C:
-------------------------------------------------------------------------------------

Is the above information relevant?
What shall i change and where?

TIA

Nanu

>>> Stephen Turner<analog-author@lists.meer.net> 24/02/2009 13:21:13 >>>
2009/2/24 Nanu Kalmanovitz <nanu@kalmanovitz.co.il>:
> In the analog.cfg i have the line LOGFORMAT COMBINE
>

But the lines you posted were not in COMBINED format because they
don't have referrer and browser on the end. Have you got logfile lines
in more than one format?

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
On 2/25/2009 2:47 AM, Nanu Kalmanovitz wrote:

> What shall i change and where?

Your logfile isn't particularly big. Open it in a text editor and have a
look at the entries. It appears that your web server should be creating
logfile entries that have a Referrer field and a Browser Agent field,
("Combined" format) but the only lines that we've seen are lines that
don't contain those fields. That suggests that your web server is
generating a logfile with 2 different formats. That's an Apache
question, not an Analog question.

Because you've told Analog to use the COMBINED format to read the
logfile, it marks any lines that aren't in COMBINED format as corrupt.

The simple answer is that you can probably add LOGFORMAT COMMON
immediately after your LOGFORMAT COMBINED line in your analog.cfg file,
and Analog will parse both types of lines. But that won't tell you why
your web server is using two different logformats, (or whether it's
recording the same request twice).

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
2009/2/25 Aengus <analog07@eircom.net>:
> On 2/25/2009 2:47 AM, Nanu Kalmanovitz wrote:
>
> The simple answer is that you can probably add LOGFORMAT COMMON immediately
> after your LOGFORMAT COMBINED line in your analog.cfg file, and Analog will
> parse both types of lines. But that won't tell you why your web server is
> using two different logformats, (or whether it's recording the same request
> twice).
>

And if it is recording the same request twice, you really don't want
to do this, or you will double count everything. As Aengus says, you
really need to look at what's in your logfile to diagnose it.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
Hi!

I chated with the Suse\Novell support. They saw the access_log lines:

192.168.254.254 - - [26/Feb/2009:11:13:45 +0200] "GET /req.png HTTP/1.0" 304 - "http://www.kalmanovitz.co.il/Analog_Report.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18) Gecko/20081031 SUSE/2.0.0.18-0.2.1 Firefox/2.0.0.18"
::1 - - [26/Feb/2009:11:13:57 +0200] "GET /" 400 991
::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991
::1 - - [26/Feb/2009:11:13:59 +0200] "GET /" 400 991
::1 - - [26/Feb/2009:11:14:00 +0200] "GET /" 400 991
38.99.13.125 - - [26/Feb/2009:11:14:45 +0200] "GET /k_comm/Israel/English/Maps/Rezervations/EinHemed33/obj/pages/P7270096_jpg.htm HTTP/1.0" 200 2299 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"

They said:

... this is not a second logformat, it is the HTTP status response is 400 "bad request"
request, but rather a port scan, or something similar.
That is why the data about the request is not logged - because there was no HTTP data available, since it wasn't an HTTP request.
That will happen every time someone connects to port 80 on the server via something else than HTTP protocol...

Is it any possibility to configure Analog to interpret the above in a right way?

TIA

Nanu

>>> Stephen Turner<analog-author@lists.meer.net> 25/02/2009 15:07:27 >>>
2009/2/25 Aengus <analog07@eircom.net>:
> On 2/25/2009 2:47 AM, Nanu Kalmanovitz wrote:
>
> The simple answer is that you can probably add LOGFORMAT COMMON immediately
> after your LOGFORMAT COMBINED line in your analog.cfg file, and Analog will
> parse both types of lines. But that won't tell you why your web server is
> using two different logformats, (or whether it's recording the same request
> twice).
>

And if it is recording the same request twice, you really don't want
to do this, or you will double count everything. As Aengus says, you
really need to look at what's in your logfile to diagnose it.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
On 2/26/2009 6:53 AM, Nanu Kalmanovitz wrote:
>
> 192.168.254.254 - - [26/Feb/2009:11:13:45 +0200] "GET /req.png HTTP/1.0" 304 - "http://www.kalmanovitz.co.il/Analog_Report.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18) Gecko/20081031 SUSE/2.0.0.18-0.2.1 Firefox/2.0.0.18"
> ::1 - - [26/Feb/2009:11:13:57 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:59 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:14:00 +0200] "GET /" 400 991
> 38.99.13.125 - - [26/Feb/2009:11:14:45 +0200] "GET /k_comm/Israel/English/Maps/Rezervations/EinHemed33/obj/pages/P7270096_jpg.htm HTTP/1.0" 200 2299 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
>
> They said:
>
> ... this is not a second logformat, it is the HTTP status response is 400 "bad request"
> request, but rather a port scan, or something similar.
> That is why the data about the request is not logged - because there was no HTTP data available, since it wasn't an HTTP request.
> That will happen every time someone connects to port 80 on the server via something else than HTTP protocol...

That sounds like nonsense to me. If they're not being logged by the HTTP
engine, thy shouldn't be in the GTTP access log, and if they are being
logged by the HTTP engine, they should be logged in the correct format.
How do you make a request to an IP port without an IP address anyway?
(Are you sure that it's not an IPv6 request that your web server is
getting confused about?)

> Is it any possibility to configure Analog to interpret the above in a right way?

You can write a LOGFORMAT to match those strings:
LOGFORMAT (%j[%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)

http://analog.cx/docs/logfmt.html

But you don't want to add those requests to your current Analog report.
If you do create a new report to count these lines, you'll need to look
at the Failure Reports, because the Response code is 400.

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
Thanks!

I sent your answer to Suse-Novell people, following is the reply:


Please ask Analog people how is the log entry supposed to be in the correct format, when it is not a HTTP request at all?

400 means "bad request", as in, garbled request, or simple npn-conforming request.
the status code on those requests was 400.
I believe that Apache takes all the information, including the IP address, from the HTTP request body,since this is not a valid HTTP request, it doesn't carry any data.
They are right in assuming that this looks like an ipv6 request, but that is not the problem here.

There is a way to turn off logging this status code.
http://httpd.apache.org/docs/2.2/logs.html#conditional

For the "bad" lines.

> 192.168.254.254 - - [26/Feb/2009:11:13:45 +0200] "GET /req.png HTTP/1.0" 304 - "http://www.kalmanovitz.co.il/Analog_Report.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18) Gecko/20081031 SUSE/2.0.0.18-0.2.1 Firefox/2.0.0.18"
> ::1 - - [26/Feb/2009:11:13:57 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:59 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:14:00 +0200] "GET /" 400 991
> 38.99.13.125 - - [26/Feb/2009:11:14:45 +0200] "GET /k_comm/Israel/English/Maps/Rezervations/EinHemed33/obj/pages/P7270096_jpg.htm HTTP/1.0" 200 2299 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"

SetEnvIf Remote_Addr "::1" dontlog should take care of that.
Please put this into default_server.conf or, if you plan to use Virtualhosts, into the proper config file for that host.

What is your opinion?

In the last answer you said:



TIA

Nanu


+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
>Please ask Analog people how is the log entry supposed to be in the correct format, when it is not a HTTP request at all?

Yes it is.

It came in on port 80, or whatever port the web server was listening to, and the listening port was listening for HTTP traffic. The request that came in, regardless of what it was supposed to be, WAS an HTTP request at that point as far as the web server was concerned. Clearly it was malformed or otherwise in error, so the server assigned it the HTTP status code of 400. That status code is part of the HTTP protocol.

If the request had come in on a different port, then it would not have been seen by the web server and would not have been recorded in the log file.

The format of data recorded to the web server's log file is usually handled by the web server's settings.

Certainly ANY connection to a port on a server is coming from a specific machine with an address, whether that be a real-world IP address, a "localhost" designator which can be written as 127.0.0.1, a private network IP address, or whatever. And that information and other elements should be recorded in the log.

Some something is clearly not working correctly.

-Spode


At 11:12 AM +0200 2/27/09, Nanu Kalmanovitz wrote:
>Thanks!
>
>I sent your answer to Suse-Novell people, following is the reply:
>
>Please ask Analog people how is the log entry supposed to be in the correct format, when it is not a HTTP request at all?
>
>400 means "bad request", as in, garbled request, or simple npn-conforming request.
>the status code on those requests was 400.
>I believe that Apache takes all the information, including the IP address, from the HTTP request body,since this is not a valid HTTP request, it doesn't carry any data.
>They are right in assuming that this looks like an ipv6 request, but that is not the problem here.
>
>There is a way to turn off logging this status code.
>http://httpd.apache.org/docs/2.2/logs.html#conditional
>
>For the "bad" lines.
>
>> 192.168.254.254 - - [26/Feb/2009:11:13:45 +0200] "GET /req.png HTTP/1.0" 304 - "http://www.kalmanovitz.co.il/Analog_Report.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18) Gecko/20081031 SUSE/2.0.0.18-0.2.1 Firefox/2.0.0.18"
>> ::1 - - [26/Feb/2009:11:13:57 +0200] "GET /" 400 991
>> ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991
>> ::1 - - [26/Feb/2009:11:13:59 +0200] "GET /" 400 991
>> ::1 - - [26/Feb/2009:11:14:00 +0200] "GET /" 400 991
>> 38.99.13.125 - - [26/Feb/2009:11:14:45 +0200] "GET /k_comm/Israel/English/Maps/Rezervations/EinHemed33/obj/pages/P7270096_jpg.htm HTTP/1.0" 200 2299 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
>
>SetEnvIf Remote_Addr "::1" dontlog should take care of that.
>Please put this into default_server.conf or, if you plan to use Virtualhosts, into the proper config file for that host.
>
>What is your opinion?
>
>In the last answer you said:
>
>
>
>TIA
>
>Nanu
>
>
>+------------------------------------------------------------------------
>| TO UNSUBSCRIBE from this list:
>| http://lists.meer.net/mailman/listinfo/analog-help
>|
>| Analog Documentation: http://analog.cx/docs/Readme.html
>| List archives: http://www.analog.cx/docs/mailing.html#listarchives
>| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
>+------------------------------------------------------------------------


--
Edward F Spodick, Information Technology Manager
Hong Kong University of Science & Technology Library
lbspodic@ust.hk tel:852-2358-6743 fax:852-2358-1043
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Re: Re: Re: Corrupt logfile lines [ In reply to ]
It certainly would be normal to record those lines in the same format
as everything else with empty referrer and browser. So it does look to
me as if the server is misconfigured -- or at best, strangely
configured. However, if you can't get it changed, analog can still
read those lines with the LOGFORMAT Aengus sent earlier.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Corrupt logfile lines [ In reply to ]
Can Analog be configured to ignore the IPv6 like records ( ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991) in the access_log file?

TIA

Nanu

>>> Aengus<analog07@eircom.net> 26/02/2009 14:26:41 >>>
On 2/26/2009 6:53 AM, Nanu Kalmanovitz wrote:
>
> 192.168.254.254 - - [26/Feb/2009:11:13:45 +0200] "GET /req.png HTTP/1.0" 304 - "http://www.kalmanovitz.co.il/Analog_Report.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18) Gecko/20081031 SUSE/2.0.0.18-0.2.1 Firefox/2.0.0.18"
> ::1 - - [26/Feb/2009:11:13:57 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:13:59 +0200] "GET /" 400 991
> ::1 - - [26/Feb/2009:11:14:00 +0200] "GET /" 400 991
> 38.99.13.125 - - [26/Feb/2009:11:14:45 +0200] "GET /k_comm/Israel/English/Maps/Rezervations/EinHemed33/obj/pages/P7270096_jpg.htm HTTP/1.0" 200 2299 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
>
> They said:
>
> ... this is not a second logformat, it is the HTTP status response is 400 "bad request"
> request, but rather a port scan, or something similar.
> That is why the data about the request is not logged - because there was no HTTP data available, since it wasn't an HTTP request.
> That will happen every time someone connects to port 80 on the server via something else than HTTP protocol...

That sounds like nonsense to me. If they're not being logged by the HTTP
engine, thy shouldn't be in the GTTP access log, and if they are being
logged by the HTTP engine, they should be logged in the correct format.
How do you make a request to an IP port without an IP address anyway?
(Are you sure that it's not an IPv6 request that your web server is
getting confused about?)

> Is it any possibility to configure Analog to interpret the above in a right way?

You can write a LOGFORMAT to match those strings:
LOGFORMAT (%j[%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)

http://analog.cx/docs/logfmt.html

But you don't want to add those requests to your current Analog report.
If you do create a new report to count these lines, you'll need to look
at the Failure Reports, because the Response code is 400.

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
On 3/10/2009 9:31 AM, Nanu Kalmanovitz wrote:
> Can Analog be configured to ignore the IPv6 like records ( ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991) in the access_log file?

LOGFORMAT (::%j)

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Corrupt logfile lines [ In reply to ]
How Analog will interpret and execute 2 consecutive LOGFORMAT instructions:

LOGFORMAT (::%j)
LOGFORMAT combined

TIA

Nanu

>>> Aengus<analog07@eircom.net> 10/03/2009 15:44:16 >>>
On 3/10/2009 9:31 AM, Nanu Kalmanovitz wrote:
> Can Analog be configured to ignore the IPv6 like records ( ::1 - - [26/Feb/2009:11:13:58 +0200] "GET /" 400 991) in the access_log file?

LOGFORMAT (::%j)

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------




+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Re: Re: Re: Corrupt logfile lines [ In reply to ]
On 3/11/2009 2:59 AM, Nanu Kalmanovitz wrote:
> How Analog will interpret and execute 2 consecutive LOGFORMAT instructions:
>
> LOGFORMAT (::%j)
> LOGFORMAT combined

Did you try it?


http://analog.cx/docs/logfmt.html

"A logfile can sometimes have lines in several different formats. So you
can specify several LOGFORMAT commands in a row, and they will all apply
to the next logfile. This is also useful if the format of your logfile
changes half way through."

Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------