Mailing List Archive: log files

log files

dwetzel at nerim

Mar 29, 2007, 9:20 AM

Post #1 of 20 (5695 views)

Hello,
could someone tell me if the shmlogs are saved in a textfile and
where is it ?
Thanks,
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damien WETZEL ("`-/")_.-'"``-._
ATANAR TECHNOLOGIES . . `; -._ )-;-,_`)
(v_,)' _ )`-.\ ``-'
Phone:+33 1 45 43 02 90 _.- _..-_/ / ((.'
- So much to do, so little time - ((,.-' ((,/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

log files [ In reply to ]

Mar 29, 2007, 9:56 AM

Post #2 of 20 (5665 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> could someone tell me if the shmlogs are saved in a textfile and
> where is it ?

No, you have to run varnishlog and / or varnishncsa yourself to
capture the logs to a file. For the next release, I will work with
the packagers to make sure binary packages ship with an init script
which starts varnishlog.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

dwetzel at nerim

Mar 29, 2007, 10:49 AM

Post #3 of 20 (5666 views)

thanks,
i think it could be interresting to add a field in the ncsa logs telling
if the object was delivered from the cache or had to be fetched.
Is there a way to get w3C log ?
and last is there a ay to remove ping pong lines from the logs ?
Damien,
Dag-Erling Sm?rgrav writes:
> Damien Wetzel <dwetzel at nerim.net> writes:
> > could someone tell me if the shmlogs are saved in a textfile and
> > where is it ?
>
> No, you have to run varnishlog and / or varnishncsa yourself to
> capture the logs to a file. For the next release, I will work with
> the packagers to make sure binary packages ship with an init script
> which starts varnishlog.
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no

log files [ In reply to ]

Mar 30, 2007, 12:00 AM

Post #4 of 20 (5655 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> i think it could be interresting to add a field in the ncsa logs telling
> if the object was delivered from the cache or had to be fetched.
> Is there a way to get w3C log ?

That's what varnishncsa does.

> and last is there a ay to remove ping pong lines from the logs ?

All the information you need is in the man page.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

dwetzel at nerim

Mar 30, 2007, 6:45 AM

Post #5 of 20 (5656 views)

Dag-Erling Sm?rgrav writes:
> Damien Wetzel <dwetzel at nerim.net> writes:
> > i think it could be interresting to add a field in the ncsa logs telling
> > if the object was delivered from the cache or had to be fetched.
> > Is there a way to get w3C log ?
>
> That's what varnishncsa does.
i haven't seen any fields using varnishncsa which refers to cache hits or miss
the man talk about the possibility to get the logs in combined format but
i don't how to do it

>
> > and last is there a ay to remove ping pong lines from the logs ?
>
> All the information you need is in the man page.
it's my bad the man pages weren't installed after the make install command.

PS: I don't think ESI for a v2 are a good addon.
ESI is mainly an akamai tool to catch customers but is not of real use.
the standard hasn't changed since 2002 and doesn't seem to be widely used.

>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no

log files [ In reply to ]

Apr 4, 2007, 2:15 AM

Post #6 of 20 (5666 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> Dag-Erling Sm?rgrav writes:
> > Damien Wetzel <dwetzel at nerim.net> writes:
> > > i think it could be interresting to add a field in the ncsa logs
> > > telling if the object was delivered from the cache or had to be
> > > fetched. Is there a way to get w3C log ?
> > That's what varnishncsa does.
> i haven't seen any fields using varnishncsa which refers to cache
> hits or miss the man talk about the possibility to get the logs in
> combined format but i don't how to do it

The output of varnishncsa *is* the so-called combined log format. It
contains no information on cache hits / misses.

> > > and last is there a ay to remove ping pong lines from the logs ?
> > All the information you need is in the man page.
> it's my bad the man pages weren't installed after the make install
> command.

They most likely were. Perhaps your MANPATH is wrong?

> PS: I don't think ESI for a v2 are a good addon. ESI is mainly an
> akamai tool to catch customers but is not of real use. the standard
> hasn't changed since 2002 and doesn't seem to be widely used.

You are mistaken. ESI is widely used by large online news outlets and
portal sites, including some of Linpro's largest customers.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

dwetzel at nerim

Apr 4, 2007, 6:03 AM

Post #7 of 20 (5673 views)

Dag-Erling Sm?rgrav writes:
> Damien Wetzel <dwetzel at nerim.net> writes:
> > Dag-Erling Sm?rgrav writes:
> > > Damien Wetzel <dwetzel at nerim.net> writes:
> > > > i think it could be interresting to add a field in the ncsa logs
> > > > telling if the object was delivered from the cache or had to be
> > > > fetched. Is there a way to get w3C log ?
> > > That's what varnishncsa does.
> > i haven't seen any fields using varnishncsa which refers to cache
> > hits or miss the man talk about the possibility to get the logs in
> > combined format but i don't how to do it
>
> The output of varnishncsa *is* the so-called combined log format. It
> contains no information on cache hits / misses.

i believe there would be a way to get W3C logs which are widely used
in writing a specific program.
I still think that information about hits/misses in the logs would be usefull.

> > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an
> > akamai tool to catch customers but is not of real use. the standard
> > hasn't changed since 2002 and doesn't seem to be widely used.
>
> You are mistaken. ESI is widely used by large online news outlets and
> portal sites, including some of Linpro's largest customers.

For me ESI are so web 1.0 , now the tendency is to deport the personalisation
on the client side.

Damien,
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no

log files [ In reply to ]

dwetzel at nerim

Apr 4, 2007, 6:51 AM

Post #8 of 20 (5662 views)

Denis Br??khus writes:
> ----- Damien Wetzel <dwetzel at nerim.net> wrote:

> And you do have that information, however not in the ncsa logs, due to the fact that the combined logformat does not have a field for that..

--> at the contrary of ncsa logs ,w3c logs are extensible, parametrable and allow the
addition of such a field.

> Use varnishlog for that information.
>
> > For me ESI are so web 1.0 , now the tendency is to deport the
> > personalisation on the client side.
>
> Eh.. If you do believe that all web applications from this day on will be javascript apps where all the content handling is on the client side, good for you. Personally I think there will be a large number of sites / applications where that will not apply. If we can have ESI in Varnish that sounds like an excellent feature to me. We might or might not use it, depending on the work to be done.

Everyone can have its own opinion, mine is that the server side should do a minimum of
work and let the client do it if possible ;)

>
> Writing it off as "web 1.0" isn't very constructive.

>
> Regards
> --
> Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
> http://www.startsiden.no
>

log files [ In reply to ]

Apr 4, 2007, 8:22 AM

Post #9 of 20 (5660 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> Dag-Erling Sm?rgrav writes:
> > Damien Wetzel <dwetzel at nerim.net> writes:
> > > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an
> > > akamai tool to catch customers but is not of real use. the standard
> > > hasn't changed since 2002 and doesn't seem to be widely used.
> > You are mistaken. ESI is widely used by large online news outlets
> > and portal sites, including some of Linpro's largest customers.
> For me ESI are so web 1.0 , now the tendency is to deport the
> personalisation on the client side.

That may be your opinion, but the reality of the marketplace is that
personalization is done on the server side and ESI is alive and well.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

dwetzel at nerim

Apr 4, 2007, 8:35 AM

Post #10 of 20 (5660 views)

hello,
Do you agree that using ESI means using Akamai ?
www.esi.org redirects you through akamai.com

What (features) do you thing is usefull in ESI ?

Damien,

Dag-Erling Sm?rgrav writes:
> Damien Wetzel <dwetzel at nerim.net> writes:
> > Dag-Erling Sm?rgrav writes:
> > > Damien Wetzel <dwetzel at nerim.net> writes:
> > > > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an
> > > > akamai tool to catch customers but is not of real use. the standard
> > > > hasn't changed since 2002 and doesn't seem to be widely used.
> > > You are mistaken. ESI is widely used by large online news outlets
> > > and portal sites, including some of Linpro's largest customers.
> > For me ESI are so web 1.0 , now the tendency is to deport the
> > personalisation on the client side.
>
> That may be your opinion, but the reality of the marketplace is that
> personalization is done on the server side and ESI is alive and well.
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no

log files [ In reply to ]

ask at develooper

Apr 4, 2007, 11:00 AM

Post #11 of 20 (5662 views)

On Apr 4, 2007, at 6:51 AM, Damien Wetzel wrote:

> Everyone can have its own opinion, mine is that the server side
> should do a minimum of
> work and let the client do it if possible ;)

Hi Damien,

If you just need to handle more traffic on your server that might be
a good idea. If you want your users to have a good experience...
Well, not necessarily such a good idea then.

- ask

--
http://develooper.com/ - http://askask.com/

log files [ In reply to ]

dwetzel at nerim

Apr 4, 2007, 3:08 PM

Post #12 of 20 (5657 views)

hi ask,

Ask Bj?rn Hansen writes:
>
> On Apr 4, 2007, at 6:51 AM, Damien Wetzel wrote:
>
> > Everyone can have its own opinion, mine is that the server side
> > should do a minimum of
> > work and let the client do it if possible ;)
>
> Hi Damien,
>
> If you just need to handle more traffic on your server that might be
> a good idea. If you want your users to have a good experience...
> Well, not necessarily such a good idea then.
could detail why leting the client do the work is not a good idea ?
Regarding ESI, I'm pretty sure it is a thing of the past, but again it's
my opinion.
Damien,

>
>
> - ask
>
> --
> http://develooper.com/ - http://askask.com/
>
>

log files [ In reply to ]

Apr 4, 2007, 3:30 PM

Post #13 of 20 (5663 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> Do you agree that using ESI means using Akamai ?

No. Oracle's CMS (whatever it is called, forget at the moment) uses
ESI, and Oracle WebCache supports it. Escenic also supports ESI.

> What (features) do you thing is usefull in ESI ?

What the name says: edge-side includes.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

ask at develooper

Apr 4, 2007, 3:35 PM

Post #14 of 20 (5659 views)

On Apr 4, 2007, at 15:08, Damien Wetzel wrote:

>> If you just need to handle more traffic on your server that might be
>> a good idea. If you want your users to have a good experience...
>> Well, not necessarily such a good idea then.

> could detail why leting the client do the work is not a good idea ?

Have you made any busy/complex websites?

Getting good client-side performance is usually much harder than
scaling and optimizing the server-side. There are things you must do
on the client-side for a good ajax-y experience, but you don't want
to give the clients extra work "just because you can".

- ask

--
http://develooper.com/ - http://askask.com/

log files [ In reply to ]

Apr 4, 2007, 3:37 PM

Post #15 of 20 (5659 views)

Damien Wetzel <dwetzel at nerim.net> writes:
> Regarding ESI, I'm pretty sure it is a thing of the past, but again
> it's my opinion.

Yes, it is your opinion. Unfortunately, it is not founded in fact.
You are still welcome to express it, but when you persist in doing so
despite being repeatedly confronted with conflicting evidence, it
becomes tiring. Can we please move on to a more productive topic?

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

log files [ In reply to ]

Jun 19, 2007, 4:41 AM

Post #16 of 20 (5661 views)

On Thursday 29 March 2007 18:56, Dag-Erling Sm?rgrav wrote:
<snip>
> No, you have to run varnishlog and / or varnishncsa yourself to
> capture the logs to a file. For the next release, I will work with
> the packagers to make sure binary packages ship with an init script
> which starts varnishlog.
>
> DES

Did you ever get around to writing that init script?
If not, how would you recommend going about it?

Adding it to /etc/init.d/varnish, or perhaps a separate initscript?

This would be the bare necessities I guess?

/usr/bin/nohup \
/usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log &

Regards

Gaute Amundsen

log files [ In reply to ]

Jun 19, 2007, 4:45 AM

Post #17 of 20 (5663 views)

Gaute Amundsen <gaute at pht.no> writes:

> Did you ever get around to writing that init script? If not, how
> would you recommend going about it?
>
> Adding it to /etc/init.d/varnish, or perhaps a separate initscript?
>
> This would be the bare necessities I guess?
>
> /usr/bin/nohup \
> /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log &

One separate init script (for Debian) is available at
http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varnish.varnishlog.init

--
Stig Sandbeck Mathisen, Linpro

log files [ In reply to ]

ingvar at linpro

Jun 19, 2007, 4:58 AM

Post #18 of 20 (5661 views)

* Gaute Amundsen
>> Did you ever get around to writing that init script? If not, how
>> would you recommend going about it?
>>
>> Adding it to /etc/init.d/varnish, or perhaps a separate initscript?
>>
>> This would be the bare necessities I guess?
>>
>> /usr/bin/nohup \
>> /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log &

* Stig Sandbeck Mathisen
> One separate init script (for Debian) is available at
> http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varnish.varnishlog.init

Just for the record, the RedHat subdirectory has a similar script. It is
also included in the RHEL packages at sourceforge and in the Fedora package.

Ingvar

--
At barnebokfigurer er snille er bare ljug
Mummipappa dater Barbamamma i smug

log files [ In reply to ]

Jun 19, 2007, 6:56 AM

Post #19 of 20 (5670 views)

<snip>
> * Stig Sandbeck Mathisen
>
> > One separate init script (for Debian) is available at
> > http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varn
> >ish.varnishlog.init
>
> Just for the record, the RedHat subdirectory has a similar script. It is
> also included in the RHEL packages at sourceforge and in the Fedora
> package.
>
> Ingvar

Sorry folks, it seems I'm totaly blind.
I've had /etc/init.d/varnishlog all the time.
(since I upgraded to 1.0.4 I guess)

So then you would pipe this log into "varnishncsa -r /dev/stdin"?
This seems to work for me in AWstats :-)

LogFile="cat /var/log/varnish/varnish.log | varnishncsa -c -r /dev/stdin | "
or perhaps better:
LogFile="/usr/bin/varnishncsa -c -r /var/log/varnish/varnish.log | "

Seems there are ways to set up awstats to process this separately for the
different vhosts too :-)

G.

log files [ In reply to ]

Jun 19, 2007, 3:45 PM

Post #20 of 20 (5657 views)

Gaute Amundsen <gaute at pht.no> writes:
> Did you ever get around to writing that init script?

Yes, 1.0.4 ships with init scripts for both varnishd and varnishlog.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no