Mailing List Archive: Varnish Logging

Varnish Logging

Dec 7, 2007, 12:16 AM

Post #1 of 3 (166 views)

Hello, all!

First of all, thanks to everyone for making such a great product.
Varnish has performed wonderfully. It really does 'just work'.

I am having some problems regarding logging, however. Our specific app
requires detailed logging data for every transaction. Namely, we have
to track the number of bytes leaving our app so that we can charge
clients based on data transfer usage. So, for using varnish, we've got
two problems:

1) varnishncsa seems to only log the size (in bytes) of the file being
transfered. We deal a lot with large files (30MB-100MB), so many of our
file transfers are cut short. We need to be able to log exactly how
many bytes were actually transfered. Is this possible?

2) We've also noticed that if the file transfer is aborted before about
2MB is transfered, the file transfer isn't logged at all to access.log.
Is this by design? For the same reasons stated in #1, we need to find a
reason to log these transfers as well.

On a side note, I tried using Squid (yuck) and found that it satisfied
point 1, but /very/ strangely also had the same 'problem' described in
point 2. (Although the 'logging threshold' was lower, around 1MB or
so) So weird.

I'd love to hear your thoughts! If we can't get this working, I'm going
to have to resort to a suboptimal solution using lighttpd and rsync...(yuck)

Thanks in advance,
-Rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-misc/attachments/20071207/dbee5dfb/attachment.htm

Varnish Logging [ In reply to ]

gwenael.saint-genest at makina-corpus

Dec 7, 2007, 2:43 AM

Post #2 of 3 (164 views)

Permalink

Rich Christiansen wrote:
> Hello, all!

Hi !

> 1) varnishncsa seems to only log the size (in bytes) of the file being
> transfered. We deal a lot with large files (30MB-100MB), so many of our
> file transfers are cut short. We need to be able to log exactly how
> many bytes were actually transfered. Is this possible?

I think you can't get this information for the moment.

In your case, file transfer is made by RES_WriteObj() (cache_response.c)
with a call to WRK_Sendfile() (cache_pool.c). If you look into thins
function :

if (WRK_Flush(w) == 0 &&
sendfile(*w->wfd, fd, &off, len) != len)
w->werr++;

The number of bytes copied (returned by sendfile) is not saved :(

With a little patch you can save this value and use it for statistics.

Gwen

--
Saint-Genest Gwenael <gwenael.saint-genest at makina-corpus.com>
Makina Corpus - http://www.makina-corpus.com/
44 boulevard des Pas enchant?s FR-44230 Saint-S?bastien-sur-Loire
T : +33 (0) 2 51 79 80 80 P: +33 (0)6 81 05 03 00

Varnish Logging [ In reply to ]

phk at phk

Dec 7, 2007, 4:45 AM

Post #3 of 3 (164 views)

Permalink

In message <475915CD.2010900 at makina-corpus.com>, Saint-Genest Gwenael writes:
>Rich Christiansen wrote:

>> transfered. We deal a lot with large files (30MB-100MB), so many of our
>> file transfers are cut short. We need to be able to log exactly how
>> many bytes were actually transfered. Is this possible?
>
> if (WRK_Flush(w) == 0 &&
> sendfile(*w->wfd, fd, &off, len) != len)
> w->werr++;
>
>The number of bytes copied (returned by sendfile) is not saved :(
>
>With a little patch you can save this value and use it for statistics.

Please open a ticket for this, so we get the number properly integrated
in the statistics.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.