Mailing List Archive

Whether this is part of a different specification or not, it's
unreasonable for Varnish to NOT cache static, cacheable objects by
default, solely because these requests were sent along with Cookies.

In effect, the default Varnish policy of not caching Cookied requests
causes Varnish not to cache anything at all for most sites (you know,
there are tons of people out there using Google Analytics). Think about
it: why would people want the overhead of a non-caching accelerating
proxy?

On s?b, 2007-06-30 at 23:41 +0200, Dag-Erling Sm?rgrav wrote:
> [moved from -dev to -misc]
>
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > As you might already have noted, I reported a bug on varnish caching
> > files indiscriminately.
>
> This is not a bug, it is a misunderstanding. It appears you expect
> Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a
> "surrogate" (see the "Edge Architecture Specification" by Oracle and
> Akamai, although Varnish does not yet fully implement that specification
> either)
>
> > My page sets a few cookies. That'd be okay and it should produce
> > dynamic pages, which Varnish is furnishing through my backend. The
> > thing is, these cookies are sent along requests for CSS and PNG and JPG
> > and JS files, which causes varnish to contact the backend. I don't want
> > that to happen (I'm happy with them being cached by Varnish 120
> > seconds).
> >
> > How can I tell Varnish that requests with a response that includes ETag
> > (a discriminant for static files) should be forcibly cached?
>
> This is basically the same issue as in your previous email, and the
> answer is the same.
>
> DES
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok:
Small things make base men proud.
-- William Shakespeare, "Henry VI"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070630/10af24f2/attachment.pgp

"Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> In effect, the default Varnish policy of not caching Cookied requests
> causes Varnish not to cache anything at all for most sites (you know,
> there are tons of people out there using Google Analytics). Think about
> it: why would people want the overhead of a non-caching accelerating
> proxy?

Perhaps your assumptions are flawed?

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Dear Manuel,

The Varnish *DEFAULT* VCL code implements what we have decided is a
sensible default policy, which will Do The Right Thing for most people,
at least to get them going with Varnish.

Your complaints all seem to center on the fact that the default
VCL code doesn't work for you.

You seem to have overlooked the fact that you are not forced to
run with the default VCL code.

Varnish is on track to offer the most comprehensive and flexible
configuration mechanism yet to be seen on any web server or cache.

Now, instead of whining about the default, tailor Varnish to your
situation.

Thankyou!

Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Hi,

>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 30 Jun 2007 19:10:16 -0500
> From: "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com>
> Subject: Re: Problem with varnish and caching
> To: Dag-Erling Sm?rgrav <des at linpro.no>
> Cc: varnish-misc at projects.linpro.no
> Message-ID: <1183248616.3955.7.camel at gabriela>
> Content-Type: text/plain; charset="utf-8"
>
> Whether this is part of a different specification or not, it's
> unreasonable for Varnish to NOT cache static, cacheable objects by
> default, solely because these requests were sent along with Cookies.

I am not aware of any HTTP accelerator that would caching this by
default. Since I believe it is a direct violation of all standards.
Think of it this way: Why cache pages that are said to be dynamic by the
webpage.

Squid and Bluecoats will not cache these pages, and the mechanism(s) to
tune that are far from fine grained.

> In effect, the default Varnish policy of not caching Cookied requests
> causes Varnish not to cache anything at all for most sites (you know,
> there are tons of people out there using Google Analytics). Think about
> it: why would people want the overhead of a non-caching accelerating
> proxy?

Here you assume "most sites" use Google Analytics. I am not so sure
about that. Either way, while Varnish will suite almost any site you can
imagine, it's strengths will be most appreciated by "larger" sites. With
larger sites I mean sites that would not dream of using Google Analytics.

YS
Anders Berg


> On s?b, 2007-06-30 at 23:41 +0200, Dag-Erling Sm?rgrav wrote:
>> [moved from -dev to -misc]
>>
>> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
>>> As you might already have noted, I reported a bug on varnish caching
>>> files indiscriminately.
>> This is not a bug, it is a misunderstanding. It appears you expect
>> Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a
>> "surrogate" (see the "Edge Architecture Specification" by Oracle and
>> Akamai, although Varnish does not yet fully implement that specification
>> either)
>>
>>> My page sets a few cookies. That'd be okay and it should produce
>>> dynamic pages, which Varnish is furnishing through my backend. The
>>> thing is, these cookies are sent along requests for CSS and PNG and JPG
>>> and JS files, which causes varnish to contact the backend. I don't want
>>> that to happen (I'm happy with them being cached by Varnish 120
>>> seconds).
>>>
>>> How can I tell Varnish that requests with a response that includes ETag
>>> (a discriminant for static files) should be forcibly cached?
>> This is basically the same issue as in your previous email, and the
>> answer is the same.
>>
>> DES
> Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
> The R Zone - http://rudd-o.com/
> GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/
>
> Now playing, courtesy of Amarok:
> Small things make base men proud.
> -- William Shakespeare, "Henry VI"

Perhaps.

But my data seems to contradict your assumption that my assumptions are
flawed, since for each request on my Varnish log, there's a matching
request on my Apache log.

If Varnish isn't accelerating anything (not even static content) due to
Cookies, then... hmmmm.... all that fancy new-world engineering
architecture is useless. By the way, I find the Varnish cache
architecture absolutely sensible and I commend you guys for it in
earnest, Squid should work like that.

At the moment I have found a temporary hackish solution where:


if (req.http.Cookie ~ "wordpresspass") {
pipe;
}

if (req.http.Authenticate) {
pipe;
}

if (req.http.Expect) {
pipe;
}

if (req.http.Pragma ~ "no-cache") {
pipe;
}

lookup;

And it seems to be working fine (see the "wordpresspass" regexp match
for the cookie).

By the way, the manual page doesn't explain the difference between pipe
and pass. Yes, technically it explains the difference, but it doesn't
discuss the implications of the difference for the caching subsystem,
and the implications aren't obvious. I initially used "pass" but that
made my pages malfunction, so I reverted to pipe (which was
counterintuitive). I was expecting "pass" to feed data straight from
the Web server to the client without caching the data, and for some
reason my site malfunctioned (redirects didn't work, logins didn't work,
etcetera). I verified these statements using Wireshark.


On dom, 2007-07-01 at 10:53 +0200, Dag-Erling Sm?rgrav wrote:
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > In effect, the default Varnish policy of not caching Cookied requests
> > causes Varnish not to cache anything at all for most sites (you know,
> > there are tons of people out there using Google Analytics). Think about
> > it: why would people want the overhead of a non-caching accelerating
> > proxy?
>
> Perhaps your assumptions are flawed?
>
> DES
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: UB40 - Reggae music
O, what a tangled web we weave, When first we practice to deceive.
-- Sir Walter Scott, "Marmion"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070701/67fe9cf3/attachment.pgp

See my last e-mail to find out the hack I had to implement in order for
Varnish to sort of work on my site.

I moved from Squid to Varnish and got stumped by your default policy.
Squid accelerated static objects by default, cookies or no cookies.
Varnish doesn't. Period.

It took me 15 minutes to install Squid, learn how to set it up as an
accelerator, and set it up. It took me over two hours just to answer
the question "why isn't Varnish accelerating my site?", and over 15
minutes to configure it to act sensibly. Doesn't look too well for your
product now, does it?

Now I'm going to state what up until now I've only been implying: your
default policy is braindead and stupid. For 99% of your potential user
base (who tend to use these annoying little things called cookies),
Varnish by default won't accelerate squat -- it will, however, introduce
additional overhead. The other 1% serving all-static content with no
cookies most probably doesn't need an accelerator anyway.

Let me say it again: the default policy is braindead and stupid. It is
braindead and stupid because no large site (you know, the type of site
which actually needs acceleration) will run without a cookie-based
visitor tracking system, and as a consequence, Varnish won't accelerate
anything.

I suggest you alter your default VCL policy to roughly match Squid's:

- Cache everything cacheable with an adaptive TTL based on the
Last-Modified header. Or 120s. I don't care, but I'd prefer an
adaptive TTL -- old objects have low probability of being modified.
- Do not use a client's Cookie headers as a sign that an object should
be re-fetched from the backend. Web applications that need to issue
fresh objects already implement either varying URLs or
Cache-Control/Pragma headers to instruct caches not to cache content.
- Anytime a client sends Cache-Control: no-cache headers, re-fetch the
content from the backend.
- Obey the backend's Cache-Control and Pragma no-cache headers. Also
obey the Expires header.

You see, Squid at least *accelerates something* out-of-the-box. In the
meantime, try not to advertise Varnish as an accelerator, when it
effectively isn't without heavy user intervention.

I'm tired of arguing with you guys over something that should have been
*obvious* for expert engineers who, in fact, have written a solid and
revolutionary piece of software. It seems you're overlooking the
details, and regrettably in this case the devil *is* in the details.

Thanks for your time.

On dom, 2007-07-01 at 09:05 +0000, Poul-Henning Kamp wrote:
> Dear Manuel,
>
> The Varnish *DEFAULT* VCL code implements what we have decided is a
> sensible default policy, which will Do The Right Thing for most people,
> at least to get them going with Varnish.
>
> Your complaints all seem to center on the fact that the default
> VCL code doesn't work for you.
>
> You seem to have overlooked the fact that you are not forced to
> run with the default VCL code.
>
> Varnish is on track to offer the most comprehensive and flexible
> configuration mechanism yet to be seen on any web server or cache.
>
> Now, instead of whining about the default, tailor Varnish to your
> situation.
>
> Thankyou!
>
> Poul-Henning
>
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: UB40 - Reggae music
Don't Worry, Be Happy.
-- Meher Baba
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070701/970a9ab6/attachment.pgp

In message <1183318740.14999.38.camel at gabriela>, "Manuel Amador (Rudd-O)" write
s:

>I moved from Squid to Varnish and got stumped by your default policy.
>Squid accelerated static objects by default, cookies or no cookies.
>Varnish doesn't. Period.
>
>It took me 15 minutes to install Squid, learn how to set it up as an
>accelerator, and set it up. It took me over two hours just to answer
>the question "why isn't Varnish accelerating my site?", and over 15
>minutes to configure it to act sensibly. Doesn't look too well for your
>product now, does it?

Nobody forces you to use Varnish, and given your unnecssarily snotty
attitude, I suggest you stick with Squid.

This is of course pretty harsh on the squid project, but you will
almost certainly not be able to contribute positively to the Varnish
project with that attitude.

Killfile, meet Manuel, Manuel, bye!

Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

"Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> But my data seems to contradict your assumption that my assumptions are
> flawed, since for each request on my Varnish log, there's a matching
> request on my Apache log.

You assume that your site is typical of those that use Varnish. You
assume that we developed Varnish without any reference to real-life
sites, applications and users. You assume that you are infinintely
smarter and wiser than us, and that because of this, insulting us and
swearing at us will make us realize the error of our ways and embrace
your interpretation of reality. All of these assumptions are incorrect.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Sigh, I keep going back to the engineering, and you keep personalizing
the matter and returning to your lies.

I never insulted any of you. You're personalizing the matter, and
attempting to somehow equate my bug reports with personal criticism
towards the developers.

You're wrong about the majority of sites being accelerated by Varnish.
Almost no modern site works without cookies -- hence, almost no modern
site is cached according to Varnish default policies. You have not
provided a single counterexample or a single snippet of VCL that could
solve the problems I have, or a single snippet of VCL that you guys are
actually using on production servers.

I can admit that I'm considerably less capable than you, after all I
didn't write Varnish whereas you *did*. However, I'm considerably less
stupid than you would have me be. Here I am, finding a real problem in
the real-life world -- and here you are, dismissing the problem like
you're the owner of the universe and the truth, just because YOU chose
to interpret my e-mails as direct criticism of YOU.

Want direct criticism and harsh words to match your interpretations?
Well, here they go, try and eat it with a side dish:

YOU're a rude thick-headed and thin-skinned individual who hasn't taken
a single minute out of his blame-me routine to actually HELP. You're
carrying your heart on your sleeve. Grow up, stop acting like a prima
donna, and understand that careless engineering and glib remarks aren't
gonna be tolerated by your users, no matter how cool you think you are
or how open-source-you're-free-not-to-use-us your project may be.

This discussion has been absolutely useless.

(Suggestion: perhaps you want to add me to your killfile like your
partner did; otherwise you could conceivably receive more "insults and
swears" may get to your inbox.)

On lun, 2007-07-02 at 07:29 +0200, Dag-Erling Sm?rgrav wrote:
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > But my data seems to contradict your assumption that my assumptions are
> > flawed, since for each request on my Varnish log, there's a matching
> > request on my Apache log.
>
> You assume that your site is typical of those that use Varnish. You
> assume that we developed Varnish without any reference to real-life
> sites, applications and users. You assume that you are infinintely
> smarter and wiser than us, and that because of this, insulting us and
> swearing at us will make us realize the error of our ways and embrace
> your interpretation of reality. All of these assumptions are incorrect.
>
> DES
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: Q: How many IBM CPU's does it take to do a logical right shift?
A: 33. 1 to hold the bits and 32 to push the register.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070702/77a1f813/attachment.pgp

Manuel Amador (Rudd-O) wrote:
> site is cached according to Varnish default policies. You have not
> provided a single counterexample or a single snippet of VCL that could
> solve the problems I have, or a single snippet of VCL that you guys are
> actually using on production servers.
>
>
"man vcl" does provide an example about how to cache even if Cookies are
present.
Did i miss something?

Regards
A.S

On Jul 2, 2007, at 7:26, Manuel Amador (Rudd-O) wrote:

> I never insulted any of you.

You really should go see a therapist. Just tell him or her: "People
think I'm a rude idiot - please help me."

In your weblog post you called Dag-Erling and Poul-Henning "thin-
skinned". Rather than disagree (although I do) with that I'll
contest the assumption that having "thick skin" should be a
prerequisite for building or participating in an open source community.

To be on-topic: I agree with the current conservative, RFC compliant
default behavior. (Or to be more accurate: it was caching just fine
for my use-case, too).


- ask

--
http://develooper.com/ - http://askask.com/

----- Anup Shukla <anup at iamcool.net> wrote:
> Manuel Amador (Rudd-O) wrote:
> > site is cached according to Varnish default policies. You have not
> > provided a single counterexample or a single snippet of VCL that
> > could solve the problems I have, or a single snippet of VCL that you guys
> > are actually using on production servers.
> "man vcl" does provide an example about how to cache even if Cookies
> are present.
> Did i miss something?

Yeah, I think you missed the fact that Manuel isn't really interested in anything else than spewing out insults and bulls*** on the list. Nothing to see / hear here people move on, just an angry troll.

Regards
--
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no

Thanks, Anup. I did read the vcl man page several times. The technical
problem with Varnish is that, if I set it up to cache things even if
Cookies are present, my site malfunctions because pages for logged-in
users are cached. I have to make the caching conditional to the login
Cookie specifically (I did use a conditional "hack" to get this in
default.vcl) otherwise the site completely malfunctions.

The optimum behaviour would be Varnish caching static files
(img,css,jpg,png, based on response HTTP headers) and not caching
dynamic ones. But as it currently stands, Varnish configuration
language doesn't allow caching to be controlled based on the response
headers, only on the request ones. Squid, however, does.

On mar, 2007-07-03 at 10:12 +0530, Anup Shukla wrote:
> Manuel Amador (Rudd-O) wrote:
> > site is cached according to Varnish default policies. You have not
> > provided a single counterexample or a single snippet of VCL that could
> > solve the problems I have, or a single snippet of VCL that you guys are
> > actually using on production servers.
> >
> >
> "man vcl" does provide an example about how to cache even if Cookies are
> present.
> Did i miss something?
>
> Regards
> A.S
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok:
Q: What do you say to a New Yorker with a job?
A: Big Mac, fries and a Coke, please!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070703/54f915b1/attachment.pgp

On lun, 2007-07-02 at 22:42 -0700, Ask Bj?rn Hansen wrote:
> On Jul 2, 2007, at 7:26, Manuel Amador (Rudd-O) wrote:
>
> > I never insulted any of you.
>
> You really should go see a therapist. Just tell him or her: "People
> think I'm a rude idiot - please help me."

I will convey your thoughts to my therapist. We'll probably laugh about
it over a bottle (or two) of scotch.

>
> In your weblog post you called Dag-Erling and Poul-Henning "thin-
> skinned". Rather than disagree (although I do) with that I'll
> contest the assumption that having "thick skin" should be a
> prerequisite for building or participating in an open source community.

Yes, having a thick skin is a prerequisite for participating in a
community where everyone can actually *talk* to you and *reach* you.
Being thick-skinned means:

- not taking criticism of your product PERSONALLY
- conceding that others may have a point, however wrong their arguments
appear to be
- empathy

Under that definition, neither Dag-Erling, nor Poul-Henning have a thick
skin.

>
> To be on-topic: I agree with the current conservative, RFC compliant
> default behavior. (Or to be more accurate: it was caching just fine
> for my use-case, too).

Well, then Varnish is a perfect fit for you. It wasn't for me, and (I
spent some time reading Varnish's varnish-misc mailing list) I
discovered others had experienced the same problem.

>
>
> - ask
>
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok:
Never reveal your best argument.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070703/98b06b78/attachment.pgp

Manuel Amador (Rudd-O) wrote:
> The optimum behaviour would be Varnish caching static files
> (img,css,jpg,png, based on response HTTP headers) and not caching
> dynamic ones. But as it currently stands, Varnish configuration
> language doesn't allow caching to be controlled based on the response
> headers, only on the request ones. Squid, however, does.
>
>

I had a similar situation.
In my case the site is 100% php, so in effect anything that comes out
with Content-Type text/html is php generated and given the nature of the
site, its not cacheable.

I have also put below my VCL configuration which works fine for me.
Its not much of a high quality configuration, but works for me.

Suggestion, as always, are welcome :)

sub vcl_recv {
if (req.request == "GET" && req.http.cookie) {
lookup;
}
}

sub vcl_fetch {
if (!obj.valid) {
error;
}
if (!obj.cacheable) {
pass;
}
if (resp.http.Content-Type ~ "text/html") {
pass;
}
if (resp.http.Set-Cookie) {
insert;
}
insert;
}

Hope this helps.

Regards
A.S

Thanks a lot. Will use this to tune Varnish better. This functionality
is not mentioned in the manual page of vcl, nor was it conveyed to me by
Dag-Erling or Poul-Henning.

On mi?, 2007-07-04 at 09:37 +0530, Anup Shukla wrote:
> Manuel Amador (Rudd-O) wrote:
> > The optimum behaviour would be Varnish caching static files
> > (img,css,jpg,png, based on response HTTP headers) and not caching
> > dynamic ones. But as it currently stands, Varnish configuration
> > language doesn't allow caching to be controlled based on the response
> > headers, only on the request ones. Squid, however, does.
> >
> >
>
> I had a similar situation.
> In my case the site is 100% php, so in effect anything that comes out
> with Content-Type text/html is php generated and given the nature of the
> site, its not cacheable.
>
> I have also put below my VCL configuration which works fine for me.
> Its not much of a high quality configuration, but works for me.
>
> Suggestion, as always, are welcome :)
>
> sub vcl_recv {
> if (req.request == "GET" && req.http.cookie) {
> lookup;
> }
> }
>
> sub vcl_fetch {
> if (!obj.valid) {
> error;
> }
> if (!obj.cacheable) {
> pass;
> }
> if (resp.http.Content-Type ~ "text/html") {
> pass;
> }
> if (resp.http.Set-Cookie) {
> insert;
> }
> insert;
> }
>
> Hope this helps.
>
> Regards
> A.S
Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
The R Zone - http://rudd-o.com/
GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: VA - Ingen sommar utan reggae - Markoolio
As flies to wanton boys are we to the gods; they kill us for their sport.
-- Shakespeare, "King Lear"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20070704/a95d7642/attachment.pgp