Mailing List Archive

Ricardo Newbery <ric at digitalmarbles.com> writes:
> I assume Varnish supports ETags and conditional GETs "If-Modified-
> Since" (IMS) and "If-None-Match" (INM)?

Varnish supports IMS but does not know about ETags or INM.

> Do the Cache-Control headers moderating conditional requests ("must-
> revalidate", "proxy-revalidate", and "no-cache") modify this
> heuristic?

In its default configuration, Varnish ignores Cache-Control directives
from clients.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

In message <64E001C5-7D33-42F8-A41E-13EF0EED84F7 at digitalmarbles.com>, Ricardo N
ewbery writes:
>
>I assume Varnish supports ETags and conditional GETs "If-Modified-
>Since" (IMS) and "If-None-Match" (INM)?

Right now we only support IMS, the ETags are mostly a crutch for
clock-less clients/servers, and varnish isn't relevant in those
cases.

>Are IMS/INM requests relayed to the backend before the content is
>served out of cache?

No, we check IMS and if it passses, we serve out of cache if cache
is still valid.

>Do the Cache-Control headers moderating conditional requests ("must-
>revalidate", "proxy-revalidate", and "no-cache") modify this heuristic?

In general we don't respect cache-control from clients as we are not
a proxy in the RFC2616 sense, but rather part of the server.

Varnish' job is to protect the backend from unnecessary traffic
and since we are in cohort with the backend, we know our content
to be up to date and valid (anything else would be a configuration
error).

If you want to respect cache-control, you can implement it in
vcl_recv()

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

On Jul 26, 2007, at 1:40 AM, Poul-Henning Kamp wrote:

> Ricardo Newbery writes:
>>
>> I assume Varnish supports ETags and conditional GETs "If-Modified-
>> Since" (IMS) and "If-None-Match" (INM)?
>
> Right now we only support IMS, the ETags are mostly a crutch for
> clock-less clients/servers, and varnish isn't relevant in those
> cases.



Sorry, what does this mean precisely? When an ETagged page is served
through Varnish, does it not cache? If it does cache, is it possible
to code the INM test in vcl?

Vary support without ETag concerns me a bit. See https://
bugzilla.mozilla.org/show_bug.cgi?id=269303




>> Are IMS/INM requests relayed to the backend before the content is
>> served out of cache?
>
> No, we check IMS and if it passses, we serve out of cache if cache
> is still valid.


How about an "INM without IMS" request? Still served out of cache I
suppose. Is it possible to modify this behavior via vcl?



>> Do the Cache-Control headers moderating conditional requests ("must-
>> revalidate", "proxy-revalidate", and "no-cache") modify this
>> heuristic?
>
> In general we don't respect cache-control from clients as we are not
> a proxy in the RFC2616 sense, but rather part of the server.


Okay. If I understand correctly, since Varnish doesn't respect cache-
control from clients, then backend-generated cache-control like "must-
revalidate", "proxy-revalidate", and "no-cache" are unnecessary
since the "max-age" and "s-maxage" response cache-controls are
sufficient -- at least as far as the Varnish heuristic is concerned.
Correct?

Does Varnish respect "private" and "no-store" in the response?

How about "public"?

Ric

In message <E7EB8D13-D7B6-42CD-89F0-524B819B6527 at digitalmarbles.com>, Ricardo N
ewbery writes:

>Vary support without ETag concerns me a bit. See https://
>bugzilla.mozilla.org/show_bug.cgi?id=269303

Vary will respect any header listed.

>How about an "INM without IMS" request? Still served out of cache I
>suppose. Is it possible to modify this behavior via vcl?

Everything is served out of cache except Cookies and Authentication.

IMS is only special in that it does not return the body unless necessary.

INM is ignored and hence the body is always returned.

>Okay. If I understand correctly, since Varnish doesn't respect cache-
>control from clients, then backend-generated cache-control like "must-
>revalidate", "proxy-revalidate", and "no-cache" are unnecessary
>since the "max-age" and "s-maxage" response cache-controls are
>sufficient -- at least as far as the Varnish heuristic is concerned.
>Correct?

You may still want to generate the headers for the clients to see.

>Does Varnish respect "private" and "no-store" in the response?

No. Varnish is considered part of the web-server, and thus these
don't apply to it (unless people implement them in VCL)

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

On Jul 26, 2007, at 9:55 PM, Poul-Henning Kamp wrote:

> Ricardo Newbery writes:
>> How about an "INM without IMS" request? Still served out of cache I
>> suppose. Is it possible to modify this behavior via vcl?
>
> Everything is served out of cache except Cookies and Authentication.
>
> IMS is only special in that it does not return the body unless
> necessary.
>
> INM is ignored and hence the body is always returned.


Yes, but is it possible to get INM behavior via a custom VCL? In
other words, Varnish already has a mechanism to serve 304s -- is it
possible, using VCL, to tie into this mechanism? And is it possible,
using VCL, to test for the ETag value in the cached copy?




>> Okay. If I understand correctly, since Varnish doesn't respect
>> cache-
>> control from clients, then backend-generated cache-control like
>> "must-
>> revalidate", "proxy-revalidate", and "no-cache" are unnecessary
>> since the "max-age" and "s-maxage" response cache-controls are
>> sufficient -- at least as far as the Varnish heuristic is concerned.
>> Correct?
>
> You may still want to generate the headers for the clients to see.


Yes, I understand this. What I need to do for the clients is a
separate issue.

I'm a little unsure as I didn't hear an affirmation to my question...
so does Varnish respect max-age and s-maxage in the response?



>> Does Varnish respect "private" and "no-store" in the response?
>
> No. Varnish is considered part of the web-server, and thus these
> don't apply to it (unless people implement them in VCL)



Okay... How should the backend inform Varnish that a certain item
should not be stored in the cache?

Ric

Ricardo Newbery <ric at digitalmarbles.com> writes:
> I'm a little unsure as I didn't hear an affirmation to my question...
> so does Varnish respect max-age and s-maxage in the response?

Yes, see r1386.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

On Jul 28, 2007, at 1:08 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> I'm a little unsure as I didn't hear an affirmation to my question...
>> so does Varnish respect max-age and s-maxage in the response?
>
> Yes, see r1386.


Ahh... okay. Thanks.

Now back to the question about INM requests. After some thought, I
guess I'm less interested about trying to replicate INM in VCL but am
still interested in how Varnish refreshes it's own cache.

Since Varnish respects the maxage controls, this implies that when
the next request comes in after expiry it will generate a conditional
request (if Last-Modified is set) to refresh it's cache. Correct?

But no INM to the backend will ever be generated regardless of the
presence of an ETag. Correct?

In my own use case (Plone CMS), etags are leveraged as a convenient
index hash for the server's pagecache in RAM. But in order to serve
from the pagecache, we need an INM in the request. So I'm trying to
understand the implications of Varnish's ETag behavior on this process.

Ric

On Jul 28, 2007, at 3:47 PM, Ricardo Newbery wrote:

>
> On Jul 28, 2007, at 1:08 AM, Dag-Erling Sm?rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> I'm a little unsure as I didn't hear an affirmation to my
>>> question...
>>> so does Varnish respect max-age and s-maxage in the response?
>>
>> Yes, see r1386.
>
>
> Ahh... okay. Thanks.
>
> Now back to the question about INM requests. After some thought, I
> guess I'm less interested about trying to replicate INM in VCL but am
> still interested in how Varnish refreshes it's own cache.
>
> Since Varnish respects the maxage controls, this implies that when
> the next request comes in after expiry it will generate a conditional
> request (if Last-Modified is set) to refresh it's cache. Correct?
>
> But no INM to the backend will ever be generated regardless of the
> presence of an ETag. Correct?
>
> In my own use case (Plone CMS), etags are leveraged as a convenient
> index hash for the server's pagecache in RAM. But in order to serve
> from the pagecache, we need an INM in the request. So I'm trying to
> understand the implications of Varnish's ETag behavior on this
> process.
>
> Ric


Correction. Upon reflection the explanation of the server pagecache
behavior didn't sound right. So I checked the code, and I was wrong
--- the pagecache doesn't need an INM; it regenerates the ETag based
on the request before querying the pagecache.

Ric

Ricardo Newbery <ric at digitalmarbles.com> writes:
> Since Varnish respects the maxage controls, this implies that when the
> next request comes in after expiry it will generate a conditional
> request (if Last-Modified is set) to refresh it's cache. Correct?

No, at present it will generate an unconditional request because the
expired version will already have been removed from the cache.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

On Jul 29, 2007, at 2:10 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> Since Varnish respects the maxage controls, this implies that when
>> the
>> next request comes in after expiry it will generate a conditional
>> request (if Last-Modified is set) to refresh it's cache. Correct?
>
> No, at present it will generate an unconditional request because the
> expired version will already have been removed from the cache.
>
> DES


Okay. I think I'm getting a better handle on this now. Thanks for
your patience.

One more question regarding conditional requests:

What happens when a client sends a conditional request (either INM or
IMS) and the content is not available in the Varnish cache? I assume
that Varnish will just relay the original conditional request through
to the backend. But then what happens when the backend responds with
a 304? I'm guessing Varnish won't cache a response until a non-304
is returned. Correct?

Ric

Ricardo Newbery <ric at digitalmarbles.com> writes:
> What happens when a client sends a conditional request (either INM or
> IMS) and the content is not available in the Varnish cache? I assume
> that Varnish will just relay the original conditional request through
> to the backend.

No, it will strip the conditional header before requesting the page from
the backend.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Ricardo Newbery <ric at digitalmarbles.com> writes:
> On Jul 29, 2007, at 2:10 AM, Dag-Erling Sm?rgrav wrote:
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> Since Varnish respects the maxage controls, this implies that when
>>> the next request comes in after expiry it will generate a
>>> conditional request (if Last-Modified is set) to refresh it's cache.
>>> Correct?
>> No, at present it will generate an unconditional request because the
>> expired version will already have been removed from the cache.
> What happens when a client sends a conditional request (either INM or
> IMS) and the content is not available in the Varnish cache? I assume
> that Varnish will just relay the original conditional request through
> to the backend. But then what happens when the backend responds with
> a 304? I'm guessing Varnish won't cache a response until a non-304
> is returned. Correct?

"No, at present it will generate an unconditional request"

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> What happens when a client sends a conditional request (either INM or
>> IMS) and the content is not available in the Varnish cache? I assume
>> that Varnish will just relay the original conditional request through
>> to the backend.
>
> No, it will strip the conditional header before requesting the page
> from
> the backend.


Okay, I think I understand. So with Varnish in front, using the
default VCL, there is no set of circumstances that will ever result
in a conditional request to the backend. Correct?

But I can still change this behavior with a custom VCL. Correct?

Would the following work:


# perpetually refresh my cache
vcl_timeout {
fetch;
}


# hold on to cache for at least a day
vcl_fetch {
if (obj.ttl < 86400s) {
set obj.ttl = 86400s;
}


# if not available in cache, pass client's IMS to backend
# and pass response back to client without caching
vcl_miss {
if (req.http.If-Modified-Since) {
pass;
}
}


If I wanted to selectively apply the first two above, could I just
leverage a few custom headers like so?


# selectively force a perpetually refreshing cache (is "resp"
available here?)
vcl_timeout {
if (resp.http.X-Cache-Refetch) {
fetch;
}
}


# selectively apply a local ttl without messing up downstream proxies
vcl_fetch {
if (resp.http.X-Cache-TTL) {
set obj.ttl = resp.http.X-Cache-TTL;
}
}

Ricardo Newbery <ric at digitalmarbles.com> writes:

> On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm?rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> What happens when a client sends a conditional request (either INM or
>>> IMS) and the content is not available in the Varnish cache? I assume
>>> that Varnish will just relay the original conditional request through
>>> to the backend.
>>
>> No, it will strip the conditional header before requesting the page
>> from
>> the backend.
>
>
> Okay, I think I understand. So with Varnish in front, using the
> default VCL, there is no set of circumstances that will ever result
> in a conditional request to the backend. Correct?

Yes.

> But I can still change this behavior with a custom VCL. Correct?

Yes, by editing bereq in vcl_miss.

> # perpetually refresh my cache
> vcl_timeout {
> fetch;
> }

This probably won't, but it's on our list for 2.0.

> # if not available in cache, pass client's IMS to backend
> # and pass response back to client without caching
> vcl_miss {
> if (req.http.If-Modified-Since) {
> pass;
> }
> }

You probably don't want to do that. It basically opens up a back door,
allowing anyone to bypass Varnish and hit your backend directly just by
including a bogus IMS in their request.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

In message <3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877 at digitalmarbles.com>, Ricardo Newbery writes
:
>
>On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm=F8rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> What happens when a client sends a conditional request (either INM or
>>> IMS) and the content is not available in the Varnish cache? I assume
>>> that Varnish will just relay the original conditional request through
>>> to the backend.
>>
>> No, it will strip the conditional header before requesting the page =
>
>> from
>> the backend.
>
>
>Okay, I think I understand. So with Varnish in front, using the =
>
>default VCL, there is no set of circumstances that will ever result =
>
>in a conditional request to the backend. Correct?

Correct.

>But I can still change this behavior with a custom VCL. Correct?
>
>Would the following work:
>
>
># perpetually refresh my cache
>vcl_timeout {
> fetch;
>}

This is not implemented yet, but once we have it...

># if not available in cache, pass client's IMS to backend
># and pass response back to client without caching
>vcl_miss {
> if (req.http.If-Modified-Since) {
> pass;
> }
>}

This should work.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.