Mailing List Archive

[Varnish 4] Respecting client's Cache-Control: max-age= as TTL
Hi,

I have been reading quite a bit about Varnish and VCL but found almost no
examples with Cache-Control coming from the client request [1].

What I want to achieve: if the client sends Cache-Control: max-age=60, TTL
becomes 60 s. If the cache hit is fresher than 60 s, deliver it, otherwise
fetch a new response from backend (I hope I'm not misusing the VCL terms
here) *and* cache it.

I had hacked this together in the vcl_fetch section in Varnish 3.x by
setting the req.http.Cache-Control max-age value as beresp.ttl, but
vcl_fetch is gone in Varnish 4.x.

I have received a suggestion to use vcl_hit and/or grace [2], but again --
no examples...

Could anyone provide some VCL pseudo-code that uses req.http.Cache-Control
value to override TTL? max-age number parsing not necessary, I have figure
that out.

Thanks,

Martynas

[1]
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#Cache_request_directives
[2]
https://github.com/varnishcache/varnish-cache/issues/2014#issuecomment-319096566
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
On github I pointed to the doc explaining how you can return(fetch) to
ignore a cached object, possibly based on ttl, so you already have half the
answer.

The other part of the equation is just converting req.http.cache-control to
a duration and comparing that to obj.ttl. It will be similar to what you
have done on v3.

--
Guillaume Quintard

On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com> wrote:

> Hi,
>
> I have been reading quite a bit about Varnish and VCL but found almost no
> examples with Cache-Control coming from the client request [1].
>
> What I want to achieve: if the client sends Cache-Control: max-age=60, TTL
> becomes 60 s. If the cache hit is fresher than 60 s, deliver it, otherwise
> fetch a new response from backend (I hope I'm not misusing the VCL terms
> here) *and* cache it.
>
> I had hacked this together in the vcl_fetch section in Varnish 3.x by
> setting the req.http.Cache-Control max-age value as beresp.ttl, but
> vcl_fetch is gone in Varnish 4.x.
>
> I have received a suggestion to use vcl_hit and/or grace [2], but again --
> no examples...
>
> Could anyone provide some VCL pseudo-code that uses req.http.Cache-Control
> value to override TTL? max-age number parsing not necessary, I have figure
> that out.
>
> Thanks,
>
> Martynas
>
> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/
> Headers/Cache-Control#Cache_request_directives
> [2] https://github.com/varnishcache/varnish-cache/
> issues/2014#issuecomment-319096566
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
Thanks. What was mostly unclear to me is passing the req header value all
the way to where it's used to set TTL.

Why doesn't bereq contain the req headers? At least Cache-Control is gone.

But I guess that can be done using obj.ttl, which I didn't know about. Any
documentation on that?
On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> On github I pointed to the doc explaining how you can return(fetch) to
> ignore a cached object, possibly based on ttl, so you already have half the
> answer.
>
> The other part of the equation is just converting req.http.cache-control
> to a duration and comparing that to obj.ttl. It will be similar to what you
> have done on v3.
>
> --
> Guillaume Quintard
>
> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
> wrote:
>
>> Hi,
>>
>> I have been reading quite a bit about Varnish and VCL but found almost no
>> examples with Cache-Control coming from the client request [1].
>>
>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>> otherwise fetch a new response from backend (I hope I'm not misusing the
>> VCL terms here) *and* cache it.
>>
>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>> vcl_fetch is gone in Varnish 4.x.
>>
>> I have received a suggestion to use vcl_hit and/or grace [2], but again
>> -- no examples...
>>
>> Could anyone provide some VCL pseudo-code that
>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>> not necessary, I have figure that out.
>>
>> Thanks,
>>
>> Martynas
>>
>> [1]
>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#Cache_request_directives
>> [2]
>> https://github.com/varnishcache/varnish-cache/issues/2014#issuecomment-319096566
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
man vcl

bereq is filtered to avoid side effects of the client forcing the ttl to
the backed.

Anyway, by the time you have access to bereq, it's too late for you since
the decision to go to the backend has already been been made.

--
Guillaume Quintard


On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas@atomgraph.com> wrote:

Thanks. What was mostly unclear to me is passing the req header value all
the way to where it's used to set TTL.

Why doesn't bereq contain the req headers? At least Cache-Control is gone.

But I guess that can be done using obj.ttl, which I didn't know about. Any
documentation on that?

On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> On github I pointed to the doc explaining how you can return(fetch) to
> ignore a cached object, possibly based on ttl, so you already have half the
> answer.
>
> The other part of the equation is just converting req.http.cache-control
> to a duration and comparing that to obj.ttl. It will be similar to what you
> have done on v3.
>
> --
> Guillaume Quintard
>
> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
> wrote:
>
>> Hi,
>>
>> I have been reading quite a bit about Varnish and VCL but found almost no
>> examples with Cache-Control coming from the client request [1].
>>
>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>> otherwise fetch a new response from backend (I hope I'm not misusing the
>> VCL terms here) *and* cache it.
>>
>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>> vcl_fetch is gone in Varnish 4.x.
>>
>> I have received a suggestion to use vcl_hit and/or grace [2], but again
>> -- no examples...
>>
>> Could anyone provide some VCL pseudo-code that
>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>> not necessary, I have figure that out.
>>
>> Thanks,
>>
>> Martynas
>>
>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/
>> Headers/Cache-Control#Cache_request_directives
>> [2] https://github.com/varnishcache/varnish-cache/
>> issues/2014#issuecomment-319096566
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
Thanks Guillaume.

First I tried

sub vcl_hit {
if (req.http.Cache-Control ~ "max-age=[0-9]*") {
set req.http.Max-Age = regsub(req.http.Cache-Control,
"max-age=([0-9]*)", "\1");
if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
req.http.Max-Age);
return(fetch);
}
}
...

but I got an error:

- VCL_call HIT
- ReqHeader Max-Age: 69
- VCL_Log obj.age: 102.306 req.http.Max-Age: 69
- VCL_return fetch
- VCL_Error change return(fetch) to return(miss) in vcl_hit{}
- VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
- VCL_call PASS
- VCL_return fetch

I did as told and I tried


On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> man vcl
>
> bereq is filtered to avoid side effects of the client forcing the ttl to
> the backed.
>
> Anyway, by the time you have access to bereq, it's too late for you since
> the decision to go to the backend has already been been made.
>
> --
> Guillaume Quintard
>
>
> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas@atomgraph.com>
> wrote:
>
> Thanks. What was mostly unclear to me is passing the req header value all
> the way to where it's used to set TTL.
>
> Why doesn't bereq contain the req headers? At least Cache-Control is gone.
>
> But I guess that can be done using obj.ttl, which I didn't know about. Any
> documentation on that?
>
> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> On github I pointed to the doc explaining how you can return(fetch) to
>> ignore a cached object, possibly based on ttl, so you already have half the
>> answer.
>>
>> The other part of the equation is just converting req.http.cache-control
>> to a duration and comparing that to obj.ttl. It will be similar to what you
>> have done on v3.
>>
>> --
>> Guillaume Quintard
>>
>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have been reading quite a bit about Varnish and VCL but found almost
>>> no examples with Cache-Control coming from the client request [1].
>>>
>>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>>> otherwise fetch a new response from backend (I hope I'm not misusing the
>>> VCL terms here) *and* cache it.
>>>
>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>> vcl_fetch is gone in Varnish 4.x.
>>>
>>> I have received a suggestion to use vcl_hit and/or grace [2], but again
>>> -- no examples...
>>>
>>> Could anyone provide some VCL pseudo-code that
>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>> not necessary, I have figure that out.
>>>
>>> Thanks,
>>>
>>> Martynas
>>>
>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>> s/Cache-Control#Cache_request_directives
>>> [2] https://github.com/varnishcache/varnish-cache/issues/
>>> 2014#issuecomment-319096566
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
Sorry, sent too soon. Here it goes:

Thanks Guillaume.

First I tried return(fetch) as you suggested

sub vcl_hit {
if (req.http.Cache-Control ~ "max-age=[0-9]*") {
set req.http.Max-Age = regsub(req.http.Cache-Control,
"max-age=([0-9]*)", "\1");
if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
req.http.Max-Age);
return(fetch);
}
}
...

but I got an error:

- VCL_call HIT
- ReqHeader Max-Age: 69
- VCL_Log obj.age: 102.306 req.http.Max-Age: 69
- VCL_return fetch
- VCL_Error change return(fetch) to return(miss) in vcl_hit{}
- VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
- VCL_call PASS
- VCL_return fetch

I did as told and I tried return(miss)

sub vcl_hit {
if (req.http.Cache-Control ~ "max-age=[0-9]*") {
set req.http.Max-Age = regsub(req.http.Cache-Control,
"max-age=([0-9]*)", "\1");
if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
req.http.Max-Age);
return(miss);
}
}
...

but then I got another error:

- VCL_call HIT
- ReqHeader Max-Age: 69
- VCL_Log obj.age: 195.391 req.http.Max-Age: 69
- VCL_return miss
- VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
- VCL_call PASS
- VCL_return fetch

So it looks like the max-age logic is triggered correctly, but what is
wrong with the return values?

On Tue, Aug 1, 2017 at 12:01 AM, Martynas Jusevi?ius <martynas@atomgraph.com
> wrote:

> Thanks Guillaume.
>
> First I tried
>
> sub vcl_hit {
> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
> set req.http.Max-Age = regsub(req.http.Cache-Control,
> "max-age=([0-9]*)", "\1");
> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
> req.http.Max-Age);
> return(fetch);
> }
> }
> ...
>
> but I got an error:
>
> - VCL_call HIT
> - ReqHeader Max-Age: 69
> - VCL_Log obj.age: 102.306 req.http.Max-Age: 69
> - VCL_return fetch
> - VCL_Error change return(fetch) to return(miss) in vcl_hit{}
> - VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
> - VCL_call PASS
> - VCL_return fetch
>
> I did as told and I tried
>
>
> On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> man vcl
>>
>> bereq is filtered to avoid side effects of the client forcing the ttl to
>> the backed.
>>
>> Anyway, by the time you have access to bereq, it's too late for you since
>> the decision to go to the backend has already been been made.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>> wrote:
>>
>> Thanks. What was mostly unclear to me is passing the req header value all
>> the way to where it's used to set TTL.
>>
>> Why doesn't bereq contain the req headers? At least Cache-Control is gone.
>>
>> But I guess that can be done using obj.ttl, which I didn't know about.
>> Any documentation on that?
>>
>> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
>> guillaume@varnish-software.com> wrote:
>>
>>> On github I pointed to the doc explaining how you can return(fetch) to
>>> ignore a cached object, possibly based on ttl, so you already have half the
>>> answer.
>>>
>>> The other part of the equation is just converting req.http.cache-control
>>> to a duration and comparing that to obj.ttl. It will be similar to what you
>>> have done on v3.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have been reading quite a bit about Varnish and VCL but found almost
>>>> no examples with Cache-Control coming from the client request [1].
>>>>
>>>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>>>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>>>> otherwise fetch a new response from backend (I hope I'm not misusing the
>>>> VCL terms here) *and* cache it.
>>>>
>>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>>> vcl_fetch is gone in Varnish 4.x.
>>>>
>>>> I have received a suggestion to use vcl_hit and/or grace [2], but again
>>>> -- no examples...
>>>>
>>>> Could anyone provide some VCL pseudo-code that
>>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>>> not necessary, I have figure that out.
>>>>
>>>> Thanks,
>>>>
>>>> Martynas
>>>>
>>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>>> s/Cache-Control#Cache_request_directives
>>>> [2] https://github.com/varnishcache/varnish-cache/issues/201
>>>> 4#issuecomment-319096566
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>
>>
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
Ah, right, I totally forgot about that, sorry.

Sooooooo, there's no real clean way to do it (that I can see, smarter
people than me may have a solution), but here's what I can offer.

sub vcl_recv {
if (req.restarts == 0) {
unset req.http.force_miss;
} else if (req.http.force_miss) {
set req.hash_alway_miss
}
}

sub vcl_hit {
if (CONDITION TO BYPASS CACHE) {
set req.http.force_miss = "yes";
return(restart);
}
}



--
Guillaume Quintard

On Tue, Aug 1, 2017 at 12:10 AM, Martynas Jusevi?ius <martynas@atomgraph.com
> wrote:

> Sorry, sent too soon. Here it goes:
>
> Thanks Guillaume.
>
> First I tried return(fetch) as you suggested
>
> sub vcl_hit {
> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
> set req.http.Max-Age = regsub(req.http.Cache-Control,
> "max-age=([0-9]*)", "\1");
> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
> req.http.Max-Age);
> return(fetch);
> }
> }
> ...
>
> but I got an error:
>
> - VCL_call HIT
> - ReqHeader Max-Age: 69
> - VCL_Log obj.age: 102.306 req.http.Max-Age: 69
> - VCL_return fetch
> - VCL_Error change return(fetch) to return(miss) in vcl_hit{}
> - VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
> - VCL_call PASS
> - VCL_return fetch
>
> I did as told and I tried return(miss)
>
> sub vcl_hit {
> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
> set req.http.Max-Age = regsub(req.http.Cache-Control,
> "max-age=([0-9]*)", "\1");
> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
> req.http.Max-Age);
> return(miss);
> }
> }
> ...
>
> but then I got another error:
>
> - VCL_call HIT
> - ReqHeader Max-Age: 69
> - VCL_Log obj.age: 195.391 req.http.Max-Age: 69
> - VCL_return miss
> - VCL_Error vcl_hit{} returns miss without busy object. Doing pass.
> - VCL_call PASS
> - VCL_return fetch
>
> So it looks like the max-age logic is triggered correctly, but what is
> wrong with the return values?
>
> On Tue, Aug 1, 2017 at 12:01 AM, Martynas Jusevi?ius <
> martynas@atomgraph.com> wrote:
>
>> Thanks Guillaume.
>>
>> First I tried
>>
>> sub vcl_hit {
>> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>> set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>> return(fetch);
>> }
>> }
>> ...
>>
>> but I got an error:
>>
>> - VCL_call HIT
>> - ReqHeader Max-Age: 69
>> - VCL_Log obj.age: 102.306 req.http.Max-Age: 69
>> - VCL_return fetch
>> - VCL_Error change return(fetch) to return(miss) in vcl_hit{}
>> - VCL_Error vcl_hit{} returns miss without busy object. Doing
>> pass.
>> - VCL_call PASS
>> - VCL_return fetch
>>
>> I did as told and I tried
>>
>>
>> On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
>> guillaume@varnish-software.com> wrote:
>>
>>> man vcl
>>>
>>> bereq is filtered to avoid side effects of the client forcing the ttl to
>>> the backed.
>>>
>>> Anyway, by the time you have access to bereq, it's too late for you
>>> since the decision to go to the backend has already been been made.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>>
>>> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>>> wrote:
>>>
>>> Thanks. What was mostly unclear to me is passing the req header value
>>> all the way to where it's used to set TTL.
>>>
>>> Why doesn't bereq contain the req headers? At least Cache-Control is
>>> gone.
>>>
>>> But I guess that can be done using obj.ttl, which I didn't know about.
>>> Any documentation on that?
>>>
>>> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
>>> guillaume@varnish-software.com> wrote:
>>>
>>>> On github I pointed to the doc explaining how you can return(fetch) to
>>>> ignore a cached object, possibly based on ttl, so you already have half the
>>>> answer.
>>>>
>>>> The other part of the equation is just converting
>>>> req.http.cache-control to a duration and comparing that to obj.ttl. It will
>>>> be similar to what you have done on v3.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have been reading quite a bit about Varnish and VCL but found almost
>>>>> no examples with Cache-Control coming from the client request [1].
>>>>>
>>>>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>>>>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>>>>> otherwise fetch a new response from backend (I hope I'm not misusing the
>>>>> VCL terms here) *and* cache it.
>>>>>
>>>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>>>> vcl_fetch is gone in Varnish 4.x.
>>>>>
>>>>> I have received a suggestion to use vcl_hit and/or grace [2], but
>>>>> again -- no examples...
>>>>>
>>>>> Could anyone provide some VCL pseudo-code that
>>>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>>>> not necessary, I have figure that out.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martynas
>>>>>
>>>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>>>> s/Cache-Control#Cache_request_directives
>>>>> [2] https://github.com/varnishcache/varnish-cache/issues/201
>>>>> 4#issuecomment-319096566
>>>>>
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc@varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>
>>>
>>
>
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
On Tue, Aug 1, 2017 at 9:34 AM, Guillaume Quintard
<guillaume@varnish-software.com> wrote:
> Ah, right, I totally forgot about that, sorry.
>
> Sooooooo, there's no real clean way to do it (that I can see, smarter people
> than me may have a solution), but here's what I can offer.

First, I would say that honoring a client's cache-control is a
terrible idea: it's a glorified DoS vector.

Now if you really want to do that (eg. you have a trustworthy client,
say a script, that drives caching decisions) you still can't outlive
the object's TTL (the one that was picked at the end of
vcl_backend_getch). Of course you need to assert the trustworthiness
of such requests too. It becomes quite cumbersome to allow clients to
make decisions.

Sorry, I can't help, I will only say "don't do it" :)

Cheers

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL [ In reply to ]
Guillaume,

after I fixed a couple of typos, this *seems* to work now:

sub vcl_recv {
...
if (req.restarts == 0) {
unset req.http.force_miss;

if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
else if (req.http.force_miss) {
set req.hash_always_miss = true;
}
...


sub vcl_hit {
if (req.http.Cache-Control ~ "max-age=[0-9]*") {
set req.http.Max-Age = regsub(req.http.Cache-Control,
"max-age=([0-9]*)", "\1");
if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {

set req.http.force_miss = "yes";
return(restart);
}
}
...

After return(restart) in vcl_hit I can see a new BeReq which calls
BACKEND_FETCH, new BeResp which calls BACKEND_RESPONSE, and then restarted
Request which triggers hash_always_miss and then calls RECV, HASH, MISS,
and finally DELIVER.

If that looks right, it would be nice to have it documented somewhere for
future reference.

Thanks for your help.


On Tue, Aug 1, 2017 at 9:34 AM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Ah, right, I totally forgot about that, sorry.
>
> Sooooooo, there's no real clean way to do it (that I can see, smarter
> people than me may have a solution), but here's what I can offer.
>
> sub vcl_recv {
> if (req.restarts == 0) {
> unset req.http.force_miss;
> } else if (req.http.force_miss) {
> set req.hash_alway_miss
> }
> }
>
> sub vcl_hit {
> if (CONDITION TO BYPASS CACHE) {
> set req.http.force_miss = "yes";
> return(restart);
> }
> }
>
>
>
> --
> Guillaume Quintard
>
> On Tue, Aug 1, 2017 at 12:10 AM, Martynas Jusevi?ius <
> martynas@atomgraph.com> wrote:
>
>> Sorry, sent too soon. Here it goes:
>>
>> Thanks Guillaume.
>>
>> First I tried return(fetch) as you suggested
>>
>> sub vcl_hit {
>> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>> set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>> return(fetch);
>> }
>> }
>> ...
>>
>> but I got an error:
>>
>> - VCL_call HIT
>> - ReqHeader Max-Age: 69
>> - VCL_Log obj.age: 102.306 req.http.Max-Age: 69
>> - VCL_return fetch
>> - VCL_Error change return(fetch) to return(miss) in vcl_hit{}
>> - VCL_Error vcl_hit{} returns miss without busy object. Doing
>> pass.
>> - VCL_call PASS
>> - VCL_return fetch
>>
>> I did as told and I tried return(miss)
>>
>> sub vcl_hit {
>> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>> set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>> return(miss);
>> }
>> }
>> ...
>>
>> but then I got another error:
>>
>> - VCL_call HIT
>> - ReqHeader Max-Age: 69
>> - VCL_Log obj.age: 195.391 req.http.Max-Age: 69
>> - VCL_return miss
>> - VCL_Error vcl_hit{} returns miss without busy object. Doing
>> pass.
>> - VCL_call PASS
>> - VCL_return fetch
>>
>> So it looks like the max-age logic is triggered correctly, but what is
>> wrong with the return values?
>>
>> On Tue, Aug 1, 2017 at 12:01 AM, Martynas Jusevi?ius <
>> martynas@atomgraph.com> wrote:
>>
>>> Thanks Guillaume.
>>>
>>> First I tried
>>>
>>> sub vcl_hit {
>>> if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>>> set req.http.Max-Age = regsub(req.http.Cache-Control,
>>> "max-age=([0-9]*)", "\1");
>>> if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>>> std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>>> req.http.Max-Age);
>>> return(fetch);
>>> }
>>> }
>>> ...
>>>
>>> but I got an error:
>>>
>>> - VCL_call HIT
>>> - ReqHeader Max-Age: 69
>>> - VCL_Log obj.age: 102.306 req.http.Max-Age: 69
>>> - VCL_return fetch
>>> - VCL_Error change return(fetch) to return(miss) in vcl_hit{}
>>> - VCL_Error vcl_hit{} returns miss without busy object. Doing
>>> pass.
>>> - VCL_call PASS
>>> - VCL_return fetch
>>>
>>> I did as told and I tried
>>>
>>>
>>> On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
>>> guillaume@varnish-software.com> wrote:
>>>
>>>> man vcl
>>>>
>>>> bereq is filtered to avoid side effects of the client forcing the ttl
>>>> to the backed.
>>>>
>>>> Anyway, by the time you have access to bereq, it's too late for you
>>>> since the decision to go to the backend has already been been made.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>>
>>>> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>>>> wrote:
>>>>
>>>> Thanks. What was mostly unclear to me is passing the req header value
>>>> all the way to where it's used to set TTL.
>>>>
>>>> Why doesn't bereq contain the req headers? At least Cache-Control is
>>>> gone.
>>>>
>>>> But I guess that can be done using obj.ttl, which I didn't know about.
>>>> Any documentation on that?
>>>>
>>>> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
>>>> guillaume@varnish-software.com> wrote:
>>>>
>>>>> On github I pointed to the doc explaining how you can return(fetch) to
>>>>> ignore a cached object, possibly based on ttl, so you already have half the
>>>>> answer.
>>>>>
>>>>> The other part of the equation is just converting
>>>>> req.http.cache-control to a duration and comparing that to obj.ttl. It will
>>>>> be similar to what you have done on v3.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas@atomgraph.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have been reading quite a bit about Varnish and VCL but found
>>>>>> almost no examples with Cache-Control coming from the client request [1].
>>>>>>
>>>>>> What I want to achieve: if the client sends Cache-Control:
>>>>>> max-age=60, TTL becomes 60 s. If the cache hit is fresher than 60 s,
>>>>>> deliver it, otherwise fetch a new response from backend (I hope I'm not
>>>>>> misusing the VCL terms here) *and* cache it.
>>>>>>
>>>>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>>>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>>>>> vcl_fetch is gone in Varnish 4.x.
>>>>>>
>>>>>> I have received a suggestion to use vcl_hit and/or grace [2], but
>>>>>> again -- no examples...
>>>>>>
>>>>>> Could anyone provide some VCL pseudo-code that
>>>>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>>>>> not necessary, I have figure that out.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Martynas
>>>>>>
>>>>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>>>>> s/Cache-Control#Cache_request_directives
>>>>>> [2] https://github.com/varnishcache/varnish-cache/issues/201
>>>>>> 4#issuecomment-319096566
>>>>>>
>>>>>> _______________________________________________
>>>>>> varnish-misc mailing list
>>>>>> varnish-misc@varnish-cache.org
>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>
>>>>>
>>>>
>>>
>>
>