Mailing List Archive: Same URL: Don't cache with store response header, but cache when it's not there

Same URL: Don't cache with store response header, but cache when it's not there

Jan 20, 2019, 4:12 AM

Post #1 of 4 (657 views)

Hi Everyone,

I am unable to get caching to work in a particular scenario.

I need to cache the url eg: /app/profile/GDPR1?hostingUrl=http%3A%2F%
2Fitvidffya.blogspot.com%2F2019%2F01%2Fvar-nish-issue-fix.html

However, this is embedded in a 3rd party site (www.anotherdomain.com calls
mydomain.com in iframe ) so on Apple Safari we can't drop cookies ( as 3rd
party cookies are blocked by default ).

As a result, for Apple Safari, when this url is hit, our backend returns
top.location.href='someurlonoursite_to_set_cookie' which is fired in
context of 3rd party site since they embed using our javascript.

This way a cookie is dropped on our domain and user gets back to the url
inside the iframe.

Now, when I hit this url in chrome, it always picks up
top.location.href='....' as it got cached by safari.

But, chrome was supposed to get the actual page content since it's not
blocking 3rd party cookies.

So, I went ahead and added a custom header, "store" for cases of apple
safari in our backend.
I skip unsetting cookie (hence varnish cache) for this url in this case.

But, it still doesn't cache on chrome in subsequent hits.

Always goes to backend, never goes to the cache.

Is it because I have one url -
- that can return 2 responses based on browser
- I told it to not cache first time when store header was there, but when
store header is not there i ask it to cache it.

Still doesn't work.

Config

sub vcl_recv {
std.log("vtglog: in vcl_recv " + req.url);
# Only cache GET or HEAD requests. This makes sure the POST requests
are always passed.
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}

if(req.http.host == "sqa03.mydomain.com" && req.url ~ "/app/profile"){
std.log("vtglog: unsetting cookies");
unset req.http.cookie;
} else{
return(pass);
}

if (req.http.Authorization) {
# Not cacheable by default
return (pass);
}
}

sub vcl_backend_response {
std.log("vtglog: vcl_backend_response" + bereq.url) ;
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
"/app/profile") {
std.log("vtglog: inside condition in backend response");
std.log("vtglog: store header value is " + beresp.http.store);
if ( beresp.http.Set-Cookie ) {
if ( !beresp.http.store ) {
std.log("vtglog: since no store headers, cache it by
unsetting cookies");
unset beresp.http.Set-Cookie;
} else {
std.log("vtglog: store header found, dont cache");
}
}
}

if (beresp.status == 301 || beresp.status == 302) {
set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+",
"");
}

# Don't cache 50x responses
if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
503 || beresp.status == 504) {
return (abandon);
}
}

Re: Same URL: Don't cache with store response header, but cache when it's not there [ In reply to ]

guillaume at varnish-software

Jan 20, 2019, 7:39 PM

Post #2 of 4 (656 views)

Permalink

Hi,

> Is it because I have one url -
> - that can return 2 responses based on browser

That's the one, you need to tell varnish to care about the browser. You
have two ways to do that: either put that in in the hash key directly, or
put it in in a header, and tell that to varnish via the vary header.

First option looks like this:

sub vcl_recv {
if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
set req.http.browser = "safari";
} else {
set req.http.browser = "somethingelse";
}
}

sub vcl_hash {
hash_data(req.http.browser);
# do NOT return, let the built-in vcl run
}

The pro is that you don't need to change the backend, the con is that if
you purge, you have to purge twice because they are two different objects.

The second option is to tell varnish that the browser is important, but
just a variant of the same object:

sub vcl_recv {
if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
set req.http.browser = "safari";
} else {
set req.http.browser = "somethingelse";
}
}

sub vcl_backend_response {
set beresp.http.vary = beresp.http.vary + ",browser";
}

Note that you can also have the backend return the vary header for you
directly

There are a couple of cleaner versions, but they are a bit more involved,
so let's start with that.

Side note: I understand where you are coming from with the std.log
debugging, but that info is actually super redundant with what's in the log.

--
Guillaume Quintard

On Sun, Jan 20, 2019 at 4:14 AM Maninder Singh <mandys@gmail.com> wrote:

> Hi Everyone,
>
> I am unable to get caching to work in a particular scenario.
>
> I need to cache the url eg: /app/profile/GDPR1?hostingUrl=http%3A%2F%
> 2Fitvidffya.blogspot.com%2F2019%2F01%2Fvar-nish-issue-fix.html
>
> However, this is embedded in a 3rd party site (www.anotherdomain.com
> calls mydomain.com in iframe ) so on Apple Safari we can't drop cookies (
> as 3rd party cookies are blocked by default ).
>
> As a result, for Apple Safari, when this url is hit, our backend returns
> top.location.href='someurlonoursite_to_set_cookie' which is fired in
> context of 3rd party site since they embed using our javascript.
>
> This way a cookie is dropped on our domain and user gets back to the url
> inside the iframe.
>
> Now, when I hit this url in chrome, it always picks up
> top.location.href='....' as it got cached by safari.
>
> But, chrome was supposed to get the actual page content since it's not
> blocking 3rd party cookies.
>
> So, I went ahead and added a custom header, "store" for cases of apple
> safari in our backend.
> I skip unsetting cookie (hence varnish cache) for this url in this case.
>
> But, it still doesn't cache on chrome in subsequent hits.
>
> Always goes to backend, never goes to the cache.
>
> Is it because I have one url -
> - that can return 2 responses based on browser
> - I told it to not cache first time when store header was there, but when
> store header is not there i ask it to cache it.
>
> Still doesn't work.
>
> Config
>
> sub vcl_recv {
> std.log("vtglog: in vcl_recv " + req.url);
> # Only cache GET or HEAD requests. This makes sure the POST requests
> are always passed.
> if (req.method != "GET" && req.method != "HEAD") {
> return (pass);
> }
>
> if(req.http.host == "sqa03.mydomain.com" && req.url ~ "/app/profile"){
> std.log("vtglog: unsetting cookies");
> unset req.http.cookie;
> } else{
> return(pass);
> }
>
> if (req.http.Authorization) {
> # Not cacheable by default
> return (pass);
> }
> }
>
> sub vcl_backend_response {
> std.log("vtglog: vcl_backend_response" + bereq.url) ;
> # Happens after we have read the response headers from the backend.
> #
> # Here you clean the response headers, removing silly Set-Cookie
> headers
> # and other mistakes your backend does.
> if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
> "/app/profile") {
> std.log("vtglog: inside condition in backend response");
> std.log("vtglog: store header value is " + beresp.http.store);
> if ( beresp.http.Set-Cookie ) {
> if ( !beresp.http.store ) {
> std.log("vtglog: since no store headers, cache it by
> unsetting cookies");
> unset beresp.http.Set-Cookie;
> } else {
> std.log("vtglog: store header found, dont cache");
> }
> }
> }
>
> if (beresp.status == 301 || beresp.status == 302) {
> set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+",
> "");
> }
>
> # Don't cache 50x responses
> if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
> 503 || beresp.status == 504) {
> return (abandon);
> }
> }
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Re: Same URL: Don't cache with store response header, but cache when it's not there [ In reply to ]

mandys at gmail

Jan 22, 2019, 8:53 PM

Post #3 of 4 (650 views)

Permalink

Thank you Guillaume for your suggestions.

This seemed to have worked.

I am now using the following vcl config.

>> I understand where you are coming from with the std.log debugging, but
that info is actually super redundant with what's in the log.
I know. But, varnishlog has lot of info and this was done so that I could
just do varnishlog | grep "vtglog" to see the flow as per my liking :-)
This has now been removed.

This should now keep 2 copies of the same url - one for Safari and one for
the rest - right ?

Question:
What's a good way to check how many cache copies are present for a url ?

Once again thank you for your help!

- Mandy.

vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}

sub vcl_recv {
if (req.http.User-Agent ~ "Safari") {
if (req.http.User-Agent ~ "Chrome") {
set req.http.browser = "other";
} else {
set req.http.browser = "safari";
}
} else {
set req.http.browser = "other";
}

# Only cache GET or HEAD requests. This makes sure the POST requests
are always passed.
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}

if(req.http.host == "sqa03.mydomain.com" && req.url ~ "/app/profile"){
unset req.http.cookie;
} else{
return(pass);
}

if (req.http.Authorization) {
# Not cacheable by default
return (pass);
}
}

sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
"/app/profile") {
set beresp.ttl = 180 s;
if ( beresp.http.Set-Cookie ) {
unset beresp.http.Set-Cookie;
}
}

if (beresp.status == 301 || beresp.status == 302) {
set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+",
"");
}

# Don't cache 50x responses
if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
503 || beresp.status == 504) {
return (abandon);
}
}

sub vcl_hash {
hash_data(req.http.browser);
}

sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
#
# You can do accounting or modifying the final object here.
if (obj.hits > 0) {
# Add debug header to see if it's a HIT/MISS and the number of hits,
disable when not needed
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}

set resp.http.X-Cache-Hits = obj.hits;
}

On Mon, 21 Jan 2019 at 09:09, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Hi,
>
> > Is it because I have one url -
> > - that can return 2 responses based on browser
>
> That's the one, you need to tell varnish to care about the browser. You
> have two ways to do that: either put that in in the hash key directly, or
> put it in in a header, and tell that to varnish via the vary header.
>
> First option looks like this:
>
> sub vcl_recv {
> if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
> set req.http.browser = "safari";
> } else {
> set req.http.browser = "somethingelse";
> }
> }
>
> sub vcl_hash {
> hash_data(req.http.browser);
> # do NOT return, let the built-in vcl run
> }
>
> The pro is that you don't need to change the backend, the con is that if
> you purge, you have to purge twice because they are two different objects.
>
> The second option is to tell varnish that the browser is important, but
> just a variant of the same object:
>
> sub vcl_recv {
> if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
> set req.http.browser = "safari";
> } else {
> set req.http.browser = "somethingelse";
> }
> }
>
> sub vcl_backend_response {
> set beresp.http.vary = beresp.http.vary + ",browser";
> }
>
> Note that you can also have the backend return the vary header for you
> directly
>
> There are a couple of cleaner versions, but they are a bit more involved,
> so let's start with that.
>
> Side note: I understand where you are coming from with the std.log
> debugging, but that info is actually super redundant with what's in the log.
>
> --
> Guillaume Quintard
>
>
> On Sun, Jan 20, 2019 at 4:14 AM Maninder Singh <mandys@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I am unable to get caching to work in a particular scenario.
>>
>> I need to cache the url eg: /app/profile/GDPR1?hostingUrl=http%3A%2F%
>> 2Fitvidffya.blogspot.com%2F2019%2F01%2Fvar-nish-issue-fix.html
>>
>> However, this is embedded in a 3rd party site (www.anotherdomain.com
>> calls mydomain.com in iframe ) so on Apple Safari we can't drop cookies
>> ( as 3rd party cookies are blocked by default ).
>>
>> As a result, for Apple Safari, when this url is hit, our backend returns
>> top.location.href='someurlonoursite_to_set_cookie' which is fired in
>> context of 3rd party site since they embed using our javascript.
>>
>> This way a cookie is dropped on our domain and user gets back to the url
>> inside the iframe.
>>
>> Now, when I hit this url in chrome, it always picks up
>> top.location.href='....' as it got cached by safari.
>>
>> But, chrome was supposed to get the actual page content since it's not
>> blocking 3rd party cookies.
>>
>> So, I went ahead and added a custom header, "store" for cases of apple
>> safari in our backend.
>> I skip unsetting cookie (hence varnish cache) for this url in this case.
>>
>> But, it still doesn't cache on chrome in subsequent hits.
>>
>> Always goes to backend, never goes to the cache.
>>
>> Is it because I have one url -
>> - that can return 2 responses based on browser
>> - I told it to not cache first time when store header was there, but when
>> store header is not there i ask it to cache it.
>>
>> Still doesn't work.
>>
>> Config
>>
>> sub vcl_recv {
>> std.log("vtglog: in vcl_recv " + req.url);
>> # Only cache GET or HEAD requests. This makes sure the POST requests
>> are always passed.
>> if (req.method != "GET" && req.method != "HEAD") {
>> return (pass);
>> }
>>
>> if(req.http.host == "sqa03.mydomain.com" && req.url ~
>> "/app/profile"){
>> std.log("vtglog: unsetting cookies");
>> unset req.http.cookie;
>> } else{
>> return(pass);
>> }
>>
>> if (req.http.Authorization) {
>> # Not cacheable by default
>> return (pass);
>> }
>> }
>>
>> sub vcl_backend_response {
>> std.log("vtglog: vcl_backend_response" + bereq.url) ;
>> # Happens after we have read the response headers from the backend.
>> #
>> # Here you clean the response headers, removing silly Set-Cookie
>> headers
>> # and other mistakes your backend does.
>> if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
>> "/app/profile") {
>> std.log("vtglog: inside condition in backend response");
>> std.log("vtglog: store header value is " + beresp.http.store);
>> if ( beresp.http.Set-Cookie ) {
>> if ( !beresp.http.store ) {
>> std.log("vtglog: since no store headers, cache it by
>> unsetting cookies");
>> unset beresp.http.Set-Cookie;
>> } else {
>> std.log("vtglog: store header found, dont cache");
>> }
>> }
>> }
>>
>> if (beresp.status == 301 || beresp.status == 302) {
>> set beresp.http.Location = regsub(beresp.http.Location,
>> ":[0-9]+", "");
>> }
>>
>> # Don't cache 50x responses
>> if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
>> 503 || beresp.status == 504) {
>> return (abandon);
>> }
>> }
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>

Re: Same URL: Don't cache with store response header, but cache when it's not there [ In reply to ]

guillaume at varnish-software

Jan 25, 2019, 1:24 PM

Post #4 of 4 (646 views)

Permalink

> I know. But, varnishlog has lot of info and this was done so that I could
just do varnishlog | grep "vtglog" to see the flow as per my liking :-)
> This has now been removed.

varnishlog -i NameOfTheTag is your friend

> This should now keep 2 copies of the same url - one for Safari and one
for the rest - right ?

Yessir

> What's a good way to check how many cache copies are present for a url ?

that's not something you can do at the moment

--
Guillaume Quintard

On Tue, Jan 22, 2019 at 8:54 PM Maninder Singh <mandys@gmail.com> wrote:

> Thank you Guillaume for your suggestions.
>
> This seemed to have worked.
>
> I am now using the following vcl config.
>
> that info is actually super redundant with what's in the log.
> I know. But, varnishlog has lot of info and this was done so that I could
> just do varnishlog | grep "vtglog" to see the flow as per my liking :-)
> This has now been removed.
>
> This should now keep 2 copies of the same url - one for Safari and one for
> the rest - right ?
>
> Question:
> What's a good way to check how many cache copies are present for a url ?
>
> Once again thank you for your help!
>
> - Mandy.
>
> vcl 4.0;
>
> # Default backend definition. Set this to point to your content server.
> backend default {
> .host = "127.0.0.1";
> .port = "8080";
> }
>
> sub vcl_recv {
> if (req.http.User-Agent ~ "Safari") {
> if (req.http.User-Agent ~ "Chrome") {
> set req.http.browser = "other";
> } else {
> set req.http.browser = "safari";
> }
> } else {
> set req.http.browser = "other";
> }
>
> # Only cache GET or HEAD requests. This makes sure the POST requests
> are always passed.
> if (req.method != "GET" && req.method != "HEAD") {
> return (pass);
> }
>
> if(req.http.host == "sqa03.mydomain.com" && req.url ~ "/app/profile"){
> unset req.http.cookie;
> } else{
> return(pass);
> }
>
> if (req.http.Authorization) {
> # Not cacheable by default
> return (pass);
> }
> }
>
> sub vcl_backend_response {
> # Happens after we have read the response headers from the backend.
> #
> # Here you clean the response headers, removing silly Set-Cookie
> headers
> # and other mistakes your backend does.
> if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
> "/app/profile") {
> set beresp.ttl = 180 s;
> if ( beresp.http.Set-Cookie ) {
> unset beresp.http.Set-Cookie;
> }
> }
>
> if (beresp.status == 301 || beresp.status == 302) {
> set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+",
> "");
> }
>
> # Don't cache 50x responses
> if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
> 503 || beresp.status == 504) {
> return (abandon);
> }
> }
>
> sub vcl_hash {
> hash_data(req.http.browser);
> }
>
> sub vcl_deliver {
> # Happens when we have all the pieces we need, and are about to send
> the
> # response to the client.
> #
> # You can do accounting or modifying the final object here.
> if (obj.hits > 0) {
> # Add debug header to see if it's a HIT/MISS and the number of hits,
> disable when not needed
> set resp.http.X-Cache = "HIT";
> } else {
> set resp.http.X-Cache = "MISS";
> }
>
> set resp.http.X-Cache-Hits = obj.hits;
> }
>
> On Mon, 21 Jan 2019 at 09:09, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> Hi,
>>
>> > Is it because I have one url -
>> > - that can return 2 responses based on browser
>>
>> That's the one, you need to tell varnish to care about the browser. You
>> have two ways to do that: either put that in in the hash key directly, or
>> put it in in a header, and tell that to varnish via the vary header.
>>
>> First option looks like this:
>>
>> sub vcl_recv {
>> if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
>> set req.http.browser = "safari";
>> } else {
>> set req.http.browser = "somethingelse";
>> }
>> }
>>
>> sub vcl_hash {
>> hash_data(req.http.browser);
>> # do NOT return, let the built-in vcl run
>> }
>>
>> The pro is that you don't need to change the backend, the con is that if
>> you purge, you have to purge twice because they are two different objects.
>>
>> The second option is to tell varnish that the browser is important, but
>> just a variant of the same object:
>>
>> sub vcl_recv {
>> if (SOME_TEST_TO_CHECK_ITS_SAFARI) {
>> set req.http.browser = "safari";
>> } else {
>> set req.http.browser = "somethingelse";
>> }
>> }
>>
>> sub vcl_backend_response {
>> set beresp.http.vary = beresp.http.vary + ",browser";
>> }
>>
>> Note that you can also have the backend return the vary header for you
>> directly
>>
>> There are a couple of cleaner versions, but they are a bit more involved,
>> so let's start with that.
>>
>> Side note: I understand where you are coming from with the std.log
>> debugging, but that info is actually super redundant with what's in the log.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Sun, Jan 20, 2019 at 4:14 AM Maninder Singh <mandys@gmail.com> wrote:
>>
>>> Hi Everyone,
>>>
>>> I am unable to get caching to work in a particular scenario.
>>>
>>> I need to cache the url eg: /app/profile/GDPR1?hostingUrl=http%3A%2F%
>>> 2Fitvidffya.blogspot.com%2F2019%2F01%2Fvar-nish-issue-fix.html
>>>
>>> However, this is embedded in a 3rd party site (www.anotherdomain.com
>>> calls mydomain.com in iframe ) so on Apple Safari we can't drop cookies
>>> ( as 3rd party cookies are blocked by default ).
>>>
>>> As a result, for Apple Safari, when this url is hit, our backend returns
>>> top.location.href='someurlonoursite_to_set_cookie' which is fired in
>>> context of 3rd party site since they embed using our javascript.
>>>
>>> This way a cookie is dropped on our domain and user gets back to the url
>>> inside the iframe.
>>>
>>> Now, when I hit this url in chrome, it always picks up
>>> top.location.href='....' as it got cached by safari.
>>>
>>> But, chrome was supposed to get the actual page content since it's not
>>> blocking 3rd party cookies.
>>>
>>> So, I went ahead and added a custom header, "store" for cases of apple
>>> safari in our backend.
>>> I skip unsetting cookie (hence varnish cache) for this url in this case.
>>>
>>> But, it still doesn't cache on chrome in subsequent hits.
>>>
>>> Always goes to backend, never goes to the cache.
>>>
>>> Is it because I have one url -
>>> - that can return 2 responses based on browser
>>> - I told it to not cache first time when store header was there, but
>>> when store header is not there i ask it to cache it.
>>>
>>> Still doesn't work.
>>>
>>> Config
>>>
>>> sub vcl_recv {
>>> std.log("vtglog: in vcl_recv " + req.url);
>>> # Only cache GET or HEAD requests. This makes sure the POST requests
>>> are always passed.
>>> if (req.method != "GET" && req.method != "HEAD") {
>>> return (pass);
>>> }
>>>
>>> if(req.http.host == "sqa03.mydomain.com" && req.url ~
>>> "/app/profile"){
>>> std.log("vtglog: unsetting cookies");
>>> unset req.http.cookie;
>>> } else{
>>> return(pass);
>>> }
>>>
>>> if (req.http.Authorization) {
>>> # Not cacheable by default
>>> return (pass);
>>> }
>>> }
>>>
>>> sub vcl_backend_response {
>>> std.log("vtglog: vcl_backend_response" + bereq.url) ;
>>> # Happens after we have read the response headers from the backend.
>>> #
>>> # Here you clean the response headers, removing silly Set-Cookie
>>> headers
>>> # and other mistakes your backend does.
>>> if (bereq.http.host == "sqa03.mydomain.com" && bereq.url ~
>>> "/app/profile") {
>>> std.log("vtglog: inside condition in backend response");
>>> std.log("vtglog: store header value is " + beresp.http.store);
>>> if ( beresp.http.Set-Cookie ) {
>>> if ( !beresp.http.store ) {
>>> std.log("vtglog: since no store headers, cache it by
>>> unsetting cookies");
>>> unset beresp.http.Set-Cookie;
>>> } else {
>>> std.log("vtglog: store header found, dont cache");
>>> }
>>> }
>>> }
>>>
>>> if (beresp.status == 301 || beresp.status == 302) {
>>> set beresp.http.Location = regsub(beresp.http.Location,
>>> ":[0-9]+", "");
>>> }
>>>
>>> # Don't cache 50x responses
>>> if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
>>> 503 || beresp.status == 504) {
>>> return (abandon);
>>> }
>>> }
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>