Mailing List Archive

Hello all,
First let me say thanks to everyone for taking time to work on this
project...I am still in the testing phase, however if all goes well
Varnish is going to come in quite handy.

So here is my problem:

I would like to use varnish to cache urls for say a period of 30
minutes. The caching seems fine as long as I use the same browser to
request the url. But I am looking for a stop gap solution at this time
so in the event that the web server becomes unavailable....varnish can
serve the docs for a little while, so we can get things back in order.
Anyway..as it stands right now..I can request a url from wget or GET or
firefox... and turn off the webserver and I can refresh the page and all
is well.

What I cannot do...is request a url using GET on one machine..then stop
the webserver and request that same url from firefox (different machine)
and get a valid page...I get a 503 error...Here is the vcl.conf file
(which I took most of from the web) I am using:

Any help would be great!

Thanks,

Shain

sub vcl_recv {
if (req.request != "GET" && req.request != "HEAD") {
# PURGE request if zope asks nicely
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
lookup;
}
pipe;
}
if (req.http.Expect) {
pipe;
}

if (req.http.Authenticate || req.http.Authorization) {
pass;
}

# We only care about the "__ac.*" cookies, used for authentication
if (req.http.Cookie && req.http.Cookie ~
"__ac(|_(name|password|persistent))=") {
pass;
}

# File type that we will always cache
if (req.request == "GET" && req.url ~
"\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar)$")
{
lookup;
}

if (req.request == "POST") {
pipe;
}

# force lookup even when cookies are present
if (req.request == "GET" && req.http.cookie) {
lookup;
}
lookup;
}

sub vcl_fetch {

if (obj.ttl < 1800) {
set obj.ttl = 1800s;
}
}

# Do the PURGE thing
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged";
}
}

sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache";
}
}

Shain,

I can't say for certain without looking at it myself, but I wonder if
when you request the URL from the second machine using firefox you
are actually doing a "refresh" that's setting a Cache-Control header
that's forcing varnish to consult the origin server. I suggest
connecting to your varnish with "varnishlog" and watching the headers
that each client sends varnish. If you see headers like this:

13 RxHeader c Pragma: no-cache
13 RxHeader c Cache-Control: no-cache

Then you know that's what's happening. In general, it sounds like you
are looking for an "off-line" mode where the cache server simply
serves stale content if an origin server can not be reached. If that
feature is present in varnish, I'm not aware of it.

Adrian


On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:

> Hello all,
> First let me say thanks to everyone for taking time to work on this
> project...I am still in the testing phase, however if all goes well
> Varnish is going to come in quite handy.
>
> So here is my problem:
>
> I would like to use varnish to cache urls for say a period of 30
> minutes. The caching seems fine as long as I use the same browser to
> request the url. But I am looking for a stop gap solution at this
> time
> so in the event that the web server becomes unavailable....varnish can
> serve the docs for a little while, so we can get things back in order.
> Anyway..as it stands right now..I can request a url from wget or
> GET or
> firefox... and turn off the webserver and I can refresh the page
> and all
> is well.
>
> What I cannot do...is request a url using GET on one machine..then
> stop
> the webserver and request that same url from firefox (different
> machine)
> and get a valid page...I get a 503 error...Here is the vcl.conf file
> (which I took most of from the web) I am using:
>
> Any help would be great!
>
> Thanks,
>
> Shain
>
> sub vcl_recv {
> if (req.request != "GET" && req.request != "HEAD") {
> # PURGE request if zope asks nicely
> if (req.request == "PURGE") {
> if (!client.ip ~ purge) {
> error 405 "Not allowed.";
> }
> lookup;
> }
> pipe;
> }
> if (req.http.Expect) {
> pipe;
> }
>
> if (req.http.Authenticate || req.http.Authorization) {
> pass;
> }
>
> # We only care about the "__ac.*" cookies, used for
> authentication
> if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
> pass;
> }
>
> # File type that we will always cache
> if (req.request == "GET" && req.url ~
> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|
> swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|
> sxw|zip|gz|bz2|tgz|tar)$")
> {
> lookup;
> }
>
> if (req.request == "POST") {
> pipe;
> }
>
> # force lookup even when cookies are present
> if (req.request == "GET" && req.http.cookie) {
> lookup;
> }
> lookup;
> }
>
> sub vcl_fetch {
>
> if (obj.ttl < 1800) {
> set obj.ttl = 1800s;
> }
> }
>
> # Do the PURGE thing
> sub vcl_hit {
> if (req.request == "PURGE") {
> set obj.ttl = 0s;
> error 200 "Purged";
> }
> }
>
> sub vcl_miss {
> if (req.request == "PURGE") {
> error 404 "Not in cache";
> }
> }
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev

Shain,

I can't say for certain without looking at it myself, but I wonder if
when you request the URL from the second machine using firefox you
are actually doing a "refresh" that's setting a Cache-Control header
that's forcing varnish to consult the origin server. I suggest
connecting to your varnish with "varnishlog" and watching the headers
that each client sends varnish. If you see headers like this:

13 RxHeader c Pragma: no-cache
13 RxHeader c Cache-Control: no-cache

Then you know that's what's happening. In general, it sounds like you
are looking for an "off-line" mode where the cache server simply
serves stale content if an origin server can not be reached. If that
feature is present in varnish, I'm not aware of it.

Adrian


On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:

> Hello all,
> First let me say thanks to everyone for taking time to work on this
> project...I am still in the testing phase, however if all goes well
> Varnish is going to come in quite handy.
>
> So here is my problem:
>
> I would like to use varnish to cache urls for say a period of 30
> minutes. The caching seems fine as long as I use the same browser to
> request the url. But I am looking for a stop gap solution at this
> time
> so in the event that the web server becomes unavailable....varnish can
> serve the docs for a little while, so we can get things back in order.
> Anyway..as it stands right now..I can request a url from wget or
> GET or
> firefox... and turn off the webserver and I can refresh the page
> and all
> is well.
>
> What I cannot do...is request a url using GET on one machine..then
> stop
> the webserver and request that same url from firefox (different
> machine)
> and get a valid page...I get a 503 error...Here is the vcl.conf file
> (which I took most of from the web) I am using:
>
> Any help would be great!
>
> Thanks,
>
> Shain
>
> sub vcl_recv {
> if (req.request != "GET" && req.request != "HEAD") {
> # PURGE request if zope asks nicely
> if (req.request == "PURGE") {
> if (!client.ip ~ purge) {
> error 405 "Not allowed.";
> }
> lookup;
> }
> pipe;
> }
> if (req.http.Expect) {
> pipe;
> }
>
> if (req.http.Authenticate || req.http.Authorization) {
> pass;
> }
>
> # We only care about the "__ac.*" cookies, used for
> authentication
> if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
> pass;
> }
>
> # File type that we will always cache
> if (req.request == "GET" && req.url ~
> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|
> swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|
> sxw|zip|gz|bz2|tgz|tar)$")
> {
> lookup;
> }
>
> if (req.request == "POST") {
> pipe;
> }
>
> # force lookup even when cookies are present
> if (req.request == "GET" && req.http.cookie) {
> lookup;
> }
> lookup;
> }
>
> sub vcl_fetch {
>
> if (obj.ttl < 1800) {
> set obj.ttl = 1800s;
> }
> }
>
> # Do the PURGE thing
> sub vcl_hit {
> if (req.request == "PURGE") {
> set obj.ttl = 0s;
> error 200 "Purged";
> }
> }
>
> sub vcl_miss {
> if (req.request == "PURGE") {
> error 404 "Not in cache";
> }
> }
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev

Adrian,
Thanks for the info...here is the output from my request (ip's
changed)..in case this can shed some light on things:

0 CLI Rd ping
0 CLI Wr 0 200 PONG 1204928472
0 WorkThread 0x66b1d1b0 start
12 SessionOpen c 172.1.1.1 3690
0 WorkThread 0x6531d1b0 end
12 ReqStart c 172.1.1.1 3690 84896454
12 RxRequest c GET
12 RxURL c /index.html
12 RxProtocol c HTTP/1.1
12 RxHeader c Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword, application/x-shockwave-flash, */*
12 RxHeader c Accept-Language: en-us
12 RxHeader c UA-CPU: x86
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
12 RxHeader c Host: host.xxx.com:8080
12 RxHeader c Connection: Keep-Alive
12 RxHeader c Cookie: v1st=9771DC737ED120FD;
GUID=000CD0009ECB070B15D1037761626364
12 VCL_call c recv
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 VCL_call c miss
12 VCL_return c fetch
12 Length c 455
12 VCL_call c deliver
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 503
12 TxResponse c Service Unavailable
12 TxHeader c Server: Varnish
12 TxHeader c Retry-After: 30
12 TxHeader c Content-Type: text/html; charset=utf-8
12 TxHeader c Content-Length: 455
12 TxHeader c Date: Fri, 07 Mar 2008 22:21:14 GMT
12 TxHeader c X-Varnish: 84896454
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 ReqEnd c 84896454 1204928473.923037052 1204928474.937938929
0.004235029 1.014863968 0.000037909
0 StatAddr 172.1.1.1 0 5172 23 249 0 0 11 63867 204568
0 CLI Rd ping
0 CLI Wr 0 200 PONG 1204928475



Shain

Adrian Otto wrote:
> Shain,
>
> I can't say for certain without looking at it myself, but I wonder if
> when you request the URL from the second machine using firefox you are
> actually doing a "refresh" that's setting a Cache-Control header
> that's forcing varnish to consult the origin server. I suggest
> connecting to your varnish with "varnishlog" and watching the headers
> that each client sends varnish. If you see headers like this:
>
> 13 RxHeader c Pragma: no-cache
> 13 RxHeader c Cache-Control: no-cache
>
> Then you know that's what's happening. In general, it sounds like you
> are looking for an "off-line" mode where the cache server simply
> serves stale content if an origin server can not be reached. If that
> feature is present in varnish, I'm not aware of it.
>
> Adrian
>
>
> On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:
>
>> Hello all,
>> First let me say thanks to everyone for taking time to work on this
>> project...I am still in the testing phase, however if all goes well
>> Varnish is going to come in quite handy.
>>
>> So here is my problem:
>>
>> I would like to use varnish to cache urls for say a period of 30
>> minutes. The caching seems fine as long as I use the same browser to
>> request the url. But I am looking for a stop gap solution at this time
>> so in the event that the web server becomes unavailable....varnish can
>> serve the docs for a little while, so we can get things back in order.
>> Anyway..as it stands right now..I can request a url from wget or GET or
>> firefox... and turn off the webserver and I can refresh the page and all
>> is well.
>>
>> What I cannot do...is request a url using GET on one machine..then stop
>> the webserver and request that same url from firefox (different machine)
>> and get a valid page...I get a 503 error...Here is the vcl.conf file
>> (which I took most of from the web) I am using:
>>
>> Any help would be great!
>>
>> Thanks,
>>
>> Shain
>>
>> sub vcl_recv {
>> if (req.request != "GET" && req.request != "HEAD") {
>> # PURGE request if zope asks nicely
>> if (req.request == "PURGE") {
>> if (!client.ip ~ purge) {
>> error 405 "Not allowed.";
>> }
>> lookup;
>> }
>> pipe;
>> }
>> if (req.http.Expect) {
>> pipe;
>> }
>>
>> if (req.http.Authenticate || req.http.Authorization) {
>> pass;
>> }
>>
>> # We only care about the "__ac.*" cookies, used for
>> authentication
>> if (req.http.Cookie && req.http.Cookie ~
>> "__ac(|_(name|password|persistent))=") {
>> pass;
>> }
>>
>> # File type that we will always cache
>> if (req.request == "GET" && req.url ~
>> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar)$")
>>
>> {
>> lookup;
>> }
>>
>> if (req.request == "POST") {
>> pipe;
>> }
>>
>> # force lookup even when cookies are present
>> if (req.request == "GET" && req.http.cookie) {
>> lookup;
>> }
>> lookup;
>> }
>>
>> sub vcl_fetch {
>>
>> if (obj.ttl < 1800) {
>> set obj.ttl = 1800s;
>> }
>> }
>>
>> # Do the PURGE thing
>> sub vcl_hit {
>> if (req.request == "PURGE") {
>> set obj.ttl = 0s;
>> error 200 "Purged";
>> }
>> }
>>
>> sub vcl_miss {
>> if (req.request == "PURGE") {
>> error 404 "Not in cache";
>> }
>> }
>>
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>
>

Adrian,
Thanks for the info...here is the output from my request (ip's
changed)..in case this can shed some light on things:

0 CLI Rd ping
0 CLI Wr 0 200 PONG 1204928472
0 WorkThread 0x66b1d1b0 start
12 SessionOpen c 172.1.1.1 3690
0 WorkThread 0x6531d1b0 end
12 ReqStart c 172.1.1.1 3690 84896454
12 RxRequest c GET
12 RxURL c /index.html
12 RxProtocol c HTTP/1.1
12 RxHeader c Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword, application/x-shockwave-flash, */*
12 RxHeader c Accept-Language: en-us
12 RxHeader c UA-CPU: x86
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
12 RxHeader c Host: host.xxx.com:8080
12 RxHeader c Connection: Keep-Alive
12 RxHeader c Cookie: v1st=9771DC737ED120FD;
GUID=000CD0009ECB070B15D1037761626364
12 VCL_call c recv
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 VCL_call c miss
12 VCL_return c fetch
12 Length c 455
12 VCL_call c deliver
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 503
12 TxResponse c Service Unavailable
12 TxHeader c Server: Varnish
12 TxHeader c Retry-After: 30
12 TxHeader c Content-Type: text/html; charset=utf-8
12 TxHeader c Content-Length: 455
12 TxHeader c Date: Fri, 07 Mar 2008 22:21:14 GMT
12 TxHeader c X-Varnish: 84896454
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 ReqEnd c 84896454 1204928473.923037052 1204928474.937938929
0.004235029 1.014863968 0.000037909
0 StatAddr 172.1.1.1 0 5172 23 249 0 0 11 63867 204568
0 CLI Rd ping
0 CLI Wr 0 200 PONG 1204928475



Shain

Adrian Otto wrote:
> Shain,
>
> I can't say for certain without looking at it myself, but I wonder if
> when you request the URL from the second machine using firefox you are
> actually doing a "refresh" that's setting a Cache-Control header
> that's forcing varnish to consult the origin server. I suggest
> connecting to your varnish with "varnishlog" and watching the headers
> that each client sends varnish. If you see headers like this:
>
> 13 RxHeader c Pragma: no-cache
> 13 RxHeader c Cache-Control: no-cache
>
> Then you know that's what's happening. In general, it sounds like you
> are looking for an "off-line" mode where the cache server simply
> serves stale content if an origin server can not be reached. If that
> feature is present in varnish, I'm not aware of it.
>
> Adrian
>
>
> On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:
>
>> Hello all,
>> First let me say thanks to everyone for taking time to work on this
>> project...I am still in the testing phase, however if all goes well
>> Varnish is going to come in quite handy.
>>
>> So here is my problem:
>>
>> I would like to use varnish to cache urls for say a period of 30
>> minutes. The caching seems fine as long as I use the same browser to
>> request the url. But I am looking for a stop gap solution at this time
>> so in the event that the web server becomes unavailable....varnish can
>> serve the docs for a little while, so we can get things back in order.
>> Anyway..as it stands right now..I can request a url from wget or GET or
>> firefox... and turn off the webserver and I can refresh the page and all
>> is well.
>>
>> What I cannot do...is request a url using GET on one machine..then stop
>> the webserver and request that same url from firefox (different machine)
>> and get a valid page...I get a 503 error...Here is the vcl.conf file
>> (which I took most of from the web) I am using:
>>
>> Any help would be great!
>>
>> Thanks,
>>
>> Shain
>>
>> sub vcl_recv {
>> if (req.request != "GET" && req.request != "HEAD") {
>> # PURGE request if zope asks nicely
>> if (req.request == "PURGE") {
>> if (!client.ip ~ purge) {
>> error 405 "Not allowed.";
>> }
>> lookup;
>> }
>> pipe;
>> }
>> if (req.http.Expect) {
>> pipe;
>> }
>>
>> if (req.http.Authenticate || req.http.Authorization) {
>> pass;
>> }
>>
>> # We only care about the "__ac.*" cookies, used for
>> authentication
>> if (req.http.Cookie && req.http.Cookie ~
>> "__ac(|_(name|password|persistent))=") {
>> pass;
>> }
>>
>> # File type that we will always cache
>> if (req.request == "GET" && req.url ~
>> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar)$")
>>
>> {
>> lookup;
>> }
>>
>> if (req.request == "POST") {
>> pipe;
>> }
>>
>> # force lookup even when cookies are present
>> if (req.request == "GET" && req.http.cookie) {
>> lookup;
>> }
>> lookup;
>> }
>>
>> sub vcl_fetch {
>>
>> if (obj.ttl < 1800) {
>> set obj.ttl = 1800s;
>> }
>> }
>>
>> # Do the PURGE thing
>> sub vcl_hit {
>> if (req.request == "PURGE") {
>> set obj.ttl = 0s;
>> error 200 "Purged";
>> }
>> }
>>
>> sub vcl_miss {
>> if (req.request == "PURGE") {
>> error 404 "Not in cache";
>> }
>> }
>>
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>
>

Shain,

Based on your trace, that browser is getting a "miss" and then can't
fetch the document from the origin server. I did notice that there is
a cookie set, which might be what's causing the miss. Now, you could
get around that with the right VCL file configuration. I'm not
certain if there is a VCL configuration that would cause varnish to
simply disregard the Pragma and Cache-Control headers from the
client, because anyone doing a shift+reload while your origin server
is down is going to get sent back to the origin server as well.

Adrian

Shain,

Based on your trace, that browser is getting a "miss" and then can't
fetch the document from the origin server. I did notice that there is
a cookie set, which might be what's causing the miss. Now, you could
get around that with the right VCL file configuration. I'm not
certain if there is a VCL configuration that would cause varnish to
simply disregard the Pragma and Cache-Control headers from the
client, because anyone doing a shift+reload while your origin server
is down is going to get sent back to the origin server as well.

Adrian

Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to
> simply disregard the Pragma and Cache-Control headers from the
> client, because anyone doing a shift+reload while your origin server
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and
Cache-Control headers on incoming requests. Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
lookup;
}

If the default.vcl I have lying around is still current, then the
default vcl_recv doesn't appear to be looking at Cache-Control or
Pragma, either.

But obviously you probably want to do some checks against
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps
any other request header, before just blindly doing a lookup in the
cache. There are cases, however, where blindly doing a lookup could
make sense - such as when you are in full control of not only all of the
backends, but also all of the clients (e.g. you are putting varnish in
front of a web application that is used by another one of your systems)

Daniel

Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to
> simply disregard the Pragma and Cache-Control headers from the
> client, because anyone doing a shift+reload while your origin server
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and
Cache-Control headers on incoming requests. Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
lookup;
}

If the default.vcl I have lying around is still current, then the
default vcl_recv doesn't appear to be looking at Cache-Control or
Pragma, either.

But obviously you probably want to do some checks against
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps
any other request header, before just blindly doing a lookup in the
cache. There are cases, however, where blindly doing a lookup could
make sense - such as when you are in full control of not only all of the
backends, but also all of the clients (e.g. you are putting varnish in
front of a web application that is used by another one of your systems)

Daniel

In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:

> 12 VCL_call c miss
> 12 VCL_return c fetch

Varnish didn't find a cache-hit and I guess you didn't log the
backend transaction here, so I don't know what the backend told us.

> 12 Length c 455
> 12 VCL_call c deliver
> 12 VCL_return c deliver
> 12 TxProtocol c HTTP/1.1
> 12 TxStatus c 503
> 12 TxResponse c Service Unavailable

It looks a lot like it didn't even get hold of the backend...

Possibly because it couldn't resolve the name of it or because
it didn't have a route to the backend.

It looks like you ran varnislog with a -c argument, try leaving
that out so that the backend transaction also gets logged.


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:

> 12 VCL_call c miss
> 12 VCL_return c fetch

Varnish didn't find a cache-hit and I guess you didn't log the
backend transaction here, so I don't know what the backend told us.

> 12 Length c 455
> 12 VCL_call c deliver
> 12 VCL_return c deliver
> 12 TxProtocol c HTTP/1.1
> 12 TxStatus c 503
> 12 TxResponse c Service Unavailable

It looks a lot like it didn't even get hold of the backend...

Possibly because it couldn't resolve the name of it or because
it didn't have a route to the backend.

It looks like you ran varnislog with a -c argument, try leaving
that out so that the backend transaction also gets logged.


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

I did run varnishlog from the command line without any arguments...it
could not reach the webserver because I turned it off...I was hoping to
setup varnish in such a way that if the webserver fails...varnish will
keep serving pages.

For example: if user A requests index.html and varnish caches it...then
5 minutes later user B requests index.html (and varnish still has it in
the cache) then it should just serve up the file..without ever going to
check the backend. I know that it might not be the number one intended
use for varnish...but that is what I am trying to do...any suggestions?

Thanks in advance,
Shain

Poul-Henning Kamp wrote:
> In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:
>
>
>> 12 VCL_call c miss
>> 12 VCL_return c fetch
>>
>
> Varnish didn't find a cache-hit and I guess you didn't log the
> backend transaction here, so I don't know what the backend told us.
>
>
>> 12 Length c 455
>> 12 VCL_call c deliver
>> 12 VCL_return c deliver
>> 12 TxProtocol c HTTP/1.1
>> 12 TxStatus c 503
>> 12 TxResponse c Service Unavailable
>>
>
> It looks a lot like it didn't even get hold of the backend...
>
> Possibly because it couldn't resolve the name of it or because
> it didn't have a route to the backend.
>
> It looks like you ran varnislog with a -c argument, try leaving
> that out so that the backend transaction also gets logged.
>
>
>

I did run varnishlog from the command line without any arguments...it
could not reach the webserver because I turned it off...I was hoping to
setup varnish in such a way that if the webserver fails...varnish will
keep serving pages.

For example: if user A requests index.html and varnish caches it...then
5 minutes later user B requests index.html (and varnish still has it in
the cache) then it should just serve up the file..without ever going to
check the backend. I know that it might not be the number one intended
use for varnish...but that is what I am trying to do...any suggestions?

Thanks in advance,
Shain

Poul-Henning Kamp wrote:
> In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:
>
>
>> 12 VCL_call c miss
>> 12 VCL_return c fetch
>>
>
> Varnish didn't find a cache-hit and I guess you didn't log the
> backend transaction here, so I don't know what the backend told us.
>
>
>> 12 Length c 455
>> 12 VCL_call c deliver
>> 12 VCL_return c deliver
>> 12 TxProtocol c HTTP/1.1
>> 12 TxStatus c 503
>> 12 TxResponse c Service Unavailable
>>
>
> It looks a lot like it didn't even get hold of the backend...
>
> Possibly because it couldn't resolve the name of it or because
> it didn't have a route to the backend.
>
> It looks like you ran varnislog with a -c argument, try leaving
> that out so that the backend transaction also gets logged.
>
>
>

Thanks alot for the help...I have been trying some of these ideas with little success...even the simple example of 'lookup everything in vcl_recv' did not work...seems to me that should check the cache for everything it receives..which it might have been doing....but it still showed a cache miss..and showed me the 503 error...does anyone know how the matching works...does it only take a hash of the url..or does it use more then that...

Shain


-----Original Message-----
From: varnish-dev-bounces@projects.linpro.no on behalf of Daniel Papasian
Sent: Fri 3/7/2008 5:53 PM
To: Adrian Otto
Cc: varnish-dev at projects.linpro.no
Subject: Re: Caching issue

Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to
> simply disregard the Pragma and Cache-Control headers from the
> client, because anyone doing a shift+reload while your origin server
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and
Cache-Control headers on incoming requests. Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
lookup;
}

If the default.vcl I have lying around is still current, then the
default vcl_recv doesn't appear to be looking at Cache-Control or
Pragma, either.

But obviously you probably want to do some checks against
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps
any other request header, before just blindly doing a lookup in the
cache. There are cases, however, where blindly doing a lookup could
make sense - such as when you are in full control of not only all of the
backends, but also all of the clients (e.g. you are putting varnish in
front of a web application that is used by another one of your systems)

Daniel
_______________________________________________
varnish-dev mailing list
varnish-dev at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-dev/attachments/20080307/b2adfdf3/attachment.htm

Thanks alot for the help...I have been trying some of these ideas with little success...even the simple example of 'lookup everything in vcl_recv' did not work...seems to me that should check the cache for everything it receives..which it might have been doing....but it still showed a cache miss..and showed me the 503 error...does anyone know how the matching works...does it only take a hash of the url..or does it use more then that...

Shain


-----Original Message-----
From: varnish-dev-bounces@projects.linpro.no on behalf of Daniel Papasian
Sent: Fri 3/7/2008 5:53 PM
To: Adrian Otto
Cc: varnish-dev at projects.linpro.no
Subject: Re: Caching issue

Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to
> simply disregard the Pragma and Cache-Control headers from the
> client, because anyone doing a shift+reload while your origin server
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and
Cache-Control headers on incoming requests. Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
lookup;
}

If the default.vcl I have lying around is still current, then the
default vcl_recv doesn't appear to be looking at Cache-Control or
Pragma, either.

But obviously you probably want to do some checks against
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps
any other request header, before just blindly doing a lookup in the
cache. There are cases, however, where blindly doing a lookup could
make sense - such as when you are in full control of not only all of the
backends, but also all of the clients (e.g. you are putting varnish in
front of a web application that is used by another one of your systems)

Daniel
_______________________________________________
varnish-dev mailing list
varnish-dev at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-dev/attachments/20080307/b2adfdf3/attachment-0001.htm

In message <47D1CA5A.5050902 at npr.org>, Shain Miley writes:
>I did run varnishlog from the command line without any arguments...it
>could not reach the webserver because I turned it off...I was hoping to
>setup varnish in such a way that if the webserver fails...varnish will
>keep serving pages.
>
>For example: if user A requests index.html and varnish caches it...then
>5 minutes later user B requests index.html (and varnish still has it in
>the cache) then it should just serve up the file..without ever going to
>check the backend. I know that it might not be the number one intended
>use for varnish...but that is what I am trying to do...any suggestions?

That's certainly possible, but you need to get it set up right.

By default Varnish will respect whatever the backend says about
caching, and in this case you want to override that.

I don't know if your backend said "do not cache this" or "cache only
15 seconds", but whatever it is, you need to override it in VCL.


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <47D1CA5A.5050902 at npr.org>, Shain Miley writes:
>I did run varnishlog from the command line without any arguments...it
>could not reach the webserver because I turned it off...I was hoping to
>setup varnish in such a way that if the webserver fails...varnish will
>keep serving pages.
>
>For example: if user A requests index.html and varnish caches it...then
>5 minutes later user B requests index.html (and varnish still has it in
>the cache) then it should just serve up the file..without ever going to
>check the backend. I know that it might not be the number one intended
>use for varnish...but that is what I am trying to do...any suggestions?

That's certainly possible, but you need to get it set up right.

By default Varnish will respect whatever the backend says about
caching, and in this case you want to override that.

I don't know if your backend said "do not cache this" or "cache only
15 seconds", but whatever it is, you need to override it in VCL.


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Shain Miley <smiley at npr.org> writes:
> What I cannot do...is request a url using GET on one machine..then stop
> the webserver and request that same url from firefox (different machine)
> and get a valid page...I get a 503 error...

Most likely because Firefox has a cookie that triggers the following
code:

> # We only care about the "__ac.*" cookies, used for authentication
> if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
> pass;
> }

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Shain Miley <smiley at npr.org> writes:
> What I cannot do...is request a url using GET on one machine..then stop
> the webserver and request that same url from firefox (different machine)
> and get a valid page...I get a 503 error...

Most likely because Firefox has a cookie that triggers the following
code:

> # We only care about the "__ac.*" cookies, used for authentication
> if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
> pass;
> }

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Adrian Otto <aotto at mosso.com> writes:
> I can't say for certain without looking at it myself, but I wonder if
> when you request the URL from the second machine using firefox you
> are actually doing a "refresh" that's setting a Cache-Control header
> that's forcing varnish to consult the origin server.

For the millionth time, Varnish *intentionally* *does* *not* *obey*
these headers.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Adrian Otto <aotto at mosso.com> writes:
> I can't say for certain without looking at it myself, but I wonder if
> when you request the URL from the second machine using firefox you
> are actually doing a "refresh" that's setting a Cache-Control header
> that's forcing varnish to consult the origin server.

For the millionth time, Varnish *intentionally* *does* *not* *obey*
these headers.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Shain Miley <smiley at npr.org> writes:
> Thanks for the info...here is the output from my request (ip's
> changed)..in case this can shed some light on things:

This is useless if you don't also show us a request that worked.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Shain Miley <smiley at npr.org> writes:
> Thanks for the info...here is the output from my request (ip's
> changed)..in case this can shed some light on things:

This is useless if you don't also show us a request that worked.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Fair enough..here you go:

I used Firefox and IE from the same machine:

First the cache hit:

12 SessionOpen c 172.1.1.1 3203
12 ReqStart c 172.1.1.1 3203 42332617
12 RxRequest c GET
12 RxURL c /templates/topics/topic.php?topicId=1006
12 RxProtocol c HTTP/1.1
12 RxHeader c Host: server11.npr.org:8080
12 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
12 RxHeader c Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=$
12 RxHeader c Accept-Language: en-us,en;q=0.5
12 RxHeader c Accept-Encoding: gzip,deflate
12 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
12 RxHeader c Keep-Alive: 300
12 RxHeader c Connection: keep-alive
12 RxHeader c Cookie: v1st=7ED84E9096FDA5B; GUID=0003D141E48D07D140F1C40561626364
12 RxHeader c If-None-Match: "jpd--899294354.37421"
12 VCL_call c recv
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 Hit c 42332555
12 VCL_call c hit
12 VCL_return c deliver
12 Length c 9137
12 VCL_call c deliver
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 200
12 TxResponse c OK
12 TxHeader c Server: Apache
12 TxHeader c X-Powered-By: PHP/5.2.3
12 TxHeader c X-Cache: jpcache vv2 - npr-burn
12 TxHeader c ETag: "jpd--899294354.37421"
12 TxHeader c Cache-Control: max-age=0
12 TxHeader c Expires: Sat, 08 Mar 2008 22:41:22 GMT
12 TxHeader c Vary: Accept-Encoding
12 TxHeader c Content-Encoding: gzip
12 TxHeader c Content-Encoding: gzip
12 TxHeader c Content-Type: text/html
12 TxHeader c Content-Length: 9137
12 TxHeader c Date: Sat, 08 Mar 2008 22:43:05 GMT
12 TxHeader c X-Varnish: 42332617 42332555
12 TxHeader c Age: 102
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 ReqEnd c 42332617 1205016185.289316893 1205016185.289417028 0.006116867 0.000038147 0.000061989


Now the cache miss:

12 SessionOpen c 172.1.1.1 3211
12 ReqStart c 172.1.1.1 3211 42332632
12 RxRequest c GET
12 RxURL c /templates/topics/topic.php?topicId=1006
12 RxProtocol c HTTP/1.1
12 RxHeader c Accept: */*
12 RxHeader c Accept-Language: en-us
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
12 RxHeader c Host: server11.npr.org:8080
12 RxHeader c Connection: Keep-Alive
12 RxHeader c Cookie: v1st=2E524136463554A; GUID=000748BD59C807AC295102B561626364; LE4=+5SoM6V421+314+4
12 VCL_call c recv
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 VCL_call c miss
12 VCL_return c fetch
12 Length c 455
12 VCL_call c deliver
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 503
12 TxResponse c Service Unavailable
12 TxHeader c Server: Varnish
12 TxHeader c Retry-After: 30
12 TxHeader c Content-Type: text/html; charset=utf-8
12 TxHeader c Content-Length: 455
12 TxHeader c Date: Sat, 08 Mar 2008 22:43:15 GMT
12 TxHeader c X-Varnish: 42332632
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 ReqEnd c 42332632 1205016194.493165016 1205016195.508080959 0.005739927 1.014875889 0.000040054


And here is the current vcl.conf:

#
# This is a basic VCL configuration file for varnish. See the vcl(7)
# man page for details on VCL syntax and semantics.
#
# $Id: default.vcl 1929 2007-08-29 15:37:59Z des $
#

# Default backend definition. Set this to point to your content
# server.

backend default {
set backend.host = "172.31.2.61"; // use your own backend ip address
set backend.port = "80"; // use your own backend port
}

sub vcl_recv {

# Remove the "Cookie:" header from the request.
remove req.http.Set-Cookie;
remove req.http.Cache-Control;


if (req.request == "GET" && req.url ~ "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc)$") {
lookup;
}

if (req.http.Cache-Control ~ "no-cache") {
lookup;
}
if (req.http.Cache-Control == "max-age=0") {
lookup;
}
# Do a "lookup" in the cache. This goes to "vcl_hit", or to
# "vcl_miss" and then "vcl_fetch"
lookup;
}

# Do the PURGE thing
sub vcl_hit {
}
sub vcl_miss {
}

sub vcl_fetch {
if (obj.ttl < 7200s) {
set obj.ttl = 7200s;
}
insert;

if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "max-age=0") {
insert;
}

if (obj.ttl < 3600s) {
set obj.ttl = 3600s;
}
insert;
}

sub vcl_hash {

set req.hash += req.url;
hash;
}


Thanks,

Shain

-----Original Message-----
From: Dag-Erling Sm?rgrav [mailto:des@linpro.no]
Sent: Sat 3/8/2008 10:58 AM
To: Shain Miley
Cc: Adrian Otto; varnish-dev at projects.linpro.no
Subject: Re: Caching issue

Shain Miley <smiley at npr.org> writes:
> Thanks for the info...here is the output from my request (ip's
> changed)..in case this can shed some light on things:

This is useless if you don't also show us a request that worked.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-dev/attachments/20080308/32b20cfc/attachment.htm