Mailing List Archive

Converting 3-to-5 - strange "No backend" errors
Hi folks. I'm in the process of migrating to from v3 to v5, and have
finally got the config compiling at least. Now I'm having some problems
with the backends that I can't seem to work out. Apologies, this is a bit
of a wordy one...

Let's say I have two backends, web01 and web02, and these are put into a
director called admin_director for load-balancing. This is all in
"backends.vcl":

import directors;

probe healthcheck {
.request =
"GET / HTTP/1.1"
"Host: www.mydomain.co.uk"
"Connection: close";
.timeout = 30s;
.interval = 15s;
.window = 10;
.threshold = 8;
.expected_response = 302;
}


backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout =
600s; .probe = healthcheck; }
backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout =
600s; .probe = healthcheck; }

sub vcl_init {
new admin_director = directors.shard();
admin_director.add_backend(web01);
admin_director.add_backend(web02);
}


I've now removed all my main VCL for clarification, so my vcl_recv.vcl file
contains just:


import std;

sub vcl_recv {
set req.backend_hint = admin_director.backend();
std.log("Backend hint: " + req.backend_hint);
}



I then have my main "default.vcl":


vcl 4.0;
include "backends.vcl";
include "vcl_recv.vcl";


Now to the problem. If I try to access a URL, I always get a "FetchError:
No backend"

* << BeReq >> 3
- Begin bereq 2 fetch
- Timestamp Start: 1490802880.486355 0.000000 0.000000
- BereqMethod GET
- BereqURL /Common/Images/NewLoginImages/training.png
- BereqProtocol HTTP/1.1
- BereqHeader Host: www.mydomain.com
- BereqHeader User-Agent: curl/7.35.0
- BereqHeader Accept: */*
- BereqHeader X-Forwarded-Proto: https
- BereqHeader X-Real-Ip: 1.2.3.4
- BereqHeader Accept-Encoding: gzip
- BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
- BereqHeader X-Varnish: 3
- VCL_call BACKEND_FETCH
- VCL_return fetch
- FetchError No backend
- Timestamp Beresp: 1490802880.486447 0.000092 0.000092
- Timestamp Error: 1490802880.486452 0.000098 0.000006
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 503
- ObjReason Backend fetch failed
- ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
- ObjHeader Server: Varnish
- ObjHeader Content-Type: text/html; charset=utf-8
- ObjHeader Retry-After: 5
- Length 278
- BereqAcct 0 0 0 0 0 0
- End

* << Request >> 32770
- Begin req 32769 rxreq
- Timestamp Start: 1490803780.494500 0.000000 0.000000
- Timestamp Req: 1490803780.494500 0.000000 0.000000
- ReqStart 192.168.1.23 36896
- ReqMethod GET
- ReqURL /Common/Images/NewLoginImages/training.png
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.mydomain.com
- ReqHeader User-Agent: curl/7.35.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 1.2.3.4
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Real-Ip: 1.2.3.4
- ReqHeader Accept-Encoding: gzip
- ReqUnset X-Forwarded-For: 1.2.3.4
- ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
- VCL_call RECV
- Error shard admin_director: no backends
- VCL_Log Backend hint:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 32771 fetch
- Timestamp Fetch: 1490803780.495705 0.001204 0.001204
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT
- RespHeader Server: Varnish
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 32770
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.1)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1490803780.495723 0.001223 0.000019
- RespHeader Content-Length: 282
- Debug "RES_MODE 2"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1490803780.495806 0.001305 0.000082
- ReqAcct 238 0 238 250 282 532
- End

But my backend.list seems to always be healthy:

boot.web01 probe Healthy 10/10
Wed, 29 Mar 2017 15:54:18 GMT
boot.web02 probe Healthy 10/10
Wed, 29 Mar 2017 15:54:18 GMT

From the varnish server, I can reach the backends successfully manually:

curl --resolve www.mydomain.com:192.168.1.11
http://www.mydomain.com/Common/Images/NewLoginImages/training.png

Not sure what it all means.
Re: Converting 3-to-5 - strange "No backend" errors [ In reply to ]
I think you forgot to commit the backend addtions.

--
Guillaume Quintard

On Wed, Mar 29, 2017 at 6:12 PM, Mark Hanford <mark@hanfordonline.co.uk>
wrote:

> Hi folks. I'm in the process of migrating to from v3 to v5, and have
> finally got the config compiling at least. Now I'm having some problems
> with the backends that I can't seem to work out. Apologies, this is a bit
> of a wordy one...
>
> Let's say I have two backends, web01 and web02, and these are put into a
> director called admin_director for load-balancing. This is all in
> "backends.vcl":
>
> import directors;
>
> probe healthcheck {
> .request =
> "GET / HTTP/1.1"
> "Host: www.mydomain.co.uk"
> "Connection: close";
> .timeout = 30s;
> .interval = 15s;
> .window = 10;
> .threshold = 8;
> .expected_response = 302;
> }
>
>
> backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout
> = 600s; .probe = healthcheck; }
> backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout
> = 600s; .probe = healthcheck; }
>
> sub vcl_init {
> new admin_director = directors.shard();
> admin_director.add_backend(web01);
> admin_director.add_backend(web02);
> }
>
>
> I've now removed all my main VCL for clarification, so my vcl_recv.vcl
> file contains just:
>
>
> import std;
>
> sub vcl_recv {
> set req.backend_hint = admin_director.backend();
> std.log("Backend hint: " + req.backend_hint);
> }
>
>
>
> I then have my main "default.vcl":
>
>
> vcl 4.0;
> include "backends.vcl";
> include "vcl_recv.vcl";
>
>
> Now to the problem. If I try to access a URL, I always get a "FetchError:
> No backend"
>
> * << BeReq >> 3
> - Begin bereq 2 fetch
> - Timestamp Start: 1490802880.486355 0.000000 0.000000
> - BereqMethod GET
> - BereqURL /Common/Images/NewLoginImages/training.png
> - BereqProtocol HTTP/1.1
> - BereqHeader Host: www.mydomain.com
> - BereqHeader User-Agent: curl/7.35.0
> - BereqHeader Accept: */*
> - BereqHeader X-Forwarded-Proto: https
> - BereqHeader X-Real-Ip: 1.2.3.4
> - BereqHeader Accept-Encoding: gzip
> - BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
> - BereqHeader X-Varnish: 3
> - VCL_call BACKEND_FETCH
> - VCL_return fetch
> - FetchError No backend
> - Timestamp Beresp: 1490802880.486447 0.000092 0.000092
> - Timestamp Error: 1490802880.486452 0.000098 0.000006
> - BerespProtocol HTTP/1.1
> - BerespStatus 503
> - BerespReason Service Unavailable
> - BerespReason Backend fetch failed
> - BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
> - BerespHeader Server: Varnish
> - VCL_call BACKEND_ERROR
> - BerespHeader Content-Type: text/html; charset=utf-8
> - BerespHeader Retry-After: 5
> - VCL_return deliver
> - Storage malloc Transient
> - ObjProtocol HTTP/1.1
> - ObjStatus 503
> - ObjReason Backend fetch failed
> - ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
> - ObjHeader Server: Varnish
> - ObjHeader Content-Type: text/html; charset=utf-8
> - ObjHeader Retry-After: 5
> - Length 278
> - BereqAcct 0 0 0 0 0 0
> - End
>
> * << Request >> 32770
> - Begin req 32769 rxreq
> - Timestamp Start: 1490803780.494500 0.000000 0.000000
> - Timestamp Req: 1490803780.494500 0.000000 0.000000
> - ReqStart 192.168.1.23 36896
> - ReqMethod GET
> - ReqURL /Common/Images/NewLoginImages/training.png
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: www.mydomain.com
> - ReqHeader User-Agent: curl/7.35.0
> - ReqHeader Accept: */*
> - ReqHeader X-Forwarded-For: 1.2.3.4
> - ReqHeader X-Forwarded-Proto: https
> - ReqHeader X-Real-Ip: 1.2.3.4
> - ReqHeader Accept-Encoding: gzip
> - ReqUnset X-Forwarded-For: 1.2.3.4
> - ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
> - VCL_call RECV
> - Error shard admin_director: no backends
> - VCL_Log Backend hint:
> - VCL_return hash
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call MISS
> - VCL_return fetch
> - Link bereq 32771 fetch
> - Timestamp Fetch: 1490803780.495705 0.001204 0.001204
> - RespProtocol HTTP/1.1
> - RespStatus 503
> - RespReason Backend fetch failed
> - RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT
> - RespHeader Server: Varnish
> - RespHeader Content-Type: text/html; charset=utf-8
> - RespHeader Retry-After: 5
> - RespHeader X-Varnish: 32770
> - RespHeader Age: 0
> - RespHeader Via: 1.1 varnish (Varnish/5.1)
> - VCL_call DELIVER
> - VCL_return deliver
> - Timestamp Process: 1490803780.495723 0.001223 0.000019
> - RespHeader Content-Length: 282
> - Debug "RES_MODE 2"
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1490803780.495806 0.001305 0.000082
> - ReqAcct 238 0 238 250 282 532
> - End
>
> But my backend.list seems to always be healthy:
>
> boot.web01 probe Healthy 10/10
> Wed, 29 Mar 2017 15:54:18 GMT
> boot.web02 probe Healthy 10/10
> Wed, 29 Mar 2017 15:54:18 GMT
>
> From the varnish server, I can reach the backends successfully manually:
>
> curl --resolve www.mydomain.com:192.168.1.11 http://www.mydomain.com/
> Common/Images/NewLoginImages/training.png
>
> Not sure what it all means.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Converting 3-to-5 - strange "No backend" errors [ In reply to ]
Aha, yes that was it. I literally just spotted the fact that the shard
director needs a ".reconfigure()" after making changes.

Thanks, that seems to have fixed it.

Mark

On 30 March 2017 at 13:43, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> I think you forgot to commit the backend addtions.
>
> --
> Guillaume Quintard
>
> On Wed, Mar 29, 2017 at 6:12 PM, Mark Hanford <mark@hanfordonline.co.uk>
> wrote:
>
>> Hi folks. I'm in the process of migrating to from v3 to v5, and have
>> finally got the config compiling at least. Now I'm having some problems
>> with the backends that I can't seem to work out. Apologies, this is a bit
>> of a wordy one...
>>
>> Let's say I have two backends, web01 and web02, and these are put into a
>> director called admin_director for load-balancing. This is all in
>> "backends.vcl":
>>
>> import directors;
>>
>> probe healthcheck {
>> .request =
>> "GET / HTTP/1.1"
>> "Host: www.mydomain.co.uk"
>> "Connection: close";
>> .timeout = 30s;
>> .interval = 15s;
>> .window = 10;
>> .threshold = 8;
>> .expected_response = 302;
>> }
>>
>>
>> backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout
>> = 600s; .probe = healthcheck; }
>> backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout
>> = 600s; .probe = healthcheck; }
>>
>> sub vcl_init {
>> new admin_director = directors.shard();
>> admin_director.add_backend(web01);
>> admin_director.add_backend(web02);
>> }
>>
>>
>> I've now removed all my main VCL for clarification, so my vcl_recv.vcl
>> file contains just:
>>
>>
>> import std;
>>
>> sub vcl_recv {
>> set req.backend_hint = admin_director.backend();
>> std.log("Backend hint: " + req.backend_hint);
>> }
>>
>>
>>
>> I then have my main "default.vcl":
>>
>>
>> vcl 4.0;
>> include "backends.vcl";
>> include "vcl_recv.vcl";
>>
>>
>> Now to the problem. If I try to access a URL, I always get a "FetchError:
>> No backend"
>>
>> * << BeReq >> 3
>> - Begin bereq 2 fetch
>> - Timestamp Start: 1490802880.486355 0.000000 0.000000
>> - BereqMethod GET
>> - BereqURL /Common/Images/NewLoginImages/training.png
>> - BereqProtocol HTTP/1.1
>> - BereqHeader Host: www.mydomain.com
>> - BereqHeader User-Agent: curl/7.35.0
>> - BereqHeader Accept: */*
>> - BereqHeader X-Forwarded-Proto: https
>> - BereqHeader X-Real-Ip: 1.2.3.4
>> - BereqHeader Accept-Encoding: gzip
>> - BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
>> - BereqHeader X-Varnish: 3
>> - VCL_call BACKEND_FETCH
>> - VCL_return fetch
>> - FetchError No backend
>> - Timestamp Beresp: 1490802880.486447 0.000092 0.000092
>> - Timestamp Error: 1490802880.486452 0.000098 0.000006
>> - BerespProtocol HTTP/1.1
>> - BerespStatus 503
>> - BerespReason Service Unavailable
>> - BerespReason Backend fetch failed
>> - BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
>> - BerespHeader Server: Varnish
>> - VCL_call BACKEND_ERROR
>> - BerespHeader Content-Type: text/html; charset=utf-8
>> - BerespHeader Retry-After: 5
>> - VCL_return deliver
>> - Storage malloc Transient
>> - ObjProtocol HTTP/1.1
>> - ObjStatus 503
>> - ObjReason Backend fetch failed
>> - ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT
>> - ObjHeader Server: Varnish
>> - ObjHeader Content-Type: text/html; charset=utf-8
>> - ObjHeader Retry-After: 5
>> - Length 278
>> - BereqAcct 0 0 0 0 0 0
>> - End
>>
>> * << Request >> 32770
>> - Begin req 32769 rxreq
>> - Timestamp Start: 1490803780.494500 0.000000 0.000000
>> - Timestamp Req: 1490803780.494500 0.000000 0.000000
>> - ReqStart 192.168.1.23 36896
>> - ReqMethod GET
>> - ReqURL /Common/Images/NewLoginImages/training.png
>> - ReqProtocol HTTP/1.1
>> - ReqHeader Host: www.mydomain.com
>> - ReqHeader User-Agent: curl/7.35.0
>> - ReqHeader Accept: */*
>> - ReqHeader X-Forwarded-For: 1.2.3.4
>> - ReqHeader X-Forwarded-Proto: https
>> - ReqHeader X-Real-Ip: 1.2.3.4
>> - ReqHeader Accept-Encoding: gzip
>> - ReqUnset X-Forwarded-For: 1.2.3.4
>> - ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23
>> - VCL_call RECV
>> - Error shard admin_director: no backends
>> - VCL_Log Backend hint:
>> - VCL_return hash
>> - VCL_call HASH
>> - VCL_return lookup
>> - VCL_call MISS
>> - VCL_return fetch
>> - Link bereq 32771 fetch
>> - Timestamp Fetch: 1490803780.495705 0.001204 0.001204
>> - RespProtocol HTTP/1.1
>> - RespStatus 503
>> - RespReason Backend fetch failed
>> - RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT
>> - RespHeader Server: Varnish
>> - RespHeader Content-Type: text/html; charset=utf-8
>> - RespHeader Retry-After: 5
>> - RespHeader X-Varnish: 32770
>> - RespHeader Age: 0
>> - RespHeader Via: 1.1 varnish (Varnish/5.1)
>> - VCL_call DELIVER
>> - VCL_return deliver
>> - Timestamp Process: 1490803780.495723 0.001223 0.000019
>> - RespHeader Content-Length: 282
>> - Debug "RES_MODE 2"
>> - RespHeader Connection: keep-alive
>> - Timestamp Resp: 1490803780.495806 0.001305 0.000082
>> - ReqAcct 238 0 238 250 282 532
>> - End
>>
>> But my backend.list seems to always be healthy:
>>
>> boot.web01 probe Healthy 10/10
>> Wed, 29 Mar 2017 15:54:18 GMT
>> boot.web02 probe Healthy 10/10
>> Wed, 29 Mar 2017 15:54:18 GMT
>>
>> From the varnish server, I can reach the backends successfully manually:
>>
>> curl --resolve www.mydomain.com:192.168.1.11
>> http://www.mydomain.com/Common/Images/NewLoginImages/training.png
>>
>> Not sure what it all means.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>