Mailing List Archive

503 Backend fetch failed / (again)
Hi,

We were previously receiving Error 503 in Varnish and then it has fixed
after the fresh reinstall. But we are getting the same error again. I have
observed that this appears when Ajax requests increase.


*http://.com/wp-admin/admin-ajax.php Failed to load resource: the server
responded with a status of 503 (Backend fetch failed)*


Is there an error in Default.vcl file?

/* SET THE HOST AND PORT OF WORDPRESS
* *********************************************************/
vcl 4.0;
import std;

backend default {
.host = "******";
.port = "8080";
}

# SET THE ALLOWED IP OF PURGE REQUESTS
# ##########################################################
acl purge {
"localhost";
"127.0.0.1";
"******";
}

#THE RECV FUNCTION
# ##########################################################
sub vcl_recv {
if (req.http.Host == "***.com/forum" || req.url ~ "forum") {
return (pass);
}

# set realIP by trimming CloudFlare IP which will be used for various checks
set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", "");

# FORWARD THE IP OF THE REQUEST
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}

# Purge request check sections for hash_always_miss, purge and ban
# BLOCK IF NOT IP is not in purge acl
# ##########################################################

# Enable smart refreshing using hash_always_miss
if (req.http.Cache-Control ~ "no-cache") {
if (client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") ~
purge) {
set req.hash_always_miss = true;
}
}

if (req.method == "PURGE") {
if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") ~
purge) {
return(synth(405,"Not allowed."));
}
return (purge);

}
if (req.method == "BAN") {
# Same ACL check as above:
if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4")
~ purge) {
return(synth(403, "Not allowed."));
}
ban("req.http.host == " + req.http.host +
" && req.url == " + req.url);

# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}


# Unset cloudflare cookies
# Remove has_js and CloudFlare/Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie,
"(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
# Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

# For Testing: If you want to test with Varnish passing (not caching)
uncomment
# return( pass );


# DO NOT CACHE RSS FEED
if (req.url ~ "/feed(/)?") {
return ( pass );
}


## Do not cache search results, comment these 3 lines if you do want to
cache them

if (req.url ~ "/\?s\=") {
return ( pass );
}

# CLEAN UP THE ENCODING HEADER.
# SET TO GZIP, DEFLATE, OR REMOVE ENTIRELY. WITH VARY ACCEPT-ENCODING
# VARNISH WILL CREATE SEPARATE CACHES FOR EACH
# DO NOT ACCEPT-ENCODING IMAGES, ZIPPED FILES, AUDIO, ETC.
# ##########################################################
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
unset req.http.Accept-Encoding;
}
}

# PIPE ALL NON-STANDARD REQUESTS
# ##########################################################
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
return (pipe);
}

# ONLY CACHE GET AND HEAD REQUESTS
# ##########################################################
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}

# OPTIONAL: DO NOT CACHE LOGGED IN USERS (THIS OCCURS IN FETCH TOO, EITHER
# COMMENT OR UNCOMMENT BOTH
# ##########################################################
if ( req.http.cookie ~ "wordpress_logged_in" ) {
return( pass );
}




# IF THE REQUEST IS NOT FOR A PREVIEW, WP-ADMIN OR WP-LOGIN
# THEN UNSET THE COOKIES
# ##########################################################
if (!(req.url ~ "wp-(login|admin)")
&& !(req.url ~ "&preview=true" )
){
unset req.http.cookie;
}

# IF BASIC AUTH IS ON THEN DO NOT CACHE
# ##########################################################
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}

# IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED
# ##########################################################
return (hash);
# This is for phpmyadmin
if (req.http.Host == "ki1.org") {
return (pass);
}

if (req.http.Host == "mysql.ki1.org") {
return (pass);
}

}

# HIT FUNCTION
# ##########################################################
sub vcl_hit {
# IF THIS IS A PURGE REQUEST THEN DO THE PURGE
# ##########################################################
if (req.method == "PURGE") {
#
# This is now handled in vcl_recv.
#
# purge;
return (synth(200, "Purged."));
}
return (deliver);
}

# MISS FUNCTION
# ##########################################################
sub vcl_miss {
if (req.method == "PURGE") {
#
# This is now handled in vcl_recv.
#
# purge;
return (synth(200, "Purged."));
}
return (fetch);
}

# FETCH FUNCTION
# ##########################################################
sub vcl_backend_response {
# I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC
# TENDANCY TO SET VARY USER-AGENT. YOU MAY OR MAY NOT WANT
# TO DO THIS
# ##########################################################
set beresp.http.Vary = "Accept-Encoding";

# IF NOT WP-ADMIN THEN UNSET COOKIES AND SET THE AMOUNT OF
# TIME THIS PAGE WILL STAY CACHED (TTL)
# ##########################################################
if (!(bereq.url ~ "wp-(login|admin)|forum") && !bereq.http.cookie ~
"wordpress_logged_in" && !bereq.http.host == "***.com/forum" ) {
unset beresp.http.set-cookie;
set beresp.ttl = 52w;
# set beresp.grace =1w;
}

if (beresp.ttl <= 0s ||
beresp.http.Set-Cookie ||
beresp.http.Vary == "*") {
set beresp.ttl = 120 s;
# set beresp.ttl = 120s;
set beresp.uncacheable = true;
return (deliver);
}

return (deliver);
}

# DELIVER FUNCTION
# ##########################################################
sub vcl_deliver {
# IF THIS PAGE IS ALREADY CACHED THEN RETURN A 'HIT' TEXT
# IN THE HEADER (GREAT FOR DEBUGGING)
# ##########################################################
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
# IF THIS IS A MISS RETURN THAT IN THE HEADER
# ##########################################################
} else {
set resp.http.X-Cache = "MISS";
}
}
Re: 503 Backend fetch failed / (again) [ In reply to ]
On Wed, Nov 16, 2016 at 12:37:08PM +0300, Ayberk Kimsesiz wrote:
>
> We were previously receiving Error 503 in Varnish and then it has fixed
> after the fresh reinstall. But we are getting the same error again. I have
> observed that this appears when Ajax requests increase.

It's hard to troubleshoot anything without a varnishlog.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: 503 Backend fetch failed / (again) [ In reply to ]
Hi,

Varnishlog: *FetchError http first read error: EOF*

* << BeReq >> 3244175
- Begin bereq 3244174 pass
- Timestamp Start: 1479720397.535252 0.000000 0.000000
- BereqMethod GET
- BereqURL
/forum/js/brivium/CustomNodeStyle/custom_node.js?_v=00bdfbe8
- BereqProtocol HTTP/1.1
- BereqHeader Host:******.com
- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1003.1 Safari/535.19
Awesomium/1.7.1
- BereqHeader Accept: */*
- BereqHeader Referer: https://facebook.com/
- BereqHeader Accept-Encoding: gzip,deflate
- BereqHeader Accept-Language: en-us,en
- BereqHeader Accept-Charset: iso-8859-1,*,utf-8
- BereqHeader Cookie: _gat=1;
pps_show_100=Tue%20Nov%2022%202016%2012%3A31%3A52%20GMT+0300%20%28Turkey%20Standard%20Time%29;
pps_times_showed_100=2; _ga=GA1.2.1950747215.1479806983;
xf_session=20756c36c3041eb1b77c10d43b913cd8
- BereqHeader X-Forwarded-For: 176.240.98.52
- BereqHeader X-Varnish: 3244175
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 160 boot.default 176.53.126.10 8080 176.53.126.10 47362
- BackendStart 176.53.126.10 8080
- Timestamp Bereq: 1479720397.535321 0.000070 0.000070
*- FetchError http first read error: EOF*
- BackendClose 160 boot.default
- Timestamp Beresp: 1479720457.535409 60.000158 60.000088
- Timestamp Error: 1479720457.535417 60.000165 0.000008
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Mon, 21 Nov 2016 09:27:37 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 503
- ObjReason Backend fetch failed
- ObjHeader Date: Mon, 21 Nov 2016 09:27:37 GMT
- ObjHeader Server: Varnish
- ObjHeader Content-Type: text/html; charset=utf-8
- ObjHeader Retry-After: 5
- Length 284
- BereqAcct 643 0 643 0 0 0
- End


2016-11-16 12:56 GMT+03:00 Andreas Plesner <apj@mutt.dk>:

> On Wed, Nov 16, 2016 at 12:37:08PM +0300, Ayberk Kimsesiz wrote:
> >
> > We were previously receiving Error 503 in Varnish and then it has fixed
> > after the fresh reinstall. But we are getting the same error again. I
> have
> > observed that this appears when Ajax requests increase.
>
> It's hard to troubleshoot anything without a varnishlog.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: 503 Backend fetch failed / (again) [ In reply to ]
On 11/21/2016 10:34 AM, Ayberk Kimsesiz wrote:
>
> Varnishlog: *FetchError http first read error: EOF*

That's a first byte timeout.

> *- FetchError http first read error: EOF* - BackendClose
> 160 boot.default - Timestamp Beresp: 1479720457.535409
> 60.000158 60.000088 - Timestamp Error: 1479720457.535417
> 60.000165 0.000008

The Timestamp is telling you (2nd and 3rd numbers after
Timestamp:Beresp) that your backend "boot.default" didn't send a
response for 60 seconds before Varnish gave up, presumably because
first_byte_timeout=60s.

man vsl(7) tells you how to interpret the Timestamp log entries:

https://www.varnish-cache.org/docs/trunk/reference/vsl.html#timestamps

https://www.varnish-cache.org/docs/trunk/reference/vsl.html#backend-fetch-timestamps

So you need to fix your backend.


HTH,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
Re: 503 Backend fetch failed / (again) [ In reply to ]
Hi Geoff,

Thank you for your suggestion. My current settings:

backend default {
.host = "IP";
.port = "8080";
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
.max_connections = 800;

}

The log screen: http://i.imgur.com/RixRgM3.png


And the *http first read error: EOF* continues.

Thanks.


​


2016-11-21 13:53 GMT+03:00 Geoff Simmons <geoff@uplex.de>:

> On 11/21/2016 10:34 AM, Ayberk Kimsesiz wrote:
> >
> > Varnishlog: *FetchError http first read error: EOF*
>
> That's a first byte timeout.
>
> > *- FetchError http first read error: EOF* - BackendClose
> > 160 boot.default - Timestamp Beresp: 1479720457.535409
> > 60.000158 60.000088 - Timestamp Error: 1479720457.535417
> > 60.000165 0.000008
>
> The Timestamp is telling you (2nd and 3rd numbers after
> Timestamp:Beresp) that your backend "boot.default" didn't send a
> response for 60 seconds before Varnish gave up, presumably because
> first_byte_timeout=60s.
>
> man vsl(7) tells you how to interpret the Timestamp log entries:
>
> https://www.varnish-cache.org/docs/trunk/reference/vsl.html#timestamps
>
> https://www.varnish-cache.org/docs/trunk/reference/vsl.html#
> backend-fetch-timestamps
>
> So you need to fix your backend.
>
>
> HTH,
> Geoff
> --
> ** * * UPLEX - Nils Goroll Systemoptimierung
>
> Scheffelstraße 32
> 22301 Hamburg
>
> Tel +49 40 2880 5731
> Mob +49 176 636 90917
> Fax +49 40 42949753
>
> http://uplex.de
>
>