Mailing List Archive

Varnish Issue
Hello everyone,
today i had a strange panic error on my Varnish.
It is not really obvious to me what went wrong, any help would be much
appreciated.
I'm running varnish 6.0.3


Panic at: Tue, 19 Nov 2019 19:28:02 GMT
Assert error in vbf_stp_error(), cache/cache_fetch.c line
800:Condition((VSB_finish(synth_body)) == 0) not true.
version = varnish-6.0.3 revision 7d1ded3aa033a018317dbafc61587026ea2ef8a3,
vrt api = 7.0
ident =
Linux,3.10.0-693.11.6.el7.x86_64,x86_64,-junix,-smalloc,-sdefault,-hcritbit,epoll
now = 17076364.384963 (mono), 1574191668.252235 (real)
Backtrace:
errno = 12 (Cannot allocate memory)
thread = (cache-worker)
thr.req = (nil) {
},
thr.busyobj = 0x7fada9c7e020 {
end = 0x7fada9c8e000,
retries = 0,
sp = 0x7fad00501620 {
fd = 13433, vxid = 694059016,
t_open = 1574191668.121636,
t_idle = 1574191668.121636,
ws = 0x7fad00501660 {
id = \"ses\",
{s, f, r, e} = {0x7fad00501698, +56, (nil), +352},
},
transport = HTTP/1 {
state = HTTP1::Proc
}
client = 0.0.0.0 0 /dev/shm/varnish.sock,
},
worker = 0x7fac63522bd0 {
ws = 0x7fac63522c78 {
id = \"wrk\",
{s, f, r, e} = {0x7fac63522380, +0, (nil), +2040},
},
VCL::method = BACKEND_ERROR,
VCL::return = fail,
VCL::methods = {BACKEND_FETCH, BACKEND_RESPONSE, BACKEND_ERROR},
},
vfc = 0x7fada9c7ff10 {
failed = 1,
req = 0x7fada9c7e620,
resp = 0x7fada9c7ea98,
wrk = 0x7fac63522bd0,
oc = 0x7fac9bd5f140,
filters = {
V1F_STRAIGHT = 0x7fada9c858c0 {
priv1 = 0x7fada9c808c0,
priv2 = 19999,
closed = 0
},
},
obj_flags = 0x0,
},
ws = 0x7fada9c7e058 {
id = \"bo\",
{s, f, r, e} = {0x7fada9c7ff58, +23096, (nil), +57504},
},
ws_bo = 0x7fada9c808a8,
http[bereq] = 0x7fada9c7e620 {
ws = 0x7fada9c7e058 {
[Already dumped, see above]
},
hdrs {
\"GET\",
\"/openreqs?_=1574191667646\",
\"HTTP/1.1\",
\"Host: www.xxx.yy\",
\"X-Real-IP: 176.58.172.113\",
\"X-Forwarded-Proto: https\",
\"Accept-Encoding: gzip\",
\"CF-IPCountry: GB\",
\"accept: application/json, text/javascript, /; q=0.01\",
\"dnt: 1\",
\"x-requested-with: XMLHttpRequest\",
\"user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\",
\"sec-fetch-site: same-origin\",
\"sec-fetch-mode: cors\",
\"referer: https://www.xxx.yy/\",
\"accept-language:
en-GB,en;q=0.9,q=0.8,el;q=0.7,en-US;q=0.6,de;q=0.5\",
\"CF-Connecting-IP: 176.58.172.113\",
\"CDN-Loop:MF\",
\"X-Forwarded-For: 176.58.172.113, 176.58.172.113, 0.0.0.0\",
\"sticky: 9.385\",
\"cookie: __uid=d69a4ebccdf450dca2f56195270fb54101561187829;
_landing=1;sticky=stx9.385; _tz=120; _tz_intl=Europe%2FLondon; ",
\"X-defHash: /openreqs?_=1574191667646 + www.xxx.yy\",
\"X-Varnish: 694059018\",
},
},
http[beresp] = 0x7fada9c7ea98 {
ws = 0x7fada9c7e058 {
[Already dumped, see above]
},
hdrs {
\"HTTP/1.1\",
\"503\",
\"Backend fetch failed\",
\"Date: Tue, 19 Nov 2019 19:27:48 GMT\",
\"Server: Varnish\",
\"Content-Type: text/html; charset=utf-8\",
\"Retry-After: 5\",
},
},
objcore[fetch] = 0x7fac9bd5f140 {
refcnt = 2,
flags = {busy, hfm, private},
exp_flags = {},
boc = 0x7fac9bd61ce0 {
refcnt = 2,
state = req_done,
vary = (nil),
stevedore_priv = (nil),
},
exp = {1574191668.252177, 0.000000, 0.000000, 0.000000},
objhead = 0x7faeea0ce160,
stevedore = (nil),
},
flags = {do_stream, do_pass, uncacheable},
director_req = 0x7faeea0e1310 {
vcl_name = web01,
health = healthy,
admin_health = probe, changed = 1574176692.243255,
type = backend {
display_name = boot.web01,
ipv4 = xxx.xxx.xxx.xxx,
port = 80,
hosthdr = xxx.xxx.xxx.xxx,
n_conn = 24,
},
},
director_resp = director_req,
vcl = {
name = \"boot\",
busy = 46459,
discard = 0,
state = auto,
temp = warm,
conf = {
syntax = \"41\",
srcname = {
\"/etc/varnish/default.vcl\",
\"Builtin\",
\"/etc/varnish/origins1.vcl\",
\"/etc/varnish/origins-2.vcl\",
\"/etc/varnish/origins-3.vcl\",
\"/etc/varnish/origins-others.vcl\",
\"/etc/varnish/origins-4.vcl\",
\"/etc/varnish/origins5.vcl\",
\"/etc/varnish/ban.vcl\",
\"/etc/varnish/backend_error.vcl\",
},
},
},
},
vmods = {
std = {Varnish 6.0.3 7d1ded3aa033a018317dbafc61587026ea2ef8a3, 0.0},
directors = {Varnish 6.0.3 7d1ded3aa033a018317dbafc61587026ea2ef8a3, 0.0},
cookie = {Varnish 6.0.3 7d1ded3aa033a018317dbafc61587026ea2ef8a3, 7.0},
header = {Varnish 6.0.3 7d1ded3aa033a018317dbafc61587026ea2ef8a3, 7.0},
},


Could you please take a look and advise?


Thank you in advance

Regards
Re: Varnish Issue [ In reply to ]
On 19/11/2019 22:21, Nikos Theodosiadis wrote:
> Assert error in vbf_stp_error(), cache/cache_fetch.c line
> 800:Condition((VSB_finish(synth_body)) == 0) not true.
> version = varnish-6.0.3 revision 7d1ded3aa033a018317dbafc61587026ea2ef8a3, vrt
> api = 7.0
> ident =
> Linux,3.10.0-693.11.6.el7.x86_64,x86_64,-junix,-smalloc,-sdefault,-hcritbit,epoll
> now = 17076364.384963 (mono), 1574191668.252235 (real)
> Backtrace:
> errno = 12 (Cannot allocate memory)

This was likely an out-of-memory condition. Varnish should generally be
configured such that there is always some (100s of MB) memory available on the
system.