Mailing List Archive

Varnish 6.0.11 panic help
Hi Guys

I am really struggling to debug this panic and wondered if you could give any hints were I might start looking for answers. I have been trawling the GitHub repo and have seen several people recommend upping the thread_pool_stack value. Here is the panic and a few other bits of data that I think are useful based on the issues I have been reading:

varnishd[15482]: Child (15492) Panic at: Tue, 22 Nov 2022 16:10:26 GMT
Wrong turn at cache/cache_main.c:284:
Signal 11 (Segmentation fault) received at 0xffff887fe000 si_code 2
version = varnish-6.0.11 revision a3bc025c2df28e4a76e10c2c41217c9864e9963b, vrt api = 7.1
ident = Linux,4.14.290-217.505.amzn2.aarch64,aarch64,-junix,-smalloc,-sfile,-sdefault,-hcritbit,epoll
now = 3120063.050288 (mono), 1669133426.954528 (real)
Backtrace:
0x43f5b4: /usr/sbin/varnishd() [0x43f5b4]
0x4a968c: /usr/sbin/varnishd(VAS_Fail+0x54) [0x4a968c]
0x43a350: /usr/sbin/varnishd() [0x43a350]
0xffff90fd7668: linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff90fd7668]
0xffff90c8f540: /lib64/libc.so.6(memset+0x100) [0xffff90c8f540]
0x4b88f4: /usr/sbin/varnishd(deflateReset+0x48) [0x4b88f4]
0x42e924: /usr/sbin/varnishd(VGZ_NewGzip+0x88) [0x42e924]
0x42ebc0: /usr/sbin/varnishd() [0x42ebc0]
0x42df28: /usr/sbin/varnishd(VFP_Open+0x98) [0x42df28]
0x42b950: /usr/sbin/varnishd() [0x42b950]
thread = (cache-worker)
thr.req = (nil) {
},
thr.busyobj = 0xffff3d040020 {
end = 0xffff3d050000,
retries = 0,
sp = 0xffff3c241a20 {
fd = 28, vxid = 32945,
t_open = 1669133426.953970,
t_idle = 1669133426.953970,
ws = 0xffff3c241a60 {
id = \"ses\",
{s, f, r, e} = {0xffff3c241aa0, +96, (nil), +344},
},
transport = HTTP/1 {
state = HTTP1::Proc
}
client = 172.31.47.149 50812 :80,
},
worker = 0xffff832326c8 {
ws = 0xffff83232770 {
id = \"wrk\",
{s, f, r, e} = {0xffff83231e00, +0, (nil), +2040},
},
VCL::method = BACKEND_RESPONSE,
VCL::return = deliver,
VCL::methods = {BACKEND_FETCH, BACKEND_RESPONSE},
},
vfc = 0xffff3d041f30 {
failed = 0,
req = 0xffff3d040640,
resp = 0xffff3d040ab8,
wrk = 0xffff832326c8,
oc = 0xffff3b250640,
filters = {
gzip = 0xffff3d04a740 {
priv1 = (nil),
priv2 = 0,
closed = 0
},
V1F_STRAIGHT = 0xffff3d04a660 {
priv1 = 0xffff3d042600,
priv2 = 674132,
closed = 0
},
},
obj_flags = 0x0,
},
ws = 0xffff3d040058 {
id = \"bo\",
{s, f, r, e} = {0xffff3d041f78, +34832, (nil), +57472},
},
ws_bo = 0xffff3d0425e8,
http[bereq] = 0xffff3d040640 {
ws = 0xffff3d040058 {
[Already dumped, see above]
},
hdrs {
\"GET\",
\"/build/admin/css/oro.css?v=7c08a284\",
\"HTTP/1.1\",
\"X-Forwarded-Proto: https\",
\"X-Forwarded-Port: 443\",
\"Host: london.paperstage.doverstreetmarket.com <http://london.paperstage.doverstreetmarket.com/>\",
\"X-Amzn-Trace-Id: Root=1-637cf472-4faa5da7246eda4f0a477811\",
\"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36\",
\"X-Amz-Cf-Id: WVvRZ7m_mF9MaQ88b0rJd_vEKVJo5YqQr6Povp9ODasHVw2FQop36w==\",
\"Via: 3.0 c1685d59e35fdb859ab8a1f97feb5652.cloudfront.net <http://c1685d59e35fdb859ab8a1f97feb5652.cloudfront.net/> (CloudFront)\",
\"Cookie: BAPID=mdmh5umu6v6l1qejvo0lbhse54; __utma=58316727.170235450.1647004079.1669126271.1669131591.29; __utmb=58316727.22.10.1669131591; __utmc=58316727; __utmt=1; __utmt_b=1; __utmz=58316727.1665659005.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _csrf=5Inqz08yS1ACLDnJDFG8xoavis88G4BgHx4WpD6RuQM; _ga=GA1.2.170235450.1647004079; _ga_QNDMHZ1SBD=GS1.1.1667387720.14.1.1667387720.60.0.0; _gcl_au=1.1.255577917.1662482672; _landing_page=%2F; _orig_referrer=; _shopify_y=1531218a-8c0c-4153-86e1-37e06cc3a103; _y=1531218a-8c0c-4153-86e1-37e06cc3a103; df_preview=%7B%22preview_id%22%3A%221755%22%2C%22preview_type%22%3A%22version%22%2C%22preview_code%22%3A%22d82194c2b3c4f22164ca71d225673ff6%22%7D\",
\"Accept-Language: en-US,en;q=0.9,fr;q=0.8\",
\"Accept: text/css,*/*;q=0.1\",
\"Referer: https://london.paperstage.doverstreetmarket.com/doverstreetadmin/pages/settings/1755/en\ <https://london.paperstage.doverstreetmarket.com/doverstreetadmin/pages/settings/1755/en/>",
\"Accept-Encoding: gzip, deflate, br\",
\"pragma: no-cache\",
\"cache-control: no-cache\",
\"sec-gpc: 1\",
\"sec-fetch-site: same-origin\",
\"sec-fetch-mode: no-cors\",
\"sec-fetch-dest: style\",
\"X-Forwarded-For: 32.217.249.1, 15.158.35.113, 172.31.47.149\",
\"X-Varnish: 32947\",
},
},
http[beresp] = 0xffff3d040ab8 {
ws = 0xffff3d040058 {
[Already dumped, see above]
},
hdrs {
\"HTTP/1.1\",
\"200\",
\"OK\",
\"Server: nginx/1.20.0\",
\"Date: Tue, 22 Nov 2022 16:10:26 GMT\",
\"Content-Type: text/css\",
\"Content-Length: 674132\",
\"Last-Modified: Tue, 22 Nov 2022 15:28:13 GMT\",
\"Connection: keep-alive\",
\"Expires: Wed, 22 Nov 2023 16:10:26 GMT\",
\"Cache-Control: max-age=31536000, public, no-transform\",
\"Accept-Ranges: bytes\",
\"X-Url: /build/admin/css/oro.css?v=7c08a284\",
\"X-Host: london.paperstage.doverstreetmarket.com <http://london.paperstage.doverstreetmarket.com/>\",
},
},
objcore[fetch] = 0xffff3b250640 {
refcnt = 2,
flags = {busy, hfm, private},
exp_flags = {},
boc = 0xffff3b260160 {
refcnt = 2,
state = req_done,
vary = (nil),
stevedore_priv = (nil),
},
exp = {1669133426.954405, -1.000000, 300.000000, 0.000000},
objhead = 0xffff907d01d0,
stevedore = (nil),
},
http_conn = 0xffff3d042600 {
fd = 30 (@0xffff3be704e4),
doclose = NULL,
ws = 0xffff3d040058 {
[Already dumped, see above]
},
{rxbuf_b, rxbuf_e} = {0xffff3d042660, 0xffff3d0427a8},
{pipeline_b, pipeline_e} = {0xffff3d0427a8, 0xffff3d04a660},
content_length = 674132,
body_status = length,
first_byte_timeout = 180.000000,
between_bytes_timeout = 180.000000,
},
flags = {do_gzip, do_stream, do_pass, uncacheable},
director_req = 0xffff8ffc0190 {
vcl_name = default,
health = healthy,
admin_health = probe, changed = 1669131098.893551,
type = backend {
display_name = boot.default,
ipv4 = 127.0.0.1,
port = 8080,
hosthdr = 127.0.0.1,
n_conn = 1,
},
},
director_resp = director_req,
vcl = {
name = \"boot\",
busy = 5,
discard = 0,
state = auto,
temp = warm,
conf = {
syntax = \"40\",
srcname = {
\"/etc/varnish/default.vcl\",
\"Builtin\",
},
},
},
},
vmods = {
std = {Varnish 6.0.11 a3bc025c2df28e4a76e10c2c41217c9864e9963b, 0.0},
},
param.show thread_pool_stack
200
thread_pool_stack
Value is: 128k [bytes] (default)
Minimum is: 128k


yum info jemalloc
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
223 packages excluded due to repository priority protections
Installed Packages
Name : jemalloc
Arch : aarch64
Version : 3.6.0
Release : 1.el7
Size : 374 k
Repo : installed
From repo : epel
Summary : General-purpose scalable concurrent malloc implementation
URL : http://www.canonware.com/jemalloc/ <http://www.canonware.com/jemalloc/>
License : BSD
Description : General-purpose scalable concurrent malloc(3) implementation.
: This distribution is the stand-alone "portable" implementation of jemalloc.


Any insight you can offer would be greatly appreciated. I am going to try setting the thread_pool_stack to 1024k today, however this error is pretty intermittent so I may not know if it has resolved anything for a day or so.

Kind Regards
Luke Rotherfield
Senior Developer

https://function.london <https://function.london/>
Re: Varnish 6.0.11 panic help [ In reply to ]
On Tue, Nov 22, 2022 at 6:03 PM Luke Rotherfield
<luke.rotherfield@function.london> wrote:
>
> Hi Guys
>
> I am really struggling to debug this panic and wondered if you could give any hints were I might start looking for answers. I have been trawling the GitHub repo and have seen several people recommend upping the thread_pool_stack value. Here is the panic and a few other bits of data that I think are useful based on the issues I have been reading:
>
> varnishd[15482]: Child (15492) Panic at: Tue, 22 Nov 2022 16:10:26 GMT
>
> Wrong turn at cache/cache_main.c:284:
> Signal 11 (Segmentation fault) received at 0xffff887fe000 si_code 2
> version = varnish-6.0.11 revision a3bc025c2df28e4a76e10c2c41217c9864e9963b, vrt api = 7.1
> ident = Linux,4.14.290-217.505.amzn2.aarch64,aarch64,-junix,-smalloc,-sfile,-sdefault,-hcritbit,epoll
> now = 3120063.050288 (mono), 1669133426.954528 (real)
> Backtrace:
> 0x43f5b4: /usr/sbin/varnishd() [0x43f5b4]
> 0x4a968c: /usr/sbin/varnishd(VAS_Fail+0x54) [0x4a968c]
> 0x43a350: /usr/sbin/varnishd() [0x43a350]
> 0xffff90fd7668: linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff90fd7668]
> 0xffff90c8f540: /lib64/libc.so.6(memset+0x100) [0xffff90c8f540]
> 0x4b88f4: /usr/sbin/varnishd(deflateReset+0x48) [0x4b88f4]
> 0x42e924: /usr/sbin/varnishd(VGZ_NewGzip+0x88) [0x42e924]
> 0x42ebc0: /usr/sbin/varnishd() [0x42ebc0]
> 0x42df28: /usr/sbin/varnishd(VFP_Open+0x98) [0x42df28]
> 0x42b950: /usr/sbin/varnishd() [0x42b950]
> thread = (cache-worker)
> thr.req = (nil) {
> },
> thr.busyobj = 0xffff3d040020 {
> end = 0xffff3d050000,
> retries = 0,
> sp = 0xffff3c241a20 {
> fd = 28, vxid = 32945,
> t_open = 1669133426.953970,
> t_idle = 1669133426.953970,
> ws = 0xffff3c241a60 {
> id = \"ses\",
> {s, f, r, e} = {0xffff3c241aa0, +96, (nil), +344},
> },
> transport = HTTP/1 {
> state = HTTP1::Proc
> }
> client = 172.31.47.149 50812 :80,
> },
> worker = 0xffff832326c8 {
> ws = 0xffff83232770 {
> id = \"wrk\",
> {s, f, r, e} = {0xffff83231e00, +0, (nil), +2040},
> },
> VCL::method = BACKEND_RESPONSE,
> VCL::return = deliver,
> VCL::methods = {BACKEND_FETCH, BACKEND_RESPONSE},
> },
> vfc = 0xffff3d041f30 {
> failed = 0,
> req = 0xffff3d040640,
> resp = 0xffff3d040ab8,
> wrk = 0xffff832326c8,
> oc = 0xffff3b250640,
> filters = {
> gzip = 0xffff3d04a740 {
> priv1 = (nil),
> priv2 = 0,
> closed = 0
> },
> V1F_STRAIGHT = 0xffff3d04a660 {
> priv1 = 0xffff3d042600,
> priv2 = 674132,
> closed = 0
> },
> },
> obj_flags = 0x0,
> },
> ws = 0xffff3d040058 {
> id = \"bo\",
> {s, f, r, e} = {0xffff3d041f78, +34832, (nil), +57472},
> },
> ws_bo = 0xffff3d0425e8,
> http[bereq] = 0xffff3d040640 {
> ws = 0xffff3d040058 {
> [Already dumped, see above]
> },
> hdrs {
> \"GET\",
> \"/build/admin/css/oro.css?v=7c08a284\",
> \"HTTP/1.1\",
> \"X-Forwarded-Proto: https\",
> \"X-Forwarded-Port: 443\",
> \"Host: london.paperstage.doverstreetmarket.com\",
> \"X-Amzn-Trace-Id: Root=1-637cf472-4faa5da7246eda4f0a477811\",
> \"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36\",
> \"X-Amz-Cf-Id: WVvRZ7m_mF9MaQ88b0rJd_vEKVJo5YqQr6Povp9ODasHVw2FQop36w==\",
> \"Via: 3.0 c1685d59e35fdb859ab8a1f97feb5652.cloudfront.net (CloudFront)\",
> \"Cookie: BAPID=mdmh5umu6v6l1qejvo0lbhse54; __utma=58316727.170235450.1647004079.1669126271.1669131591.29; __utmb=58316727.22.10.1669131591; __utmc=58316727; __utmt=1; __utmt_b=1; __utmz=58316727.1665659005.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _csrf=5Inqz08yS1ACLDnJDFG8xoavis88G4BgHx4WpD6RuQM; _ga=GA1.2.170235450.1647004079; _ga_QNDMHZ1SBD=GS1.1.1667387720.14.1.1667387720.60.0.0; _gcl_au=1.1.255577917.1662482672; _landing_page=%2F; _orig_referrer=; _shopify_y=1531218a-8c0c-4153-86e1-37e06cc3a103; _y=1531218a-8c0c-4153-86e1-37e06cc3a103; df_preview=%7B%22preview_id%22%3A%221755%22%2C%22preview_type%22%3A%22version%22%2C%22preview_code%22%3A%22d82194c2b3c4f22164ca71d225673ff6%22%7D\",
> \"Accept-Language: en-US,en;q=0.9,fr;q=0.8\",
> \"Accept: text/css,*/*;q=0.1\",
> \"Referer: https://london.paperstage.doverstreetmarket.com/doverstreetadmin/pages/settings/1755/en\",
> \"Accept-Encoding: gzip, deflate, br\",
> \"pragma: no-cache\",
> \"cache-control: no-cache\",
> \"sec-gpc: 1\",
> \"sec-fetch-site: same-origin\",
> \"sec-fetch-mode: no-cors\",
> \"sec-fetch-dest: style\",
> \"X-Forwarded-For: 32.217.249.1, 15.158.35.113, 172.31.47.149\",
> \"X-Varnish: 32947\",
> },
> },
> http[beresp] = 0xffff3d040ab8 {
> ws = 0xffff3d040058 {
> [Already dumped, see above]
> },
> hdrs {
> \"HTTP/1.1\",
> \"200\",
> \"OK\",
> \"Server: nginx/1.20.0\",
> \"Date: Tue, 22 Nov 2022 16:10:26 GMT\",
> \"Content-Type: text/css\",
> \"Content-Length: 674132\",
> \"Last-Modified: Tue, 22 Nov 2022 15:28:13 GMT\",
> \"Connection: keep-alive\",
> \"Expires: Wed, 22 Nov 2023 16:10:26 GMT\",
> \"Cache-Control: max-age=31536000, public, no-transform\",
> \"Accept-Ranges: bytes\",
> \"X-Url: /build/admin/css/oro.css?v=7c08a284\",
> \"X-Host: london.paperstage.doverstreetmarket.com\",
> },
> },
> objcore[fetch] = 0xffff3b250640 {
> refcnt = 2,
> flags = {busy, hfm, private},
> exp_flags = {},
> boc = 0xffff3b260160 {
> refcnt = 2,
> state = req_done,
> vary = (nil),
> stevedore_priv = (nil),
> },
> exp = {1669133426.954405, -1.000000, 300.000000, 0.000000},
> objhead = 0xffff907d01d0,
> stevedore = (nil),
> },
> http_conn = 0xffff3d042600 {
> fd = 30 (@0xffff3be704e4),
> doclose = NULL,
> ws = 0xffff3d040058 {
> [Already dumped, see above]
> },
> {rxbuf_b, rxbuf_e} = {0xffff3d042660, 0xffff3d0427a8},
> {pipeline_b, pipeline_e} = {0xffff3d0427a8, 0xffff3d04a660},
> content_length = 674132,
> body_status = length,
> first_byte_timeout = 180.000000,
> between_bytes_timeout = 180.000000,
> },
> flags = {do_gzip, do_stream, do_pass, uncacheable},
> director_req = 0xffff8ffc0190 {
> vcl_name = default,
> health = healthy,
> admin_health = probe, changed = 1669131098.893551,
> type = backend {
> display_name = boot.default,
> ipv4 = 127.0.0.1,
> port = 8080,
> hosthdr = 127.0.0.1,
> n_conn = 1,
> },
> },
> director_resp = director_req,
> vcl = {
> name = \"boot\",
> busy = 5,
> discard = 0,
> state = auto,
> temp = warm,
> conf = {
> syntax = \"40\",
> srcname = {
> \"/etc/varnish/default.vcl\",
> \"Builtin\",
> },
> },
> },
> },
> vmods = {
> std = {Varnish 6.0.11 a3bc025c2df28e4a76e10c2c41217c9864e9963b, 0.0},
> },
>
> param.show thread_pool_stack
> 200
> thread_pool_stack
> Value is: 128k [bytes] (default)
> Minimum is: 128k
>
>
> yum info jemalloc
> Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
> 223 packages excluded due to repository priority protections
> Installed Packages
> Name : jemalloc
> Arch : aarch64
> Version : 3.6.0
> Release : 1.el7
> Size : 374 k
> Repo : installed
> From repo : epel
> Summary : General-purpose scalable concurrent malloc implementation
> URL : http://www.canonware.com/jemalloc/
> License : BSD
> Description : General-purpose scalable concurrent malloc(3) implementation.
> : This distribution is the stand-alone "portable" implementation of jemalloc.
>
>
> Any insight you can offer would be greatly appreciated. I am going to try setting the thread_pool_stack to 1024k today, however this error is pretty intermittent so I may not know if it has resolved anything for a day or so.

Hi Luke,

At first glance this does not look like something you could work
around with a larger thread stack.

If you have a github account, we can move the discussion there:

https://github.com/varnishcache/varnish-cache/issues/3867

Best,
Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc