#1670: assert: default_oc_getobj(), storage/stevedore.c line 68
----------------------+----------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: unknown
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------
Description changed by lkarsten:
Old description:
> Posting this on behalf of mattrobenolt. Originally reported on IRC.
>
> {{{
> Last panic at: Sun, 01 Feb 2015 03:40:29 GMT
> Assert error in default_oc_getobj(), storage/stevedore.c line 68:
> Condition(((o))->magic == (0x32851d42)) not true.
> thread = (cache-worker)
> version = varnish-4.0.3-rc2 revision 1b96340
> ident = Linux,3.13.0-43-generic,x86_64,-smalloc,-smalloc,-hcritbit,epoll
> Backtrace:
> 0x433d8a: /usr/sbin/varnishd() [0x433d8a]
> 0x45853f: /usr/sbin/varnishd() [0x45853f]
> 0x41ef24: /usr/sbin/varnishd(EXP_NukeOne+0x194) [0x41ef24]
> 0x459148: /usr/sbin/varnishd(STV_alloc+0xe8) [0x459148]
> 0x422b0e: /usr/sbin/varnishd(VFP_GetStorage+0x7e) [0x422b0e]
> 0x420941: /usr/sbin/varnishd() [0x420941]
> 0x436ca1: /usr/sbin/varnishd(Pool_Work_Thread+0x381) [0x436ca1]
> 0x449c58: /usr/sbin/varnishd() [0x449c58]
> 0x7fbb6d59be9a: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)
> [0x7fbb6d59be9a]
> 0x7fbb6d2c92ed: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
> [0x7fbb6d2c92ed]
> busyobj = 0x7fb5403bd020 {
> ws = 0x7fb5403bd0e0 {
> id = "bo",
> {s,f,r,e} = {0x7fb5403bf008,+2184,(nil),+57368},
> },
> [.. cut ..]
> }}}
>
> backtrace is redacted by me, I'll add the full panic dump when/if I get
> permission to make it public.
>
> According to the report, this happened after running for 90 minutes on
> live traffic with ~10kreq/s.
>
> Martin spent Monday looking at this, but did not find anything
> conclusive.
New description:
Posting this on behalf of mattrobenolt. Originally reported on IRC.
{{{
Last panic at: Sun, 01 Feb 2015 03:40:29 GMT
Assert error in default_oc_getobj(), storage/stevedore.c line 68:
Condition(((o))->magic == (0x32851d42)) not true.
thread = (cache-worker)
version = varnish-4.0.3-rc2 revision 1b96340
ident = Linux,3.13.0-43-generic,x86_64,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
0x433d8a: /usr/sbin/varnishd() [0x433d8a]
0x45853f: /usr/sbin/varnishd() [0x45853f]
0x41ef24: /usr/sbin/varnishd(EXP_NukeOne+0x194) [0x41ef24]
0x459148: /usr/sbin/varnishd(STV_alloc+0xe8) [0x459148]
0x422b0e: /usr/sbin/varnishd(VFP_GetStorage+0x7e) [0x422b0e]
0x420941: /usr/sbin/varnishd() [0x420941]
0x436ca1: /usr/sbin/varnishd(Pool_Work_Thread+0x381) [0x436ca1]
0x449c58: /usr/sbin/varnishd() [0x449c58]
0x7fbb6d59be9a: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)
[0x7fbb6d59be9a]
0x7fbb6d2c92ed: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
[0x7fbb6d2c92ed]
busyobj = 0x7fb5403bd020 {
ws = 0x7fb5403bd0e0 {
id = "bo",
{s,f,r,e} = {0x7fb5403bf008,+2184,(nil),+57368},
},
refcnt = 1
retries = 0
failed = 0
state = 2
is_do_stream
bodystatus = 0 (none),
},
http[bereq] = {
ws = 0x7fb5403bd0e0[bo]
"GET",
"/embed/comments/?base=default&f=worldstar&s_o=default&t_d=&t_e=Whitney%20Houston%E2%80%99s%20Daughter%20Found%20Unresponsive%20In%20Bathtub!&t_i=77244&t_t=Whitney%20Houston%E2%80%99s%20Daughter%20Found%20Unresponsive%20In%20Bathtub!&t_u=http%3A%2F%2Fwww.worldstarhiphop.com%2Fvideos%2Fvideo.php%3Fv%3DwshhgQrkMl5yaZeeeHN9&version=ff15479433461993d0738de53d5f22cf",
"HTTP/1.1",
"Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; SAMSUNG-SM-G900A
Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0
Chrome/30.0.0.0 Mobile Safari/537.36",
"Referer:
http://worldstar.disqus.com/", "Accept-Language: en-US",
"X-Requested-With: com.pt.wshhp",
"X-Forwarded-For: 66.117.245.64",
"X-Forwarded-Proto: http",
"Host: disqus.com",
"Disqus-Root: 1",
"Accept-Encoding: gzip",
"If-Modified-Since: Sun, 01 Feb 2015 03:40:11 GMT",
"X-Varnish: 161797784",
},
http[beresp] = {
ws = 0x7fb5403bd0e0[bo]
"HTTP/1.1",
"200",
"OK",
"Server: nginx",
"Date: Sun, 01 Feb 2015 03:40:16 GMT",
"Content-Type: text/html; charset=utf-8",
"Vary: Accept-Encoding",
"Content-Security-Policy: script-src https://*.twitter.com:*
https://api.adsnative.com/v1/ad.json *.adsafeprotected.com *.google-
analytics.com
https://glitter-services.disqus.com https://*.services.disqus.com:* disqus.com http://*.twitter.com:*
a.disquscdn.com api.taboola.com referrer.disqus.com
*.scorecardresearch.com *.moatads.com
https://admin.appnext.com/offerWallApi.aspx 'unsafe-eval'
https://mobile.adnxs.com/mob *.services.disqus.com:*",
"Surrogate-Control: max-age=5",
"Last-Modified: Sun, 01 Feb 2015 03:40:11 GMT",
"Cache-Control: s-stalewhilerevalidate=3600, stale-while-
revalidate=30, no-cache, must-revalidate, public, s-maxage=5",
"p3p: CP="DSP IDC CUR ADM DELi STP NAV COM UNI INT PHY DEM"",
"Timing-Allow-Origin: *",
"X-Content-Type-Options: nosniff",
"X-XSS-Protection: 1; mode=block",
"Content-Encoding: gzip",
"Surrogate-Grace: 3600s",
"Grace: 30s",
"Disqus-Cachetype: CACHE",
"X-Served-By: app-155.dal01, shield-1.dal01",
"X-Cache: MISS, HIT",
"X-Cache-Hits: 0, 3",
"Connection: keep-alive",
"Content-Length: 11241",
"X-Backend: embed, shield1",
},
ws = 0x7fb5403bd270 {
id = "obj",
{s,f,r,e} = {0x7fba550912b0,+1224,(nil),+1224},
},
objcore (FETCH) = 0x7fb98a715c00 {
refcnt = 2
flags = 0x0
objhead = 0x7fbaed8352e0
}
obj (FETCH) = 0x7fba55091000 {
vxid = 2309281432,
http[obj] = {
ws = (nil)[]
"HTTP/1.1",
"200",
"OK",
"Server: nginx",
"Date: Sun, 01 Feb 2015 03:40:16 GMT",
"Content-Type: text/html; charset=utf-8",
"Vary: Accept-Encoding",
"Content-Security-Policy: script-src https://*.twitter.com:*
https://api.adsnative.com/v1/ad.json *.adsafeprotected.com *.google-
analytics.com
https://glitter-services.disqus.com https://*.services.disqus.com:* disqus.com http://*.twitter.com:*
a.disquscdn.com api.taboola.com referrer.disqus.com
*.scorecardresearch.com *.moatads.com
https://admin.appnext.com/offerWallApi.aspx 'unsafe-eval'
https://mobile.adnxs.com/mob *.services.disqus.com:*",
"Surrogate-Control: max-age=5",
"Last-Modified: Sun, 01 Feb 2015 03:40:11 GMT",
"Cache-Control: s-stalewhilerevalidate=3600, stale-while-
revalidate=30, no-cache, must-revalidate, public, s-maxage=5",
"p3p: CP="DSP IDC CUR ADM DELi STP NAV COM UNI INT PHY DEM"",
"Timing-Allow-Origin: *",
"X-Content-Type-Options: nosniff",
"X-XSS-Protection: 1; mode=block",
"Content-Encoding: gzip",
"Surrogate-Grace: 3600s",
"Grace: 30s",
"Disqus-Cachetype: CACHE",
"X-Served-By: app-155.dal01, shield-1.dal01",
"X-Cache: MISS, HIT",
"X-Cache-Hits: 0, 3",
"Content-Length: 11241",
"X-Backend: embed, shield1",
},
len = 4917,
store = {
2810 {
1f 8b 08 00 00 00 00 00 00 03 ed 5d 6b 77 db c6 |...........]kw..|
b5 fd ee 5f 31 61 6e 9b b5 ee 15 49 bc f8 00 2b |..._1an....I...+|
39 95 6d d9 52 62 cb a9 45 c7 4d e3 2c 2d 90 18 |9.m.Rb..E.M.,-..|
92 b0 40 80 c5 43 0c 93 95 ff 7e f7 19 3c 48 81 |..@..C....~..<H.|
[2746 more]
},
2107 {
88 6d 64 0f 9a 04 b7 a7 2b fc e5 23 ef 79 91 59 |.md.....+..#.y.Y|
0b 5f fa 03 33 df 4d 9b 9d 2e fc eb 62 51 0e 4d |._..3.M.....bQ.M|
9b ff f8 d2 77 3e 58 6e f7 f6 1f c4 0f b1 86 8b |....w>Xn........|
97 31 6d 7e 70 47 dc 0b d8 1b 38 d8 27 38 c3 2e |.1m~pG....8.'8..|
[2043 more]
},
},
},
obj (IMS) = 0x7fb620d27800 {
vxid = 2331935377,
http[obj] = {
ws = (nil)[]
"HTTP/1.1",
"200",
"OK",
"Server: nginx",
"Date: Sun, 01 Feb 2015 03:40:16 GMT",
"Content-Type: text/html; charset=utf-8",
"Vary: Accept-Encoding",
"Content-Security-Policy: script-src https://*.twitter.com:*
https://api.adsnative.com/v1/ad.json *.adsafeprotected.com *.google-
analytics.com
https://glitter-services.disqus.com https://*.services.disqus.com:* disqus.com http://*.twitter.com:*
a.disquscdn.com api.taboola.com referrer.disqus.com
*.scorecardresearch.com *.moatads.com
https://admin.appnext.com/offerWallApi.aspx 'unsafe-eval'
https://mobile.adnxs.com/mob *.services.disqus.com:*",
"Surrogate-Control: max-age=5",
"Last-Modified: Sun, 01 Feb 2015 03:40:11 GMT",
"Cache-Control: s-stalewhilerevalidate=3600, stale-while-
revalidate=30, no-cache, must-revalidate, public, s-maxage=5",
"p3p: CP="DSP IDC CUR ADM DELi STP NAV COM UNI INT PHY DEM"",
"Timing-Allow-Origin: *",
"X-Content-Type-Options: nosniff",
"X-XSS-Protection: 1; mode=block",
"Content-Encoding: gzip",
"Surrogate-Grace: 3600s",
"Grace: 30s",
"Disqus-Cachetype: CACHE",
"X-Served-By: app-155.dal01, shield-1.dal01",
"X-Cache: MISS, HIT",
"X-Cache-Hits: 0, 1",
"Content-Length: 11241",
"X-Backend: embed, shield1",
},
len = 11241,
store = {
11241 {
1f 8b 08 00 00 00 00 00 00 03 ed 5d 6b 77 db c6 |...........]kw..|
b5 fd ee 5f 31 61 6e 9b b5 ee 15 49 bc f8 00 2b |..._1an....I...+|
39 95 6d d9 52 62 cb a9 45 c7 4d e3 2c 2d 90 18 |9.m.Rb..E.M.,-..|
92 b0 40 80 c5 43 0c 93 95 ff 7e f7 19 3c 48 81 |..@..C....~..<H.|
[11177 more]
},
},
},
}
}}}
According to the report, this happened after running for 90 minutes on
live traffic with ~10kreq/s.
Martin spent Monday looking at this, but did not find anything conclusive.
--
--
Ticket URL: <
https://www.varnish-cache.org/trac/ticket/1670#comment:1>
Varnish <
https://varnish-cache.org/>
The Varnish HTTP Accelerator
_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs