Mailing List Archive

#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
--------------------------+----------------------
Reporter: mattrobenolt | Type: defect
Status: new | Priority: normal
Milestone: | Component: varnishd
Version: unknown | Severity: major
Keywords: vary, panic |
--------------------------+----------------------
This is in 4.0.3-rc1:

{{{
varnish> panic.show
200
Last panic at: Tue, 20 Jan 2015 08:50:18 GMT
Assert error in VRY_Validate(), cache/cache_vary.c line 377:
Condition(strlen((const char*)vary+3) == vary[2]) not true.
thread = (cache-worker)
version = varnish-4.0.3-rc1 revision eea73f6
ident = Linux,3.13.0-43-generic,x86_64,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
0x4333fa: /usr/sbin/varnishd() [0x4333fa]
0x43ef5b: /usr/sbin/varnishd() [0x43ef5b]
0x43f24c: /usr/sbin/varnishd(VRY_Finish+0x2c) [0x43f24c]
0x43677f: /usr/sbin/varnishd() [0x43677f]
0x437909: /usr/sbin/varnishd(CNT_Request+0x869) [0x437909]
0x42d3cb: /usr/sbin/varnishd(HTTP1_Session+0x3eb) [0x42d3cb]
0x43b9f8: /usr/sbin/varnishd() [0x43b9f8]
0x43c7a9: /usr/sbin/varnishd(SES_pool_accept_task+0x2a9) [0x43c7a9]
0x4362d3: /usr/sbin/varnishd(Pool_Work_Thread+0x373) [0x4362d3]
0x449478: /usr/sbin/varnishd() [0x449478]
req = 0x7f8484162020 {
sp = 0x7f84835c83a0, vxid = 1093206021, step = R_STP_LOOKUP,
req_body = R_BODY_NONE,
restarts = 0, esi_level = 0,
sp = 0x7f84835c83a0 {
fd = 1780, vxid = 19464196,
client = 139.216.143.85 56733,
step = S_STP_WORKING,
},
worker = 0x7f847dbcec20 {
ws = 0x7f847dbcee38 {
id = "wrk",
{s,f,r,e} = {0x7f847dbce410,0x7f847dbce410,(nil),+2048},
},
VCL::method = 0x0,
VCL::return = lookup,
},
ws = 0x7f84841621b8 {
id = "req",
{s,f,r,e} = {0x7f8484164010,+57360,+57360,+57360},
},
http[req] = {
ws = 0x7f84841621b8[req]
"GET",
"/api/3.0/embed/threadDetails.json?thread=3437164122&post=1803766928&post=1803768038&post=1803768451&post=1803769482&post=1803769704&post=1803770164&post=1803770423&post=1803770527&post=1803771199&post=1803771778&post=1803776215&post=1803776846&post=1803772192&post=1803774533&post=1803778326&post=1803780042&post=1803779439&post=1803779000&post=1803779581&post=1803779969&post=1803779385&post=1803780619&post=1803780362&post=1803781117&post=1803780604&post=1803780970&post=1803781938&post=1803783331&post=1803784340&post=1803784695&post=1803784944&post=1803785458&post=1803785913&post=1803787313&post=1803788789&post=1803789169&post=1803789458&post=1803789692&post=1803790232&post=1803790262&post=1803787229&post=1803787445&post=1803792488&post=1803790856&post=1803791050&post=1803792845&post=1803793364&post=1803793960&post=1803794542&post=1803794777&post=1803795279&post=1803798995&post=1803797360&post=1803797904&post=1803798399&post=1803794310&post=1803795071&post=1803796109&post=1803796351&post=1803796886&post=1803797943&post=1803799538&post=1803799765&post=1803799711&post=1803800126&post=1803800771&post=1803800243&post=1803800953&post=1803801055&post=1803801933&post=1803802118&post=1803802488&post=1803803994&post=1803804069&post=1803804190&post=1803804866&post=1803803350&post=1803809275&post=1803802353&post=1803801973&post=1803802675&post=1803803880&post=1803804145&post=1803804720&post=1803805244&post=1803805465&post=1803805734&post=1803805751&post=1803806375&post=1803806393&post=1803806426&post=1803806857&post=1803806938&post=1803807262&post=1803807275&post=1803807972&post=1803808064&post=1803808686&post=1803809180&post=1803809763&post=1803810570&post=1803810684&post=1803811157&post=1803811663&post=1803812041&post=1803811693&post=1803811991&post=1803812098&post=1803813225&post=1803817939&post=1803819706&post=1803813111&post=1803813975&post=1803816704&post=1803818767&post=1803819068&post=1803819923&post=1803812856&post=1803814869&post=1803817732&post=1803818079&post=1803819293&post=1803819367&post=1803820186&post=1803821100&post=1803823636&post=1803820258&post=1803820479&post=1803820971&post=1803823673&post=1803820678&post=1803821326&post=1803821416&post=1803821985&post=1803822539&post=1803822681&post=1803822699&post=1803823038&post=1803824185&post=1803824218&post=1803820397&post=1803812639&post=1803817614&post=1803811669&post=1803813341&post=1803814807&post=1803821781&post=1803823604&post=1803825156&post=1803829041&post=1803827092&post=1803821121&post=1803821728&post=1803822445&post=1803823806&post=1803824317&post=1803825423&post=1803826169&post=1803826921&post=1803828614&post=1803828796&post=1803825323&post=1803826440&post=1803827054&post=1803828042&post=1803829050&post=1803824867&post=1803827297&post=1803827923&post=1803828684&post=1803830789&post=1803825960&post=1803832491&post=1803825548&post=1803825729&post=1803826591&post=1803827339&post=1803827684&post=1803827782&post=1803827920&post=1803830423&post=1803830606&post=1803830752&post=1803832619&post=1803830006&post=1803830862&post=1803832476&post=1803833055&post=1803831263&post=1803829759&post=1803829960&post=1803832598&post=1803833725&post=1803832018&post=1803829682&post=1803829547&post=1803830404&post=1803831253&post=1803832881&post=1803833024&post=1803836698&post=1803839456&post=1803823195&post=1803838545&post=1803838849&post=1803830290&post=1803831670&post=1803834030&post=1803834460&post=1803835632&post=1803836703&post=1803834064&post=1803834077&post=1803835226&post=1803836050&post=1803837253&post=1803839237&post=1803840274&post=1803840293&post=1803840767&post=1803840970&post=1803842194&post=1803842310&post=1803843155&post=1803844815&post=1803845448&post=1803834768&post=1803836184&post=1803840582&post=1803841416&post=1803843180&post=1803845946&post=1803836005&post=1803837048&post=1803837176&post=1803837250&post=1803837420&post=1803837771&post=1803837894&post=1803837946&post=1803837990&post=1803837995&post=1803838151&post=1803838163&post=1803838348&post=1803838534&post=1803838672&post=1803838725&post=1803839620&post=1803840125&post=1803840377&post=1803841317&post=1803841401&post=1803842130&post=1803843013&post=1803843045&post=1803843130&post=1803843259&post=1803843582&post=1803843815&post=1803843902&post=1803844408&post=1803845040&post=1803845559&post=1803845674&post=1803845816&post=1803845819&post=1803845862&post=1803845905&post=1803846354&post=1803801507&post=1803801951&post=1803803080&post=1803803396&post=1803825635&post=1803804460&post=1803846420&post=1803847030&post=1803847564&post=1803847837&post=1803847870&post=1803848375&post=1803848774&post=1803849122&post=1803846526&post=1803848120&post=1803848570&post=1803848845&post=1803847046&post=1803848500&post=1803849371&post=1803849443&post=1803849656&post=1803849738&post=1803849833&post=1803849926&post=1803849987&post=1803850244&post=1803850303&post=1803851750&post=1803851651&post=1803851795&post=1803853359&post=1803853572&post=1803854567&post=1803852417&post=1803854359&post=1803851085&post=1803851794&post=1803852380&post=1803852614&post=1803852914&post=1803853792&post=1803853892&post=1803853986&post=1803854065&post=1803854128&post=1803854221&post=1803854607&post=1803854674&post=1803855101&post=1803855501&post=1803855723&post=1803854856&post=1803850647&post=1803850688&post=1803851191&post=1803852288&post=1803850516&post=1803852522&post=1803852892&post=1803851914&post=1803846864&post=1803847831&post=1803848372&post=1803847413&post=1803853244&post=1803847081&post=1803848162&post=1803848702&post=1803849262&post=1803851376&post=1803853185&post=1803846601&post=1803846799&post=1803856015&post=1803855924&post=1803856300&post=1803856585&post=1803856617&post=1803856624&post=1803856724&post=1803856773&post=1803859083&post=1803859476&post=1803857054&post=1803857276&post=1803857224&post=1803857959&post=1803857988&post=1803858106&post=1803858194&post=1803858352&post=1803858711&post=1803858794&post=1803858945&post=1803858981&post=1803859165&post=1803859174&post=1803859467&post=1803859482&post=1803856787&post=1803857403&post=1803856333&post=1803856369&post=1803857744&post=1803856364&post=1803859520&post=1803857646&post=1803858408&post=1803857896&post=1803856641&post=1803857353&post=1803858264&post=1803857498&post=1803859790&post=1803859801&post=1803859993&post=1803860235&post=1803860568&post=1803860156&post=1803860701&post=1803860869&post=1803860229&post=1803860939&post=1803861037&post=1803861402&post=1803861784&post=1803862349&post=1803862683&post=1803863125&post=1803864025&post=1803862207&post=1803862635&post=1803864505&post=1803864679&post=1803865006&post=1803865605&post=1803865716&post=1803865738&post=1803865813&post=1803866068&post=1803866290&post=1803866351&post=1803866416&post=1803866473&post=1803866490&post=1803863422&post=1803863131&post=1803862755&post=1803862963&post=1803863603&post=1803864852&post=1803866080&post=1803866711&post=1803864475&post=1803869305&post=1803868616&post=1803868474&post=1803873984&post=1803868392&post=1803868770&post=1803868641&post=1803869466&post=1803870106&post=1803871894&post=1803872766&post=1803873866&post=1803868318&post=1803869075&post=1803869586&post=1803869821&post=1803869109&post=1803869965&post=1803870075&post=1803870398&post=1803871158&post=1803871221&post=1803872131&post=1803872711&post=1803873401&post=1803874172&post=1803874214&post=1803875041&post=1803875501&post=1803875679&post=1803876129&post=1803874834&post=1803875115&post=1803879129&post=1803878354&post=1803881080&post=1803876765&post=1803880556&post=1803881324&post=1803872316&post=1803879657&post=1803878804&post=1803879418&post=1803883657&post=1803883671&post=1803884259&post=1803884429&post=1803884352&post=1803885139&post=1803886340&api_key=xxx",
"HTTP/1.1",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0)
Gecko/20100101 Firefox/34.0",
"Accept: */*",
"Accept-Language: en-us,el;q=0.7,en;q=0.3",
"DNT: 1",
"X-Requested-With: XMLHttpRequest",
"Referer:
http://disqus.com/embed/comments/?base=default&disqus_version=7a9534a2&version=d7cfc2b66e4c2426e19da7b9195cf0c4&f=defencenet&t_i=node%2F244623&t_u=http%3A%2F%2Fwww.defencenet.gr%2Fdefence%2Fitem%2F%25CF%2584%25CE%25B5%25CF%2581%25CE%25AC%25CF%2583%25CF%2584%25CE%25B9%25CE%25B1-%25CE%25B4%25CE%25B9%25CE%25B1%25CF%2581%25CF%2581%25CE%25BF%25CE%25AE-%25CE%25B4%25CE%25B5%25CE%25BE%25CE%25B9%25CF%258E%25CE%25BD-%25CF%2588%25CE%25B7%25CF%2586%25CE%25BF%25CF%2586%25CF%258C%25CF%2581%25CF%2589%25CE%25BD-%25CF%2580%25CF%2581%25CE%25BF%25CF%2582-%25CF%2583%25CF%2585%25CF%2581%25CE%25B9%25CE%25B6%25CE%25B1-%25CE%25B4%25CE%25B5%25CE%25AF%25CF%2587%25CE%25BD%25CE%25BF%25CF%2585%25CE%25BD-%25CF%2583%25CF%2584%25CE%25BF%25CE%25BD-%25CE%25B1%25CF%2583%25CE%25B1%25CE%25BC%25CE%25B1%25CF%2581%25CE%25AC-%25CF%2584%25CE%25B7%25CE%25BD-%25CE%25AD%25CE%25BE%25CE%25BF%25CE%25B4%25CE%25BF&t_e=%CE%A4%CE%B5%CF%81%CE%AC%CF%83%CF%84%CE%B9%CE%B1%20%CE%B4%CE%B9%CE%B1%CF%81%CF%81%CE%BF%CE%AE%20%CE%B4%CE%B5%CE%BE%CE%B9%CF%8E%CE%BD%20%CF%88%CE%B7%CF%86%CE%BF%CF%86%CF%8C%CF%81%CF%89%CE%BD%20%CF%80%CF%81%CE%BF%CF%82%20%CE%A3%CE%A5%CE%A1%CE%99%CE%96%CE%91%20-%20%CE%94%CE%B5%CE%AF%CF%87%CE%BD%CE%BF%CF%85%CE%BD%20%CF%83%CF%84%CE%BF%CE%BD%20%CE%91.%CE%A3%CE%B1%CE%BC%CE%B1%CF%81%CE%AC%20%CF%84%CE%B7%CE%BD%20%CE%AD%CE%BE%CE%BF%CE%B4%CE%BF&t_d=%CE%A4%CE%B5%CF%81%CE%AC%CF%83%CF%84%CE%B9%CE%B1%20%CE%B4%CE%B9%CE%B1%CF%81%CF%81%CE%BF%CE%AE%20%CE%B4%CE%B5%CE%BE%CE%B9%CF%8E%CE%BD%20%CF%88%CE%B7%CF%86%CE%BF%CF%86%CF%8C%CF%81%CF%89%CE%BD%20%CF%80%CF%81%CE%BF%CF%82%20%CE%A3%CE%A5%CE%A1%CE%99%CE%96%CE%91%20-%20%CE%94%CE%B5%CE%AF%CF%87%CE%BD%CE%BF%CF%85%CE%BD%20%CF%83%CF%84%CE%BF%CE%BD%20%CE%91.%CE%A3%CE%B1%CE%BC%CE%B1%CF%81%CE%AC%20%CF%84%CE%B7%CE%BD%20%CE%AD%CE%BE%CE%BF%CE%B4%CE%BF%20%7C%20DefenceNet.gr&t_t=%CE%A4%CE%B5%CF%81%CE%AC%CF%83%CF%84%CE%B9%CE%B1%20%CE%B4%CE%B9%CE%B1%CF%81%CF%81%CE%BF%CE%AE%20%CE%B4%CE%B5%CE%BE%CE%B9%CF%8E%CE%BD%20%CF%88%CE%B7%CF%86%CE%BF%CF%86%CF%8C%CF%81%CF%89%CE%BD%20%CF%80%CF%81%CE%BF%CF%82%20%CE%A3%CE%A5%CE%A1%CE%99%CE%96%CE%91%20-%20%CE%94%CE%B5%CE%AF%CF%87%CE%BD%CE%BF%CF%85%CE%BD%20%CF%83%CF%84%CE%BF%CE%BD%20%CE%91.%CE%A3%CE%B1%CE%BC%CE%B1%CF%81%CE%AC%20%CF%84%CE%B7%CE%BD%20%CE%AD%CE%BE%CE%BF%CE%B4%CE%BF&s_o=default",
"Connection: keep-alive",
"X-Forwarded-For: 139.216.143.85",
"X-Forwarded-Proto: http",
"Host: disqus.com",
"Disqus-Root: 1",
"Disqus-Cookie-sessionid: c0235546-a054-11e4-a704-xxx",
"Disqus-Logged-In: 1",
"Cookie: ;;sessionid=c0235546-a054-11e4-a704-xxx",
},
vcl = {
srcname = {
"input",
"Builtin",
"includes/4.0/recv-common.vcl",
"includes/downgrade.vcl",
"includes/4.0/responses/301-moved.vcl",
"includes/4.0/responses/302-found.vcl",
"includes/4.0/responses/404-not-found.vcl",
"includes/4.0/responses/999-any.vcl",
"includes/x-served-by.vcl",
},
},
},
}}}

I haven't been able to reproduce this, but it was happening with
production traffic.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1661>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377 [ In reply to ]
#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
--------------------------+----------------------
Reporter: mattrobenolt | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: unknown
Severity: major | Resolution:
Keywords: vary, panic |
--------------------------+----------------------

Comment (by lkarsten):

Can you describe how Vary is being used from the backend in this
application?

If possible, can you provide the complete backend headers on a miss to
this API endpoint?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1661#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377 [ In reply to ]
#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
--------------------------+----------------------
Reporter: mattrobenolt | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: unknown
Severity: major | Resolution:
Keywords: vary, panic |
--------------------------+----------------------

Comment (by lkarsten):

I asked reporter for any changed parameters. startup options are: -p
shortlived=0 -p shm_reclen=4084 -p fetch_chunksize=4k -p timeout_idle=5 -p
timeout_req=2

Martin have dug into this and it looks like bad ws overflow handling in
the request parsing vary code.

Anyone hitting this assert should either reduce their workspace use in
vcl_recv (regsub/header operations or hungry vmods), or increase the
workspace_client parameter.

Martin is cooking on some (failing) test cases that will be reported as
separate bugs. Leaving this issue open until that is in place and
referenced here.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1661#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377 [ In reply to ]
#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
--------------------------+----------------------
Reporter: mattrobenolt | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: unknown
Severity: major | Resolution:
Keywords: vary, panic |
--------------------------+----------------------

Comment (by mattrobenolt):

Also, fwiw, the response Vary is simply `Vary: Accept-Encoding` if that's
still valuable. Sorry for the delay.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1661#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377 [ In reply to ]
#1661: Assert error in VRY_Validate(), cache/cache_vary.c line 377
--------------------------+----------------------
Reporter: mattrobenolt | Owner:
Type: defect | Status: closed
Priority: normal | Milestone:
Component: varnishd | Version: unknown
Severity: major | Resolution: fixed
Keywords: vary, panic |
--------------------------+----------------------
Changes (by martin):

* status: new => closed
* resolution: => fixed


Comment:

This looks like it boils down to workspace exhaustion. Some patches were
added to make it clearer that that is what happened. Not seen evidence of
this assertion happening again, so closing the bug.

Martin

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1661#comment:4>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs