Mailing List Archive

#1805: Assert error in Tcheck()
#1805: Assert error in Tcheck()
---------------------------------+--------------------
Reporter: colas | Type: defect
Status: new | Priority: normal
Milestone: Varnish 4.0 release | Component: build
Version: unknown | Severity: normal
Keywords: |
---------------------------------+--------------------
Hy,

We use varnish-4.0.3 revision b8c4a34.
Started with all this params :

{{{
NFILES=131072
MEMLOCK=164000

DAEMON_OPTS="-a :80 \
-T 0.0.0.0:6083 \
-f /etc/varnish/cfvarnish.vcl \
-s file,/opt/cache/varnishstorage1.bin,35G \
-s file,/opt/cache/varnishstorage2.bin,35G \
-s file,/opt/cache/varnishstorage3.bin,35G \
-S /etc/varnish/secret \
-p http_gzip_support=on \
-p thread_pools=6 \
-p thread_pool_min=100 \
-p thread_pool_max=800 \
-p nuke_limit=500 \
-p cli_timeout=20 \
-l 160m"
}}}


Sometimes I get panic like this. This error could be very frequent, each
minutes for exemple.


{{{
varnish> panic.show
200
Last panic at: Thu, 22 Oct 2015 09:25:49 GMT
Assert error in Tcheck(), cache/cache.h line 1296:
Condition((t.b) != 0) not true.
thread = (cache-worker)
version = varnish-4.0.3 revision b8c4a34
ident =
Linux,3.2.0-4-amd64,x86_64,-sfile,-sfile,-sfile,-smalloc,-hcritbit,epoll
Backtrace:
0x435424: /usr/sbin/varnishd() [0x435424]
0x40e7e8: /usr/sbin/varnishd() [0x40e7e8]
0x429e33: /usr/sbin/varnishd() [0x429e33]
0x42c856: /usr/sbin/varnishd(http_FilterResp+0x86) [0x42c856]
0x420d3e: /usr/sbin/varnishd() [0x420d3e]
0x42142f: /usr/sbin/varnishd() [0x42142f]
0x438301: /usr/sbin/varnishd(Pool_Work_Thread+0x381) [0x438301]
0x44b29d: /usr/sbin/varnishd() [0x44b29d]
0x7f5fe03dbb50: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)
[0x7f5fe03dbb50]
0x7f5fe012595d: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
[0x7f5fe012595d]
busyobj = 0x7f456d889020 {
ws = 0x7f456d8890e0 { OVERFLOW
id = ""o",
{s,f,r,e} = {0x7f456d88b008,+57368,(nil),+57368},
},
refcnt = 2
retries = 3
failed = 0
state = 1
is_do_stream
is_do_pass
is_uncacheable
bodystatus = 0 (none),
},
http[bereq] = {
ws = 0x7f456d8890e0["o]
"GET",
"/aaaaaaaaaaaaaaaa.htm",
"HTTP/1.1",
"User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:41.0) Gecko/20100101
Firefox/41.0",
"Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3",
"Accept-Encoding: gzip, deflate",
"Referer: http://aaaaaaaaaaaaaaaaaaaaaaaaaaa",
"Cookie: md_users_res=1280x800; p=3; md_data=cham51;,
"Cache-Control: max-age=0",
"X-UA-Device: pc",
"Via: 1.1 varnish-v4",
"grace: none",
"X-Forwarded-For: xx.xxx.xx.xxx, xx.xxx.xx.xxx",
"Host: aaaaaaaaaaaaaaaaaa",
"X-Pass: Connected",
"X-Varnish: 655363",
"X-Varnish: 2326529",
"X-Varnish: 2326530",
},
http[beresp] = {
ws = 0x7f456d8890e0["o]
"HTTP/1.1",
"Backend fetch failed",
"Server: Varnish",
},
ws = 0x7f456d889270 {
id = "obj",
{s,f,r,e} = {0x7f55ebdb12c8,0x7f55ebdb12c8,(nil),+56},
},
objcore (FETCH) = 0x7f4578840020 {
refcnt = 2
flags = 0x106
objhead = 0x7f5fdf450500
}
obj (FETCH) = 0x7f55ebdb1180 {
vxid = 2149810179,
http[obj] = {
ws = 0x7f456d889270[obj]
"HTTP/1.1",
},
len = 0,
store = {
},
},
}
}}}


The error appears when I do a return(pass) in vcl_recv

File attache : the configuration vcl file

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution: worksforme
Keywords: |
--------------------+----------------------------------
Changes (by martin):

* status: new => closed
* resolution: => worksforme


Comment:

You are running out of backend fetch workspace. Try increasing your
workspace_backend parameter.

Martin

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution: worksforme
Keywords: |
--------------------+----------------------------------

Comment (by colas):

Thank you.
It works really better now ;)

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution:
Keywords: |
--------------------+----------------------------------
Changes (by colas):

* status: closed => reopened
* resolution: worksforme =>


Comment:

We changed it to ''' workspace_backend=256k''' , but now we have some
backend_fetch_failed whereas backend respond with status code 200.

With varnishlog we get this error :


{{{
- BereqHeader X-Forwarded-For: xx.xx.xx.xx, xx.xx.xx.xx
- BereqHeader Host: www.mysite.com
- BereqHeader X-Pass: Connected
- BereqHeader X-Pass-D: Connected - hash - Pass
- BereqHeader X-Varnish: 15568164
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 192 backend1(xx.xx.xx.xx,,80) xx.xx.xx.xx 53626
- Backend 192 backend backend1(xx.xx.xx.xx,,80)
- Timestamp Bereq: 1446131181.479810 0.000912 0.000912
- BogoHeader Too many headers: Set-Cookie: CF_user_
- HttpGarbage "HTTP/1.1%00200%00OK%00%0aDate: Thu, 29 Oct 2015
15:06:21 GMT%00%0aServer: Apache%00%0aExpires: 0%00%0aCache-Control:
private, no-store, no-cache, must-revalidate%00%0aPragma: no-cache%00
%0aSet-Cookie: CFSESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT;
path=/; domain=.....%00"
}}}



Why again this error ?
Something wrong ? And other parameter to change ?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution:
Keywords: |
--------------------+----------------------------------

Comment (by daghf):

You are hitting http_max_hdr, which controls the maximum number of header
lines allowed.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:4>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution: worksforme
Keywords: |
--------------------+----------------------------------
Changes (by daghf):

* status: reopened => closed
* resolution: => worksforme


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:5>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution:
Keywords: |
--------------------+----------------------------------
Changes (by colas):

* status: closed => reopened
* resolution: worksforme =>


Comment:

Thanks I haven't this error anymore.
I get an new one :


{{{
- BereqMethod POST
- BereqURL
/proxy.php?xdp_path=http%3A%2F%2Fmonsite.com%2Fcf_log.php
- BereqProtocol HTTP/1.1
- BereqHeader Accept: application/json, text/javascript, */*; q=0.01
- BereqHeader Content-Type: application/x-www-form-urlencoded;
charset=UTF-8
- BereqHeader X-Requested-With: XMLHttpRequest
- BereqHeader Referer:
- BereqHeader Accept-Language: fr-FR,fr;q=0.5
- BereqHeader Accept-Encoding: gzip, deflate
- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64;
Trident/7.0; LCJB; rv:11.0) like Gecko
- BereqHeader Content-Length: 635
- BereqHeader DNT: 1
- BereqHeader Cache-Control: no-cache
- BereqHeader Cookie: fofirdId=c652b193-cbff-4a0e-abdf-836087b20393;
xtvrn=$390974$; xtan390974=-; xtant390974=1; CF_SESSION_SYNC=1446135592
- BereqHeader X-UA-Device: pc
- BereqHeader X-UA-Device-Simplified: desktop
- BereqHeader Via: 1.1 varnish-v4
- BereqHeader grace: none
- BereqHeader X-Forwarded-For: xx.xx.xx.xx, xx.xx.xx.xx
- BereqHeader Host: monsite.com
- BereqHeader X-Pass: POST request
- BereqHeader X-Pass-D: POST request - hash - Pass
- BereqHeader X-Varnish: 2031880
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 258 backend1(xx.xx.xx.xx,,80) xx.xx.xx.xx 40582
- Backend 258 backend backedn1(xx.xx.xx.xx,,80)
- FetchError req.body read error: 104 (Connection reset by peer)
- FetchError backend write error: 104 (Connection reset by peer)
}}}


is there any page where I could find all possible error and find how to
solve it ?
I don't get the error all time, for exemple I can't reproduce from my
browser.

the response of this call is empty because it's just log and the content-
length = 20

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution: worksforme
Keywords: |
--------------------+----------------------------------
Changes (by daghf):

* status: reopened => closed
* resolution: => worksforme


Comment:

The situation here looks to be that client has closed the connection when
Varnish attempts to read the request body. This is not a Varnish bug.

For further discussion, please use the varnish-misc mailing list for these
inquiries. This is not a support forum.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:7>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1805: Assert error in Tcheck() [ In reply to ]
#1805: Assert error in Tcheck()
--------------------+----------------------------------
Reporter: colas | Owner:
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: build | Version: unknown
Severity: normal | Resolution: worksforme
Keywords: |
--------------------+----------------------------------

Comment (by colas):

Thank you for al advices.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1805#comment:8>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs