Mailing List Archive

#1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
---------------------------------+----------------------
Reporter: geoff | Type: defect
Status: new | Priority: normal
Milestone: Varnish 4.0 release | Component: varnishd
Version: 4.0.3 | Severity: normal
Keywords: |
---------------------------------+----------------------
We are occasionally seeing responses with code 204 as chunked-encoded with
one length byte `0` in the response body:

{{{
HTTP/1.1 204 No Content
[...]
Vary: Accept-Encoding
Transfer-Encoding: chunked
Connection: keep-alive

0
}}}

These are not gzipped, as there is no Content-Encoding header, but they
look very much as if they were set up to be delivered as gzipped -- in
every example we've seen, they are always delivered for requests with
`Accept-Encoding: gzip`.

By definition, a 204 response should have no body at all. Instead, a
response as shown above is chunked-encoded with a single chunk of length
0.

Unfortunately we haven't found a way to reliably reproduce the problem --
we are seeing these cases in error reports from the load balancer in front
of Varnish, and they always seem to happen during load tests. When we test
exactly the same request manually, so far we always get a response with no
body, as expected.

After trying for a while to make it reproducible, I'd like to report the
problem now while we keep looking.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------------
Reporter: geoff | Owner: Poul-Henning Kamp <phk@…>
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution: fixed
Keywords: |
----------------------+----------------------------------------
Changes (by Poul-Henning Kamp <phk@…>):

* owner: => Poul-Henning Kamp <phk@…>
* status: new => closed
* resolution: => fixed


Comment:

In [4df51cf26691e875682a39b279cd03f7e202fae8]:
{{{
#!CommitTicketReference repository=""
revision="4df51cf26691e875682a39b279cd03f7e202fae8"
Try to sort out the delivery of resp.body for special resp.status cases.

Treat C-L or A-E in 204 backend responses as fetch_error.

This hopefully fixes #1761
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------------
Reporter: geoff | Owner: Poul-Henning Kamp <phk@…>
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution: fixed
Keywords: |
----------------------+----------------------------------------

Comment (by geoff):

I'm afraid I can't test this patch. Our problem is in 4.0.3, whereas the
patch applies to master as of July 26th, and the differences are too large
for me to get a successful backport out of the merge conflicts.

The patch is also much more comprehensive than just fixing the problem
with the 204 response -- it covers a number of different cases concerning
delivery, as well as the fetch error regarding backend responses.

So I'd like to attempt a simpler patch for 4.0.3 to address the problem
with 204 responses.

If I'm reading the patch correctly, the essential step for that is in
delivery: if resp.status == 204, set the wantbody flag (req->wantbody in
4.0.3) to false and remove the Content-Length header.

When the wantbody flag is false, then neither ESI_Deliver() nor any VDP is
called, and "Transfer-Encoding: chunked" is not generated. So no response
body is delivered, nor any response headers that would require a body to
be present.

Would that suffice, or am I missing something?

I also see `if (wantbody == 0) return;` in ESI delivery; is that necessary
as well? If the wantbody flag is false, then it seems to me that delivery
would never get to ESI delivery in the first place.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------------
Reporter: geoff | Owner: Poul-Henning Kamp <phk@…>
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------------
Changes (by geoff):

* status: closed => reopened
* resolution: fixed =>


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------------
Reporter: geoff | Owner: Poul-Henning Kamp <phk@…>
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------------

Comment (by geoff):

BTW, I'd be happy to take ownership of this ticket myself, for a simpler
patch to be applied to 4.0.x.

phk, if you can give me ACK/NACK as to whether I'm on the right track for
a fix, then I'll take it from here.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:4>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: new
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------
Changes (by fgsch):

* status: reopened => new
* owner: Poul-Henning Kamp <phk@…> => geoff


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:5>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: new
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by geoff):

Patch for 4.0.3 submitted to patchwork: https://www.varnish-
cache.org/patchwork/patch/370/

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: new
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by geoff):

Apparently I don't have the bits to close this ticket, it can be closed
now.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:7>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution: fixed
Keywords: |
----------------------+----------------------------------
Changes (by phk):

* status: new => closed
* resolution: => fixed


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:8>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------
Changes (by martin):

* status: closed => reopened
* resolution: fixed =>


Comment:

Reopened as a reminder to apply this to 4.0 maintenance branch.

The current patch looks like it misses a test case.

Geoff: Could you provide a test case and push to 4.0?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:9>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by lkarsten):

original msg id to -dev: <561AAB12.3000402@uplex.de>

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:10>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by geoff):

Unfortunately I don't have a test case, because we never discovered how to
reliably reproduce the problem. So I couldn't create a vtc that would fail
unless the patch is applied.

I'm confident that the patch resolves the problem, because it never
happened again after we applied it in production.

FWIW, the patch is a one-liner: set the wantbody flag to 0 when the
response code was 204 (previously done only when the response is 304) --
this is one of the things that happens in phk's patch for what is now 4.1.
For the reasons described in the second comment, I think this has to be
the solution. But I'm afraid I can't prove it with a vtc, only from our
experience of a bug being there and then going away.

@martin, what should we do?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:11>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by geoff):

martin and I agreed in bugwash that we'll push the patch to the 4.0 branch
and close.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:12>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: reopened
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+----------------------------------

Comment (by geoff):

patch pushed, but evidently I don't have the bits to close the ticket

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:13>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1761: 204 responses intermittently delivered as chunk-encoded with length byte = 0 [ In reply to ]
#1761: 204 responses intermittently delivered as chunk-encoded with length byte =
0
----------------------+----------------------------------
Reporter: geoff | Owner: geoff
Type: defect | Status: closed
Priority: normal | Milestone: Varnish 4.0 release
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution: fixed
Keywords: |
----------------------+----------------------------------
Changes (by Dridi):

* status: reopened => closed
* resolution: => fixed


Comment:

Fixed in [f4539dd2166bfd422a722967fe31e2e3e2c93ad5]

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1761#comment:14>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs