Mailing List Archive

How does Transfer-Encoding: chunked work with CGI?
Hi,
I haven't been able to find any documentation on how (or if) CGI handles
Transfer-Encoding: chunked. It looks to me like Apache is eating the chunk
size line, as the first byte of data I read from stdin is the data I'm
posting. I can clearly see the chunk size line preceding the data in a
packet capture of the http post.

Does Apache CGI support Transfer-Encoding: chunked? Is there documentation
of how it works?

Thanks,
Chris
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Chris Cross wrote:
> Hi,
> I haven't been able to find any documentation on how (or if) CGI handles
> Transfer-Encoding: chunked. It looks to me like Apache is eating the
> chunk size line, as the first byte of data I read from stdin is the data
> I'm posting. I can clearly see the chunk size line preceding the data in
> a packet capture of the http post.
>
> Does Apache CGI support Transfer-Encoding: chunked? Is there
> documentation of how it works?

This is not an Apache question. The CGI spec doesn't deal with
chunked data, so any HTTP/1.1 implementation of CGI (Apache included)
has to de-chunk any chunked input data on behalf of the script.

--
Nick Kew

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Nick Kew <nick@webthing.com> wrote on 09/11/2009 09:55:25 AM:
> Chris Cross wrote:
> > Hi,
> > I haven't been able to find any documentation on how (or if) CGI
handles
> > Transfer-Encoding: chunked. It looks to me like Apache is eating the
> > chunk size line, as the first byte of data I read from stdin is the
data
> > I'm posting. I can clearly see the chunk size line preceding the data
in
> > a packet capture of the http post.
> >
> > Does Apache CGI support Transfer-Encoding: chunked? Is there
> > documentation of how it works?
>
> This is not an Apache question. The CGI spec doesn't deal with
> chunked data, so any HTTP/1.1 implementation of CGI (Apache included)
> has to de-chunk any chunked input data on behalf of the script.
>
> --
> Nick Kew

Thanks for the quick reply Nick. The CONTENT_LENGTH env variable is not
being set when I post with Transfer-Encoding: chunked. Does that mean that
Apache doesn't support de-chunking for CGI?

Chris
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Chris Cross wrote:

> Thanks for the quick reply Nick. The CONTENT_LENGTH env variable is not
> being set when I post with Transfer-Encoding: chunked. Does that mean
> that Apache doesn't support de-chunking for CGI?

Hmmm. That sounds buggish (dammit, it's a long time since I did
anything nontrivial with CGI).

What Apache version, and what CGI implementation?

--
Nick Kew

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Nick Kew <nick@webthing.com> wrote on 09/11/2009 10:31:46 AM:
> Hmmm. That sounds buggish (dammit, it's a long time since I did
> anything nontrivial with CGI).
>
> What Apache version, and what CGI implementation?

From my CGI env:

SERVER_SOFTWARE=Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with
Suhosin-Patch mod_perl/2.0.4 Perl/v5.10.0
...
GATEWAY_INTERFACE=CGI/1.1
SERVER_PROTOCOL=HTTP/1.1
REQUEST_METHOD=PUT
QUERY_STRING=large-file.xml
REQUEST_URI=/cgi-bin/chunked-upload?large-file.xml
SCRIPT_NAME=/cgi-bin/chunked-upload


Chris
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Chris Cross wrote:
> Nick Kew <nick@webthing.com> wrote on 09/11/2009 10:31:46 AM:
> > Hmmm. That sounds buggish (dammit, it's a long time since I did
> > anything nontrivial with CGI).
> >
> > What Apache version, and what CGI implementation?
>
> From my CGI env:

That's not actually what I asked (many CGI implementations would
give the same). But it's probably enough to go on: a recent-
enough apache release, probably prefork, and probably mod_cgi
(any Ubuntu folks here know whether it could be something
different without the server admin having made a conscious choice)?

Quick sanity checks:

(1) if your test script just reads to EOF, does it get the right
(de-chunked) inputs?
(2) does it fail with any reasonably-standard CGI library?

--
Nick Kew

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Nick Kew <nick@webthing.com> wrote on 09/11/2009 11:18:03 AM:
>
> Chris Cross wrote:
> > Nick Kew <nick@webthing.com> wrote on 09/11/2009 10:31:46 AM:
> > > Hmmm. That sounds buggish (dammit, it's a long time since I did
> > > anything nontrivial with CGI).
> > >
> > > What Apache version, and what CGI implementation?
> >
> > From my CGI env:
>
> That's not actually what I asked (many CGI implementations would
> give the same). But it's probably enough to go on: a recent-
> enough apache release, probably prefork, and probably mod_cgi
> (any Ubuntu folks here know whether it could be something
> different without the server admin having made a conscious choice)?
>
> Quick sanity checks:
>
> (1) if your test script just reads to EOF, does it get the right
> (de-chunked) inputs?
> (2) does it fail with any reasonably-standard CGI library?
>
> --
> Nick Kew

Nick,

1) Reading till EOF works fine. I'd gotten in the habit of using
Content-Length in the non-chunked case so hadn't tried it until now. The
"Transfer-Encoding: chunked" header is set. Is that proper if "de-chunking"
has occurred?

2) I'm just doing C++ without any third party libraries.

So my test program is working now but I have another questions. Can you
elaborate on what is a CGI implementation? I'm operating at the level of
installing apache and POSTing to it and CGI is the part that calls my cgi
script...

Thanks again
Chris
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Chris Cross wrote:

> Nick,
>
> 1) Reading till EOF works fine. I'd gotten in the habit of using
> Content-Length in the non-chunked case so hadn't tried it until now. The
> "Transfer-Encoding: chunked" header is set. Is that proper if
> "de-chunking" has occurred?

Well, the CGI spec pre-dates HTTP/1.1, and is silent on the subject
of chunked encoding. But now that you mention it, it seems to me
to make sense to remove the header when we de-chunk the input.

> So my test program is working now but I have another questions. Can you
> elaborate on what is a CGI implementation?

mod_cgi is one. mod_cgid is another. Then there are a couple
of fastcgi implementations including mod_fcgid. Then there are
language-specific implementations, such as mod_perl's, the first
to run in-process CGI.

--
Nick Kew

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
Hi.
As regarding the overall content-length header, has anyone re-checked
the relevant part of the HTTP RFCs ? I haven't yet, but I seem to
remember that for chunked encoding, each chunk has an individual length,
but there is no overall content-length.
Which would meake sense, because at the time the initial headers are
sent out, the total length is not necessarily known.
In terms of CGI, that would mean that when Apache calls the CGI module,
it cannot set the CONTENT-LENGTH as an environment value, since it isn't
known yet.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: How does Transfer-Encoding: chunked work with CGI? [ In reply to ]
André Warnier wrote:
> Hi.
> As regarding the overall content-length header, has anyone re-checked
> the relevant part of the HTTP RFCs ? I haven't yet, but I seem to
> remember that for chunked encoding, each chunk has an individual length,
> but there is no overall content-length.
> Which would meake sense, because at the time the initial headers are
> sent out, the total length is not necessarily known.
> In terms of CGI, that would mean that when Apache calls the CGI module,
> it cannot set the CONTENT-LENGTH as an environment value, since it isn't
> known yet.

That's what it does, yup. In fact the combination is prohibited, so the
content length header is unset in the presence of chunked encoding.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org