Mailing List Archive

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #2 from nemo <bugs@m8y.org> ---
So, I just tried using netcat:
{ echo -ne "HTTP/1.0 200 OK\r\nContent-Type:
application/json;utf-8\r\nContent-Length: $(wc -c <test.json)\r\n\r\n"; cat
test.json; } | nc -l -s 127.0.0.1 -p 8080

And this *ALSO* worked fine...

So, that eliminates mod_proxy I think.

This *only* seems to impact the combination of the actual endpoint (rocket 0.4)
and rate_limit.
wget --limit-rate does not seem to cause issues.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #3 from Eric Covener <covener@gmail.com> ---
maybe the backend fails if a write blocks? Have you looked at a packet capture
between Apache and the backend?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #4 from nemo <bugs@m8y.org> ---
Hm. write blocks? Can you elaborate? This is a GET, BTW, not a POST. So I'm
just fetching data from the backend, not writing to it.

Haven't tried a full dump yet, apart from trace8 which didn't seem unusual.
Actually I was pretty hopeful this was just some odd interaction of mod_proxy
and rate_limit right up until the point where my netcat test worked.

I guess that's the only thing left to try though ? (I do so hate poking through
packet traces)
Apart from hunting the Rocket documentation or testing the 0.5 beta..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #5 from nemo <bugs@m8y.org> ---
BTW, another variant that failed. I tried pointing ProxyPass at the production
apache.
So. Rate limited test apache ? https://prodserver/api ?
http://localhost:8085/api

This one failed in the same fashion (any rate limit resulting in a fetch longer
than 20 seconds causing partial fetch).

Once again, curl and wget rate limits against the production server worked
fine. Even with fetch times of several minutes.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #6 from nemo <bugs@m8y.org> ---
Bit of good news for me - I reproduced it with latest release Rocket v0.4.10
and their reference examples/static_files.
STR (sorry for mercurialisms, I don't care for git's user interface or
featureset at all). Also this assumes rust stable installed. Rustup takes care
of that.

? hg clone git+https://github.com/SergioBenitez/Rocket.git;hg up -r v0.4.10
? cd Rocket/examples/static_files
? cargo build
? scp ../../target/debug/static_files testserver:
? ssh testserver
? mkdir static
? cp /var/www/html/test.json static # this is the large 2.7MB json file.
I'm pretty sure this will be a problem with any random large file though. I
will verify with something large and public like a PG text next.
? ./static_files

And, in Apache 2.4.38 I was using:
SetOutputFilter RATE_LIMIT
SetEnv rate-limit 50
SetEnv rate-initial-burst 50

Timeout 120
ProxyTimeout 300
ProxyPass "/apitest" "http://localhost:8000/"

Then just hitting http://testserver/apitest/test.json

Once again, the RATE_LIMITed Apache 2.4.38 caused truncation but a wget or curl
--limit-rate directly to port 8000 was not a problem.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #7 from nemo <bugs@m8y.org> ---
Rocket master (which is probably close to 0.5 release any-day-now) did *NOT*
trigger this issue.

So... this appears to be specific to the combination of Apache's RATE_LIMIT and
Rocket 0.4.x

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #8 from nemo <bugs@m8y.org> ---
18:16 < SergioB> There are read/write timeouts in 0.4 that aren't present in
0.5.

Disabling those prevented the issue. (read_timeout = 0 write_timeout = 0)
The timeout values by default are 5 seconds. There was no point during
trace6/trace8 where packets took longer than 5 seconds it seems to me... But
perhaps there was some bug in the implementation.

Still odd it only triggered with Apache's RATE_LIMIT and not wget/curl - I
guess that might be the only thing the Apache people might be interested in, in
case the module has some odd behaviour. I know httpd has a long history of
working around issues in other clients/servers ?

But under circumstances going to guess this is a low priority bug. ?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #9 from nemo <bugs@m8y.org> ---
One more thought. In all the cases where the transaction aborted, it tended to
happen very close to the end of the large file.
Like - 90%.
What if the read_timeout and write_timeout were *not* failing?
What if the Apache RATE_LIMIT has a bug, and overcorrects an accumulated
RATE_LIMIT average rate error towards the end?
Perhaps that's why curl and wget rate limits do not fail.
If so, there may have actually been a 5s stall.

I will try testing this in a bit.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=65348

--- Comment #10 from nemo <bugs@m8y.org> ---
Final update. I've been moved off onto other emergencies so won't be able to
test my "Apache RATE_LIMIT is suddenly halting all traffic towards the end of a
large transfer" theory.

I also noticed one additional bug with Apache RATE_LIMIT that I may have time
to write up properly someday.
If the proxied server responds with 304 plus an empty body (as it does with
If-None-Match of an existing ETag, I get an invalid request response with
RATE_LIMIT enabled).
========
Read error (The request is invalid.) in headers.
Retrying.
========

This happens in all browsers/clients - above is wget. No errors with
rate_limit disabled.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org