Mailing List Archive

[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

Archie Cobbs <archie@dellroad.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|Regression: Reverse proxy |Regression: Reverse proxy
|with websocket forceing 60 |with websocket forcing 60
|second timeout regardless |second timeout regardless
|of config |of config

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #1 from Yann Ylavic <ylavic.dev@gmail.com> ---
A possibly workaround is "ProxyWebsocketFallbackToProxyHttp off"
(https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html#proxywebsocketfallbacktoproxyhttp).

But since 2.4.52 websocket tunneling uses the smallest of Timeout and
ProxyTimeout, you may want to try 2.4.52 (and then add "Timeout 300" to your
configuration as you already tried in 2.4.51).

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #2 from Archie Cobbs <archie.cobbs@gmail.com> ---
> But since 2.4.52 websocket tunneling uses the smallest of Timeout and
> ProxyTimeout, you may want to try 2.4.52 (and then add "Timeout 300"
> to your configuration as you already tried in 2.4.51).

Wha? Isn't it wrong to make a completely undocumented and backward incompatible
change like that in a patch release?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #3 from Archie Cobbs <archie.cobbs@gmail.com> ---
FWIW, I can confirm that adding "Timeout 300" does work around the problem:

AH10212: proxy: websocket: tunnel running (timeout 300.000000)

But this still seems like a bug to me.

Or at least a poor design decision.

Why should ProxyTimeout's DEFAULT value (Timeout) ever override its CONFIGURED
value (ProxyTimeout) ? I definitely don't get it.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #4 from Ruediger Pluem <rpluem@apache.org> ---
I agree that the need to increase Timeout is not a good solution. Maybe we
should change the logic to choose the maximum of both timeouts (front end
socket, backend socket) instead of the minimum as backend timeouts can be
configured more selectively (per worker if needed) as front end timeouts and
typically the backend timeouts reflect the application requirements.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #5 from Ruediger Pluem <rpluem@apache.org> ---
Something like the following patch:

Index: modules/proxy/mod_proxy_http.c
===================================================================
--- modules/proxy/mod_proxy_http.c (revision 1898054)
+++ modules/proxy/mod_proxy_http.c (working copy)
@@ -1485,10 +1485,10 @@
return HTTP_INTERNAL_SERVER_ERROR;
}

- /* Set timeout to the lowest configured for client or backend */
+ /* Set timeout to the highest configured for client or backend */
apr_socket_timeout_get(backend->sock, &backend_timeout);
apr_socket_timeout_get(ap_get_conn_socket(c), &client_timeout);
- if (backend_timeout >= 0 && backend_timeout < client_timeout) {
+ if (backend_timeout >= 0 && backend_timeout > client_timeout) {
tunnel->timeout = backend_timeout;
}
else {
Index: modules/proxy/proxy_util.c
===================================================================
--- modules/proxy/proxy_util.c (revision 1898054)
+++ modules/proxy/proxy_util.c (working copy)
@@ -4718,10 +4718,10 @@
tunnel->origin->pfd->desc.s = ap_get_conn_socket(c_o);
tunnel->origin->pfd->client_data = tunnel->origin;

- /* Defaults to the smallest timeout of both connections */
+ /* Defaults to the biggest timeout of both connections */
apr_socket_timeout_get(tunnel->client->pfd->desc.s, &timeout);
apr_socket_timeout_get(tunnel->origin->pfd->desc.s, &tunnel->timeout);
- if (timeout >= 0 && (tunnel->timeout < 0 || tunnel->timeout > timeout)) {
+ if (timeout >= 0 && (tunnel->timeout < 0 || tunnel->timeout < timeout)) {
tunnel->timeout = timeout;
}

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #6 from Ruediger Pluem <rpluem@apache.org> ---
Another option instead of increasing the Timeout is

ProxyWebsocketFallbackToProxyHttp off

but in this case no timeout is applied to the websocket connection at all and
the proxy waits until one side closes the connection.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #7 from Yann Ylavic <ylavic.dev@gmail.com> ---
(In reply to Ruediger Pluem from comment #5)
> Something like the following patch:

+1

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #8 from Yann Ylavic <ylavic.dev@gmail.com> ---
(In reply to Archie Cobbs from comment #3)
>
> Why should ProxyTimeout's DEFAULT value (Timeout) ever override its
> CONFIGURED value (ProxyTimeout) ? I definitely don't get it.

The rationale is/was that since Timeout is the one applied on the client
connection side, and ProxyTimeout (or ProxyPass timeout= parameter) the one(s)
on the backend side, we'd use the smallest when tunneling between the client
and backend because if one side times out the tunnel should be terminated. But
this was probably a poor choice because in most cases it actually obliges to
change existing configuration to set both timeouts at the same/highest value.
Poor choices happen..

As RĂ¼diger said, note that in 2.4.43 no timeout was enforced in mod_proxy for
websocket tunneling (neither ProxyTimeout nor Timeout, so the timeout was
actually infinite), your configuration worked regardless of ProxyTimeout
probably thanks to the heartbeat. The above change was made precisely to honor
a timeout while tunneling without an ad hoc mechanism.

Let's change this again for 2.4.53.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #9 from Ruediger Pluem <rpluem@apache.org> ---
(In reply to Yann Ylavic from comment #7)
> (In reply to Ruediger Pluem from comment #5)
> > Something like the following patch:
>
> +1

r1898127

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

Ruediger Pluem <rpluem@apache.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Keywords| |FixedInTrunk,
| |PatchAvailable

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #10 from Yann Ylavic <ylavic.dev@gmail.com> ---
(In reply to Ruediger Pluem from comment #9)
> (In reply to Yann Ylavic from comment #7)
> > (In reply to Ruediger Pluem from comment #5)
> > > Something like the following patch:
> >
> > +1
>
> r1898127

Thanks! Since the mod_proxy_http hunk of your patch in comment 5 is 2.4 only,
how about we simply remove the tunnel->timeout setting there in 2.4 (on
backport) given that it's already what ap_proxy_tunnel_create() called just
above will do?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #11 from Archie Cobbs <archie.cobbs@gmail.com> ---
Thanks guys, appreciate the fast response.

I have two updates and a new question related to this.


The updates are:

1 - Adding "ProxyWebsocketFallbackToProxyHttp off" does in fact work
around the problem.

2 - Previously I said that setting "Timeout 300" also works around the
problem, but that is not entirely accurate. What that does is simply
change how long it takes for the particular problem I'm seeing to occur,
from 60 seconds to 5 minutes. This is as you would probably expect.


Here's my question:

> As RĂ¼diger said, note that in 2.4.43 no timeout was enforced in
> mod_proxy for websocket tunneling (neither ProxyTimeout nor Timeout,
> so the timeout was actually infinite), your configuration worked
> regardless of ProxyTimeout probably thanks to the heartbeat. The
> above change was made precisely to honor a timeout while tunneling
> without an ad hoc mechanism.

OK this makes sense. Basically my web application is using websockets
and sending a heartbeat every ~3 minutes. This websocket connection
was previously being proxied by mod_proxy to Tomcat, and mod_proxy
was not applying any particular timeout to that connection, so it
would stay up indefinitely.

But here's my question: isn't a websocket connection SUPPOSED to stay
up indefinitely?

Or at least, shouldn't any "timeout" value that is applied to that
connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
from when the connection was started)?

In other words, it seems to me like whatever mod_proxy was doing is
more correct, at least in the scenario of a proxied websocket connection,
than what mod_wstunnel is doing.

Can you explain to me what I'm missing? Thanks.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #12 from Ruediger Pluem <rpluem@apache.org> ---
(In reply to Archie Cobbs from comment #11)

>
> Or at least, shouldn't any "timeout" value that is applied to that
> connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
> from when the connection was started)?

It is an idle timeout. Connections with activity stay up.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #13 from Archie Cobbs <archie.cobbs@gmail.com> ---
> > Or at least, shouldn't any "timeout" value that is applied to that
> > connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
> > from when the connection was started)?
>
> It is an idle timeout. Connections with activity stay up.

Yes, you're right.

What Vaadin appears to be doing is creating more than one websocket connection,
and sending its heartbeat messages on only one of them.

Actually that does make sense - the purpose of the Vaadin heartbeat is to let
the
server know that the client is still alive, not to juice the websocket
connection(s).

Philosophically, timeouts are appropriate for a request/response world, but the
websockets world is a completely different beast. When doing websockets, it
really
should be left up to the two endpoints to decide if/when the connection has
"timed
out" and should be torn down; any intervening proxies should not have any
timeout
(in general) because (in general) they don't have any special knowledge of what
the two endpoint applications are doing with their TCP connection.

So I'd say that there is another subtle problem here, which is that there
really
should be two separate configuration variables for configuring (a) the proxy
timeout for regular connections and (b) the proxy timeout for websocket
connections.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #14 from Archie Cobbs <archie.cobbs@gmail.com> ---
> So I'd say that there is another subtle problem here, which is that there really
> should be two separate configuration variables for configuring (a) the proxy
> timeout for regular connections and (b) the proxy timeout for websocket
> connections.

Any thoughts on this question?

Right now the only way to have different timeout values for normal vs.
websockets
is to say "ProxyWebsocketFallbackToProxyHttp off" which seems like the wrong
tool.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #15 from Ruediger Pluem <rpluem@apache.org> ---
(In reply to Archie Cobbs from comment #14)
> > So I'd say that there is another subtle problem here, which is that there really
> > should be two separate configuration variables for configuring (a) the proxy
> > timeout for regular connections and (b) the proxy timeout for websocket
> > connections.
>
> Any thoughts on this question?
>
> Right now the only way to have different timeout values for normal vs.
> websockets
> is to say "ProxyWebsocketFallbackToProxyHttp off" which seems like the wrong
> tool.


You can configure an individual timeout per wss:// backend worker. Setting it
to an arbitrary high number should give you what you need.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #16 from Ruediger Pluem <rpluem@apache.org> ---
Proposed for backport as https://github.com/apache/httpd/pull/295

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #17 from Archie Cobbs <archie.cobbs@gmail.com> ---
> You can configure an individual timeout per wss:// backend worker.
> Setting it to an arbitrary high number should give you what you need.

Thanks - I didn't realize that.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

--- Comment #18 from Ruediger Pluem <rpluem@apache.org> ---
(In reply to Ruediger Pluem from comment #16)
> Proposed for backport as https://github.com/apache/httpd/pull/295

Backported as r1898559. Will be part of the next release.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65886] Regression: Reverse proxy with websocket forcing 60 second timeout regardless of config [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=65886

Yann Ylavic <ylavic.dev@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org