All,
We are running HTTPD as a proxy to a backend Tomcat application.
OS = RHEL 7.7
HTTPD:
Server version: Apache/2.4.34 (Red Hat)
Server built: Apr 3 2019 12:07:34
Server's Module Magic Number: 20120211:79
Server loaded: APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt/rh/httpd24/root/etc/httpd"
-D SUEXEC_BIN="/opt/rh/httpd24/root/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/opt/rh/httpd24/root/var/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Tomcat:
Server version: Apache Tomcat/8.5.40
Server built: Apr 10 2019 14:31:19 UTC
Server number: 8.5.40.0
OS Name: Linux
OS Version: 3.10.0-693.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_162-b12
JVM Vendor: Oracle Corporation
What we are seeing on the proxy side are 2 'proxy errors' (scrubbed):
[Tue Jan 05 12:07:19.664861 2021] [proxy:error] [pid 74638:tid
140600384845568] [client 10.10.100.10:64959] AH00898: Error reading from
remote server returned by /path/to/uri
[Tue Jan 05 12:07:19.664745 2021] [proxy_http:error] [pid 74638:tid
140600384845568] (70007)The timeout specified has expired: [client
10.10.100.10:64959] AH01102: error reading status line from remote server
node1.example.com:8002
The client listed in these (10.10.100.10) is actually an A10 that is
handling the VIP for the URL.
The configurations we have set on the proxy are:
<Proxy balancer://nodes>
BalancerMember https://node1.example.com:8002 route=node1
keepalive=On ping=3 ttl=90 timeout=60
BalancerMember https://node2.example.com:8002 route=node2
keepalive=On ping=3 ttl=90 timeout=60
ProxySet lbmethod=bybusyness
</Proxy>
On the Tomcat side we have:
<snip>
keepAliveTimeout="90000"
connectionTimeout="60000"
<snip>
My understanding is that the "ttl" on the HTTPD side corresponds to the
"keepAliveTimeout" on the Tomcat side and the "timeout" on the HTTPD side
corresponds to the "connectionTimeout" on the Tomcat side.
I am looking for some guidance into how we can successfully pinpoint where
the issue lies. Is it the fact that the application is simply not
responding to the request in the allotted configured settings? We can
certainly test that option but being a PRD environment we'll need to
schedule the process. Therefore, I'd like to get some thoughts on what we
can do to pinpoint exactly what is going on here.
Thanks in advance,
HB
We are running HTTPD as a proxy to a backend Tomcat application.
OS = RHEL 7.7
HTTPD:
Server version: Apache/2.4.34 (Red Hat)
Server built: Apr 3 2019 12:07:34
Server's Module Magic Number: 20120211:79
Server loaded: APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt/rh/httpd24/root/etc/httpd"
-D SUEXEC_BIN="/opt/rh/httpd24/root/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/opt/rh/httpd24/root/var/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Tomcat:
Server version: Apache Tomcat/8.5.40
Server built: Apr 10 2019 14:31:19 UTC
Server number: 8.5.40.0
OS Name: Linux
OS Version: 3.10.0-693.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_162-b12
JVM Vendor: Oracle Corporation
What we are seeing on the proxy side are 2 'proxy errors' (scrubbed):
[Tue Jan 05 12:07:19.664861 2021] [proxy:error] [pid 74638:tid
140600384845568] [client 10.10.100.10:64959] AH00898: Error reading from
remote server returned by /path/to/uri
[Tue Jan 05 12:07:19.664745 2021] [proxy_http:error] [pid 74638:tid
140600384845568] (70007)The timeout specified has expired: [client
10.10.100.10:64959] AH01102: error reading status line from remote server
node1.example.com:8002
The client listed in these (10.10.100.10) is actually an A10 that is
handling the VIP for the URL.
The configurations we have set on the proxy are:
<Proxy balancer://nodes>
BalancerMember https://node1.example.com:8002 route=node1
keepalive=On ping=3 ttl=90 timeout=60
BalancerMember https://node2.example.com:8002 route=node2
keepalive=On ping=3 ttl=90 timeout=60
ProxySet lbmethod=bybusyness
</Proxy>
On the Tomcat side we have:
<snip>
keepAliveTimeout="90000"
connectionTimeout="60000"
<snip>
My understanding is that the "ttl" on the HTTPD side corresponds to the
"keepAliveTimeout" on the Tomcat side and the "timeout" on the HTTPD side
corresponds to the "connectionTimeout" on the Tomcat side.
I am looking for some guidance into how we can successfully pinpoint where
the issue lies. Is it the fact that the application is simply not
responding to the request in the allotted configured settings? We can
certainly test that option but being a PRD environment we'll need to
schedule the process. Therefore, I'd like to get some thoughts on what we
can do to pinpoint exactly what is going on here.
Thanks in advance,
HB