Mailing List Archive: Connection resets / timout with Varnish 6.0 and HTTP/2

Connection resets / timout with Varnish 6.0 and HTTP/2

Jul 4, 2018, 1:43 AM

Post #1 of 5 (1397 views)

Hello everbody,

finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So far everything works fine, also HTTP/2 Support.
But as soon as we are receiving some more requests (approx. > 500req/s) varnish does not deliver all requests anymore. The syslog is flooded with messages like:

Jul 4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.289397 [ 5502] {backend-connect}: Connection refused
Jul 4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.290213 [ 5502] {backend-connect}: Connection refused
...

Jul 4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.837457 [ 5504] xxx.xxx.xxx.xxx:5835 :0 1469:1470 backend connect timeout
Jul 4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.851809 [ 5510] xxx.xxx.xxx.xxx:54396 :0 7190:7191 backend connect timeout
...

Jul 4 07:57:31 cache1-vm hitch[5480]: 20180704T075731.488096 [ 5510] {backend} Socket error: Connection reset by peer

We had some similar problems in the past with HTTPS. We could solve them by adding:

net.ipv4.ip_local_port_range = 4096 64999
net.ipv4.tcp_tw_reuse = 1

to /etc/sysctl.conf But this seems to be not the problem here.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf everything is working normally.

Are there any limitations regarding HTTP/2 within varnish?

Our config:

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -a '[::1]:6086,PROXY' -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p thread_pools=2 -p thread_pool_min=200 -p thread_pool_max=5000 -p timeout_idle=100 -p send_timeout=3600 -p feature=+http2 -s malloc,2g
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

Also DefaultLimitNOFILE is set to 250000 in /etc/systemd/system.conf.

Hitch config:

# Listening
frontend = "[*]:443"
ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
# TLS 1.0 wird gebraucht, damit alte Server (intranet-srv) eine Verbindung aufbauen k?nnen
tls-protos = TLSv1.1 TLSv1.2 TLSv1.0

# Send traffic to the Varnish backend using the PROXY protocol
backend = "[::1]:6086"
write-proxy-v2 = on
alpn-protos = "h2,http/1.1"

# Number of processes
workers = 8

We already searched on the varnish github account for similar problem, but did not found anything...

Thanks,
Thomas

________________________________
RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth

Re: Connection resets / timout with Varnish 6.0 and HTTP/2 [ In reply to ]

dridi at varni

Jul 6, 2018, 5:34 AM

Post #2 of 5 (1392 views)

Permalink

Hello Thomas,

On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
> Hello everbody,
>
> finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So far
> everything works fine, also HTTP/2 Support.
>
<snip>
>
> We had some similar problems in the past with HTTPS. We could solve them by
> adding:
>
> net.ipv4.ip_local_port_range = 4096 64999
> net.ipv4.tcp_tw_reuse = 1
>
> to /etc/sysctl.conf But this seems to be not the problem here.

Thanks again for reporting both your problem and solution.

> As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf everything
> is working normally.
>
> Are there any limitations regarding HTTP/2 within varnish?
>
<snip>
>
> We already searched on the varnish github account for similar problem, but
> did not found anything...

You may have run into a known worker thread leak [1] that could be
caused by either misbehaving browsers or bugs in our h2 stack. Leaking
too many threads may put your varnish in a deadlock [2] situation that
we have yet to fix (but much less likely in the absence of the
aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is
work in progress towards a 6.0.1 release? You may still run into a
crash but I'm waiting for a test case to be written before resuming
the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm
confusing you with someone else I'm pretty sure you were referring
to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

AW: Connection resets / timout with Varnish 6.0 and HTTP/2 [ In reply to ]

t.winkelmann at ffh

Jul 12, 2018, 4:00 AM

Post #3 of 5 (1383 views)

Permalink

Hello Dridi,

we just managed to clone your git tree and successfully build varnish from source. But the vmods are not found...

>Message from VCC-compiler:
>Could not load VMOD vsthrottle
> File name: libvmod_vsthrottle.so
> dlerror: libvarnishapi.so.1: cannot open shared object file: No such file or directory

They were compiled to /usr/local/lib/varnish/vmods/, but varnish seems to search in another directory?

As soon as I have running vsthrottle I can put some traffic to the server...

Thanks,
Thomas

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Freitag, 6. Juli 2018 14:34
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc@varnish-cache.org
Betreff: Re: Connection resets / timout with Varnish 6.0 and HTTP/2

Hello Thomas,

On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
> Hello everbody,
>
> finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
> far everything works fine, also HTTP/2 Support.
>
<snip>
>
> We had some similar problems in the past with HTTPS. We could solve
> them by
> adding:
>
> net.ipv4.ip_local_port_range = 4096 64999 net.ipv4.tcp_tw_reuse = 1
>
> to /etc/sysctl.conf But this seems to be not the problem here.

Thanks again for reporting both your problem and solution.

> As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
> everything is working normally.
>
> Are there any limitations regarding HTTP/2 within varnish?
>
<snip>
>
> We already searched on the varnish github account for similar problem,
> but did not found anything...

You may have run into a known worker thread leak [1] that could be caused by either misbehaving browsers or bugs in our h2 stack. Leaking too many threads may put your varnish in a deadlock [2] situation that we have yet to fix (but much less likely in the absence of the aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is work in progress towards a 6.0.1 release? You may still run into a crash but I'm waiting for a test case to be written before resuming the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm confusing you with someone else I'm pretty sure you were referring to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

AW: Connection resets / timout with Varnish 6.0 and HTTP/2 [ In reply to ]

t.winkelmann at ffh

Jul 12, 2018, 4:24 AM

Post #4 of 5 (1383 views)

Permalink

We could fix the problem by running ldconfig.

See: https://ma.ttias.be/varnish-varnishhistvarnishtop-error-while-loading-shared-libraries-libvarnishapi-so-1-cannot-open-shared-object-file/

In which way do you prefer to reveice the crash reports? Or which log files are helpful for you to get the h2 bug fixed?

-----Ursprüngliche Nachricht-----
Von: Winkelmann, Thomas (RADIO TELE FFH - Online)
Gesendet: Donnerstag, 12. Juli 2018 13:00
An: 'Dridi Boukelmoune'
Cc: 'varnish-misc@varnish-cache.org'
Betreff: AW: Connection resets / timout with Varnish 6.0 and HTTP/2

Hello Dridi,

we just managed to clone your git tree and successfully build varnish from source. But the vmods are not found...

>Message from VCC-compiler:
>Could not load VMOD vsthrottle
> File name: libvmod_vsthrottle.so
> dlerror: libvarnishapi.so.1: cannot open shared object file: No
>such file or directory

They were compiled to /usr/local/lib/varnish/vmods/, but varnish seems to search in another directory?

As soon as I have running vsthrottle I can put some traffic to the server...

Thanks,
Thomas

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Freitag, 6. Juli 2018 14:34
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc@varnish-cache.org
Betreff: Re: Connection resets / timout with Varnish 6.0 and HTTP/2

Hello Thomas,

On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
> Hello everbody,
>
> finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
> far everything works fine, also HTTP/2 Support.
>
<snip>
>
> We had some similar problems in the past with HTTPS. We could solve
> them by
> adding:
>
> net.ipv4.ip_local_port_range = 4096 64999 net.ipv4.tcp_tw_reuse = 1
>
> to /etc/sysctl.conf But this seems to be not the problem here.

Thanks again for reporting both your problem and solution.

> As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
> everything is working normally.
>
> Are there any limitations regarding HTTP/2 within varnish?
>
<snip>
>
> We already searched on the varnish github account for similar problem,
> but did not found anything...

You may have run into a known worker thread leak [1] that could be caused by either misbehaving browsers or bugs in our h2 stack. Leaking too many threads may put your varnish in a deadlock [2] situation that we have yet to fix (but much less likely in the absence of the aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is work in progress towards a 6.0.1 release? You may still run into a crash but I'm waiting for a test case to be written before resuming the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm confusing you with someone else I'm pretty sure you were referring to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Connection resets / timout with Varnish 6.0 and HTTP/2 [ In reply to ]

guillaume at varnish-software

Jul 12, 2018, 5:21 AM

Post #5 of 5 (1383 views)

Permalink

Hi Thomas,

You can use the vcl_path parameter to adjust where varnish will look for
the vmod files.

Cheers,

--
Guillaume Quintard

On Thu, Jul 12, 2018 at 4:24 AM, Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:

> We could fix the problem by running ldconfig.
>
> See: https://ma.ttias.be/varnish-varnishhistvarnishtop-error-
> while-loading-shared-libraries-libvarnishapi-so-1-
> cannot-open-shared-object-file/
>
> In which way do you prefer to reveice the crash reports? Or which log
> files are helpful for you to get the h2 bug fixed?
>
> -----Ursprüngliche Nachricht-----
> Von: Winkelmann, Thomas (RADIO TELE FFH - Online)
> Gesendet: Donnerstag, 12. Juli 2018 13:00
> An: 'Dridi Boukelmoune'
> Cc: 'varnish-misc@varnish-cache.org'
> Betreff: AW: Connection resets / timout with Varnish 6.0 and HTTP/2
>
> Hello Dridi,
>
> we just managed to clone your git tree and successfully build varnish from
> source. But the vmods are not found...
>
> >Message from VCC-compiler:
> >Could not load VMOD vsthrottle
> > File name: libvmod_vsthrottle.so
> > dlerror: libvarnishapi.so.1: cannot open shared object file: No
> >such file or directory
>
> They were compiled to /usr/local/lib/varnish/vmods/, but varnish seems to
> search in another directory?
>
> As soon as I have running vsthrottle I can put some traffic to the
> server...
>
> Thanks,
> Thomas
>
> -----Ursprüngliche Nachricht-----
> Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
> Gesendet: Freitag, 6. Juli 2018 14:34
> An: Winkelmann, Thomas (RADIO TELE FFH - Online)
> Cc: varnish-misc@varnish-cache.org
> Betreff: Re: Connection resets / timout with Varnish 6.0 and HTTP/2
>
> Hello Thomas,
>
> On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -
> Online) <t.winkelmann@ffh.de> wrote:
> > Hello everbody,
> >
> > finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
> > far everything works fine, also HTTP/2 Support.
> >
> <snip>
> >
> > We had some similar problems in the past with HTTPS. We could solve
> > them by
> > adding:
> >
> > net.ipv4.ip_local_port_range = 4096 64999 net.ipv4.tcp_tw_reuse = 1
> >
> > to /etc/sysctl.conf But this seems to be not the problem here.
>
> Thanks again for reporting both your problem and solution.
>
> > As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
> > everything is working normally.
> >
> > Are there any limitations regarding HTTP/2 within varnish?
> >
> <snip>
> >
> > We already searched on the varnish github account for similar problem,
> > but did not found anything...
>
> You may have run into a known worker thread leak [1] that could be caused
> by either misbehaving browsers or bugs in our h2 stack. Leaking too many
> threads may put your varnish in a deadlock [2] situation that we have yet
> to fix (but much less likely in the absence of the aforementioned leak).
>
> Could you please try building from source my 6.0 branch [3] that is work
> in progress towards a 6.0.1 release? You may still run into a crash but I'm
> waiting for a test case to be written before resuming the back-porting
> effort.
>
> You will likely need to rebuild your modules too, because unless I'm
> confusing you with someone else I'm pretty sure you were referring to our
> varnish-modules [5] collection of VMODs.
>
> Thanks,
> Dridi
>
> [1] https://github.com/varnishcache/varnish-cache/issues/2623
> [2] https://github.com/varnishcache/varnish-cache/issues/2418
> [3] https://github.com/dridi/varnish-cache/tree/6.0
> [4] https://github.com/varnishcache/varnish-cache/
> issues/2572#issuecomment-402075064
> [5] https://github.com/varnish/varnish-modules
>
> RADIO / TELE FFH GmbH & Co. Betriebs-KG
> FFH-Platz 1, 61111 Bad Vilbel
> HRA - Nr. 26092 Frankfurt/Main
> USt.IdNr. DE 112152620
> Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>