Mailing List Archive

On Wed, Sep 19, 2018 at 2:32 PM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Hello,
>
> just wanted to test Varnish 6.1 regarding H2 support, but I’m unable to compile the vmods on top of it.
>
> Is there a compatible version varnish-modules already available for 6.1?

We haven't added support for 6.1 yet, but we have a pull request to review.

> Or: Are there big differences between 6.0.1 and 6.1 regarding H/2?

The 6.0.1 release should contain all the latest h2 bug fixes. Please
note that the 6.0 branch is a long term support branch and 6.1 is not
so ignore varnish-modules for now, please be aware of this before
choosing one.

Also please note that there are a few remaining known h2 bugs, but
stability has greatly improved since the 6.0.0 release.

Cheers
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems:

https://github.com/varnishcache/varnish-cache/issues/2431
https://github.com/varnishcache/varnish-cache/issues/2418

Let us know if we can help to solve these problems.

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Mittwoch, 19. September 2018 18:59
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc
Betreff: Re: Varnish VMods for Version 6.1?

On Wed, Sep 19, 2018 at 2:32 PM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Hello,
>
> just wanted to test Varnish 6.1 regarding H2 support, but I’m unable to compile the vmods on top of it.
>
> Is there a compatible version varnish-modules already available for 6.1?

We haven't added support for 6.1 yet, but we have a pull request to review.

> Or: Are there big differences between 6.0.1 and 6.1 regarding H/2?

The 6.0.1 release should contain all the latest h2 bug fixes. Please
note that the 6.0 branch is a long term support branch and 6.1 is not
so ignore varnish-modules for now, please be aware of this before
choosing one.

Also please note that there are a few remaining known h2 bugs, but
stability has greatly improved since the 6.0.0 release.

Cheers

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Thu, Sep 20, 2018 at 10:22 AM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems:
>
> https://github.com/varnishcache/varnish-cache/issues/2431
> https://github.com/varnishcache/varnish-cache/issues/2418

2418 may very well be responsible. You could try working around this
by adding more worker threads to your pools. That would postpone
the deadlock to higher concurrent traffic. And possibly increase the
thread reserve as well.

> Let us know if we can help to solve these problems.

I'm afraid we already have a consistent reproducer, so I guess all
that remains is that we (Dag) roll up our (his) sleeves and do
something about it. Joke aside, we've looked at it then but it's not
an easy task.

In my previous message I said:

> so ignore varnish-modules for now, please be aware of this before
> choosing one.

What I really meant was:

> so ignor[ing] varnish-modules for now, please be aware of this before
> choosing one.

In other words, once varnish-modules supports 6.1, please consider
that 6.0 is an LTS series before you pick your target. Not that there
is anything wrong about picking 6.1 if you are planning to periodically
have major upgrades (I wouldn't recommend staying on an
unmaintained branch once it reaches EOL).

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks again! We already use

-p thread_pools=2
-p thread_pool_min=200
-p thread_pool_max=5000

I think this results already in a high number of threads.

So it's probably the best to wait until you have fixed the bug.

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Donnerstag, 20. September 2018 11:19
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc
Betreff: Re: Varnish VMods for Version 6.1?

On Thu, Sep 20, 2018 at 10:22 AM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems:
>
> https://github.com/varnishcache/varnish-cache/issues/2431
> https://github.com/varnishcache/varnish-cache/issues/2418

2418 may very well be responsible. You could try working around this
by adding more worker threads to your pools. That would postpone
the deadlock to higher concurrent traffic. And possibly increase the
thread reserve as well.

> Let us know if we can help to solve these problems.

I'm afraid we already have a consistent reproducer, so I guess all
that remains is that we (Dag) roll up our (his) sleeves and do
something about it. Joke aside, we've looked at it then but it's not
an easy task.

In my previous message I said:

> so ignore varnish-modules for now, please be aware of this before
> choosing one.

What I really meant was:

> so ignor[ing] varnish-modules for now, please be aware of this before
> choosing one.

In other words, once varnish-modules supports 6.1, please consider
that 6.0 is an LTS series before you pick your target. Not that there
is anything wrong about picking 6.1 if you are planning to periodically
have major upgrades (I wouldn't recommend staying on an
unmaintained branch once it reaches EOL).

Dridi

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Thu, Sep 20, 2018 at 11:39 AM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Thanks again! We already use
>
> -p thread_pools=2
> -p thread_pool_min=200
> -p thread_pool_max=5000
>
> I think this results already in a high number of threads.
>
> So it's probably the best to wait until you have fixed the bug.

Did you try increasing thread_pool_reserve as well? The default is 5%
of thread_pool_min and the maximum is 95%.

To stay with the same maximum, you could try something like:

-p thread_pools=2
-p thread_pool_min=2000
-p thread_pool_max=5000
-p thread_pool_reserve=500

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :)

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Donnerstag, 20. September 2018 14:28
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc
Betreff: Re: Varnish VMods for Version 6.1?

On Thu, Sep 20, 2018 at 11:39 AM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Thanks again! We already use
>
> -p thread_pools=2
> -p thread_pool_min=200
> -p thread_pool_max=5000
>
> I think this results already in a high number of threads.
>
> So it's probably the best to wait until you have fixed the bug.

Did you try increasing thread_pool_reserve as well? The default is 5%
of thread_pool_min and the maximum is 95%.

To stay with the same maximum, you could try something like:

-p thread_pools=2
-p thread_pool_min=2000
-p thread_pool_max=5000
-p thread_pool_reserve=500

Dridi

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Thu, Sep 20, 2018 at 3:51 PM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :)

I guess now would be a good time to bring back h2load. If you increase
thread_pool_min to 2000 (a value that shouldn't be unreasonable) that
brings the thread_pool_reserve to 400, so even if I didn't bump it further
to 500 it'd be in the same ballpark. And if that is enough to bring Varnish
to a crawl we may have a deeper problem here.

As a general rule of thumb we recommend a thread_pool_min sized for
high (for a given definition of high, let's say above average) traffic so that
sudden peaks of traffic wouldn't wait too long for threads to be created.

I don't see why that should become a problem with Varnish 6 :-/

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

I again did some testing by settings the params to

-p thread_pools=4
-p thread_pool_min=2000
-p thread_pool_max=5000
-p thread_pool_reserve=95

The document says https://varnish-cache.org/docs/trunk/reference/varnishd.html that thread_pool_reserve is a value from 0 to 95. It's working fine on one of our two servers which have less load (300 req/s) than the primaray one (> 800req/s).

On the primary one you can see decreasing the client.req in varnishstat two seconds after restarting hitch with H/2 enabled. It jumps from 100req/s to 1700req/s from one second to the next.

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:dridi@varni.sh]
Gesendet: Freitag, 21. September 2018 10:56
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-misc
Betreff: Re: Varnish VMods for Version 6.1?

On Thu, Sep 20, 2018 at 3:51 PM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :)

I guess now would be a good time to bring back h2load. If you increase
thread_pool_min to 2000 (a value that shouldn't be unreasonable) that
brings the thread_pool_reserve to 400, so even if I didn't bump it further
to 500 it'd be in the same ballpark. And if that is enough to bring Varnish
to a crawl we may have a deeper problem here.

As a general rule of thumb we recommend a thread_pool_min sized for
high (for a given definition of high, let's say above average) traffic so that
sudden peaks of traffic wouldn't wait too long for threads to be created.

I don't see why that should become a problem with Varnish 6 :-/

Dridi

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Fri, Sep 21, 2018 at 1:50 PM Winkelmann, Thomas (RADIO TELE FFH -
Online) <t.winkelmann@ffh.de> wrote:
>
> I again did some testing by settings the params to
>
> -p thread_pools=4
> -p thread_pool_min=2000
> -p thread_pool_max=5000
> -p thread_pool_reserve=95

Please note that min/max/reserve is per pool.

> The document says https://varnish-cache.org/docs/trunk/reference/varnishd.html that thread_pool_reserve is a value from 0 to 95. It's working fine on one of our two servers which have less load (300 req/s) than the primaray one (> 800req/s).

You really need to read the full description for this one:

Default is 0 to auto-tune (currently 5% of thread_pool_min).
Minimum is 1 otherwise, maximum is 95% of thread_pool_min.

In other words, if you only bump thread_pool_min, you automatically
bump thread_pool_reserve.

In my previous suggestion my mighty arithmetic powers had failed me.
5% of 2000 is obviously 100, so a thread_pool_reserve of 500 was no
longer in the same ballpark.

> On the primary one you can see decreasing the client.req in varnishstat two seconds after restarting hitch with H/2 enabled. It jumps from 100req/s to 1700req/s from one second to the next.

Like I said, until that deadlock is fixed:

That would postpone the deadlock to higher concurrent traffic.

Tweaking the worker pools won't fix the problem, and in that regard h2
traffic is even more peaky (per user) than h1 because browsers will
typically ask resources immediately (where h1 TCP pooling would
prevent too much concurrency) so seeing such peaks is not really
surprising.

Thanks for sharing your experience nevertheless :)

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc