Mailing List Archive: Performance degradation on LRU nuking

Performance degradation on LRU nuking

Sep 1, 2020, 6:52 AM

Post #1 of 4 (506 views)

Hi,

We are seeing performance issues with Varnish’s file storage when the
storage reaches its capacity and Varnish starts LRU nuking. As soon as
varnish begins nuking objects, the number of worker threads(MAIN.threads)
starts increasing until it hits the limit(thread_pool_max) and remains
pinned on the thread_pool_max. This then leads to incoming requests getting
queued.

We are running Varnish with following configuration:

Varnish version: 6.1
Storage type: file storage

Storage size: 150 Gb

Granularity: 4096

Thread_pools: 2

Thread_pool_max: 5000

Nuke_limit: 5000

OS: Centos Linux release 7.5.1804

Can we tune any varnishadm params to optimize the nuking ?

Re: Performance degradation on LRU nuking [ In reply to ]

slink at schokola

Sep 1, 2020, 7:01 AM

Post #2 of 4 (506 views)

Permalink

On 01/09/2020 15:52, Ameya Prabhu wrote:
>
> We are seeing performance issues with Varnish’s file storage when the storage
> reaches its capacity and Varnish starts LRU nuking. As soon as varnish begins
> nuking objects, the number of worker threads(MAIN.threads) starts increasing
> until it hits the limit(thread_pool_max) and remains pinned on the
> thread_pool_max. This then leads to incoming requests getting queued.
>
some 12-18 months back, I made an effort to prototype improvements to the LRU
nuking, but never reached an anything mature enough for a pull request.

That said, in my case I used malloc storage and the issue was high lock
contention. As you mention you are using file storage, I would suspect that IO
performance could be the bottleneck.

There is no magic knob I would be aware of, other than using more RAM and faster IO.

Nils

Re: Performance degradation on LRU nuking [ In reply to ]

aprabhu at squarespace

Sep 1, 2020, 1:29 PM

Post #3 of 4 (506 views)

Permalink

Thanks Nils. What were the shortcomings that you were trying to improve ?
Did any of the releases post v6.1 show any improvement ?

On Tue, Sep 1, 2020 at 10:01 AM Nils Goroll <slink@schokola.de> wrote:

> On 01/09/2020 15:52, Ameya Prabhu wrote:
>
>
> We are seeing performance issues with Varnish’s file storage when the
> storage reaches its capacity and Varnish starts LRU nuking. As soon as
> varnish begins nuking objects, the number of worker threads(MAIN.threads)
> starts increasing until it hits the limit(thread_pool_max) and remains
> pinned on the thread_pool_max. This then leads to incoming requests getting
> queued.
>
> some 12-18 months back, I made an effort to prototype improvements to the
> LRU nuking, but never reached an anything mature enough for a pull request.
>
> That said, in my case I used malloc storage and the issue was high lock
> contention. As you mention you are using file storage, I would suspect that
> IO performance could be the bottleneck.
>
> There is no magic knob I would be aware of, other than using more RAM and
> faster IO.
>
> Nils
>

Re: Performance degradation on LRU nuking [ In reply to ]

slink at schokola

Sep 1, 2020, 10:52 PM

Post #4 of 4 (501 views)

Permalink

On 01/09/2020 22:29, Ameya Prabhu wrote:
> Thanks Nils. What were the shortcomings that you were trying to improve ?

As I wrote before:

in my case I used malloc storage and the issue was high lock contention.

Also I should add that the exp_thread was basically taking a full CPU.

> Did any of the releases post v6.1 show any improvement ?

I am not aware of any work in that area.

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc