Mailing List Archive: Varnish over memory allocation

Varnish over memory allocation

AlexWakefield at fastmail

May 24, 2020, 11:26 PM

Post #1 of 8 (1245 views)

Hi all,

I currently have multiple Varnish 6.0.5 (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) instances deployed to production with Varnish modules (0.15.0) installed that consistently go over their memory allocation by several gigabytes.

These instances live on 32GB VMs (Ubuntu 18.04) but are tuned to only use 24GB of memory to allow enough overhead for fragmentation and other processes on the machine. If left alone they grow until OOM killer kicks in and kills them off. Currently they're sitting at 30GB of memory used with ~1,942,200 objects in cache according to MAIN.n_object.

In terms of traffic we serve standard HTTP traffic for several quite large websites but images and other binary objects are not stored in cache.

I understand that per 100,000 objects there is ~100MB of fragmentation/overhead so is this amount of memory over-usage to be expected? Is there any tuning I can do to try to reduce this overhead or is the answer to reduce the amount of objects in memory?

We purge assets via XKey tags that are on our pages and some standard bans via purge. All our bans are lurker friendly though, we only hover ~30-50 active bans at any time. XKey purges are a combination of soft and hard purges.

Any advice would be fantastic! Let me know if there is any further information I can provide.

Regards,
Alex
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

cosimo at streppone

May 25, 2020, 1:03 AM

Post #2 of 8 (1245 views)

On Mon, May 25, 2020, at 08:26, Alex Wakefield wrote:
>
> These instances live on 32GB VMs (Ubuntu 18.04) but are tuned to only
> use 24GB of memory to allow enough overhead for fragmentation and other
> processes on the machine. If left alone they grow until OOM killer
> kicks in and kills them off.

What storage module are you using?

--
Cosimo
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

AlexWakefield at fastmail

May 25, 2020, 1:06 AM

Post #3 of 8 (1245 views)

Whoops, knew I forgot to specify something!

We're using malloc. Command line switch is specifically `-s malloc,24GB`

I think this is what you mean?

Cheers,
Alex

On Mon, 25 May 2020, at 6:03 PM, Cosimo Streppone wrote:
> On Mon, May 25, 2020, at 08:26, Alex Wakefield wrote:
> >
> > These instances live on 32GB VMs (Ubuntu 18.04) but are tuned to only
> > use 24GB of memory to allow enough overhead for fragmentation and other
> > processes on the machine. If left alone they grow until OOM killer
> > kicks in and kills them off.
>
> What storage module are you using?
>
> --
> Cosimo
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

cosimo at streppone

May 25, 2020, 1:20 AM

Post #4 of 8 (1245 views)

On Mon, May 25, 2020, at 10:06, Alex Wakefield wrote:
>
> We're using malloc. Command line switch is specifically `-s malloc,24GB`
> I think this is what you mean?

Yes. Unfortunately (or rather fortunately) I never had problems like those with
malloc storage.

--
Cosimo
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

May 25, 2020, 3:02 AM

Post #5 of 8 (1240 views)

On Mon, May 25, 2020 at 8:07 AM Alex Wakefield
<AlexWakefield@fastmail.com.au> wrote:
>
> Whoops, knew I forgot to specify something!
>
> We're using malloc. Command line switch is specifically `-s malloc,24GB`

The -s option only specifies the storage size (HTTP responses with
some metadata). The rest of Varnish's memory footprint goes on top,
things like loaded VCLs, ongoing VCL transactions, all kinds of data
structures. VMODs like XKey may add their own footprint on top, the
list goes on.

Even on the storage side, if you only declare a malloc storage like
you did, you will get an unlimited Transient storage by default for
short-lived or uncacheable responses.

The only way today to tell a Varnish instance to limit itself to 24GB
(and still on a best-effort basis) is with Varnish Enterprise's memory
governor.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

AlexWakefield at fastmail

May 25, 2020, 5:38 AM

Post #6 of 8 (1240 views)

Hey Dridi,

Thanks for the reply. I originally thought that it might've been transient storage as well however checking that shows that only 287MB of storage has been allocated from it. Its allocated and released 1.06TB though, is that perhaps the cause of the issue?

SMA.s0.g_bytes shows the correct size of 23.99GB allocated which is what I expect however a pmap of the varnish process shows 29977920K used, 6GB over what was set.

Cheers,
Alex

On Mon, 25 May 2020, at 8:02 PM, Dridi Boukelmoune wrote:
> On Mon, May 25, 2020 at 8:07 AM Alex Wakefield
> <AlexWakefield@fastmail.com.au> wrote:
> >
> > Whoops, knew I forgot to specify something!
> >
> > We're using malloc. Command line switch is specifically `-s malloc,24GB`
>
> The -s option only specifies the storage size (HTTP responses with
> some metadata). The rest of Varnish's memory footprint goes on top,
> things like loaded VCLs, ongoing VCL transactions, all kinds of data
> structures. VMODs like XKey may add their own footprint on top, the
> list goes on.
>
> Even on the storage side, if you only declare a malloc storage like
> you did, you will get an unlimited Transient storage by default for
> short-lived or uncacheable responses.
>
> The only way today to tell a Varnish instance to limit itself to 24GB
> (and still on a best-effort basis) is with Varnish Enterprise's memory
> governor.
>
> Dridi
>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

info+varnish at shee

May 25, 2020, 6:06 AM

Post #7 of 8 (1240 views)

Am Montag, den 25.05.2020, 22:38 +1000 schrieb Alex Wakefield:
> Hey Dridi,
>
> Thanks for the reply. I originally thought that it might've been
> transient storage as well however checking that shows that only 287MB
> of storage has been allocated from it. Its allocated and released
> 1.06TB though, is that perhaps the cause of the issue?
>
> SMA.s0.g_bytes shows the correct size of 23.99GB allocated which is
> what I expect however a pmap of the varnish process shows 29977920K
> used, 6GB over what was set.

Whats your /proc/meminfo output?

--
Leon

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: Varnish over memory allocation [ In reply to ]

AlexWakefield at fastmail

May 25, 2020, 6:20 AM

Post #8 of 8 (1240 views)

Hey Leon,

$ cat /proc/meminfo
MemTotal: 32940300 kB
MemFree: 309836 kB
MemAvailable: 2083696 kB
Buffers: 127768 kB
Cached: 1999212 kB
SwapCached: 0 kB
Active: 30551008 kB
Inactive: 1561588 kB
Active(anon): 30067316 kB
Inactive(anon): 304 kB
Active(file): 483692 kB
Inactive(file): 1561284 kB
Unevictable: 82180 kB
Mlocked: 82180 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 3856 kB
Writeback: 0 kB
AnonPages: 30066172 kB
Mapped: 243516 kB
Shmem: 10936 kB
Slab: 272788 kB
SReclaimable: 186140 kB
SUnreclaim: 86648 kB
KernelStack: 6960 kB
PageTables: 66996 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 16470148 kB
Committed_AS: 31298244 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 890860 kB
DirectMap2M: 31614976 kB
DirectMap1G: 1048576 kB

Regards,
Alex

On Mon, 25 May 2020, at 11:06 PM, info+varnish@shee.org wrote:
> Am Montag, den 25.05.2020, 22:38 +1000 schrieb Alex Wakefield:
> > Hey Dridi,
> >
> > Thanks for the reply. I originally thought that it might've been
> > transient storage as well however checking that shows that only 287MB
> > of storage has been allocated from it. Its allocated and released
> > 1.06TB though, is that perhaps the cause of the issue?
> >
> > SMA.s0.g_bytes shows the correct size of 23.99GB allocated which is
> > what I expect however a pmap of the varnish process shows 29977920K
> > used, 6GB over what was set.
>
> Whats your /proc/meminfo output?
>
> --
> Leon
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc