Mailing List Archive

Hello David,

Have a look at varnishstat ("varnishstat -1 | grep -e g_space -e g_bytes").
When you are passing, varnish is going to consume Transient storage.

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 7:19 AM, FULLER, David <FULLERD@parliament.uk>
wrote:

> We currently have an issue with memory utilisation in Varnish 5.2.1, we
> are only using Reverse Proxy not the caching functionality.
>
>
>
> We are running it in an AWS ECS Docker container, with 1GB of memory
> allocated. Memory increases daily by around 8% until it tops out and site
> connectivity problems occur. Redeploying the container resolves the
> problem and the cycle starts again.
>
>
>
> When Varnish starts we have ‘malloc’ set at 100MB, from my understanding
> this setting is only relevant if caching is being used, which in our case
> it isn’t.
>
>
>
> Has anyone seen a similar problem?
>
>
>
> Thanks
>
>
> UK Parliament Disclaimer: This e-mail is confidential to the intended
> recipient. If you have received it in error, please notify the sender and
> delete it from your system. Any unauthorised use, disclosure, or copying is
> not permitted. This e-mail has been checked for viruses, but no liability
> is accepted for any damage caused by any virus transmitted by this e-mail.
> This e-mail address is not secure, is not encrypted and should not be used
> for sensitive data.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>

Let's keep the mailing list in CC :-)

http://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage

You also have Reza's post:
https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage

Finally, memory is will also be consumed by workspaces (one per thread).

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 8:32 AM, FULLER, David <FULLERD@parliament.uk>
wrote:

> Hi Guillaume,
>
>
>
> Thanks for the response, I’ve run the command you’ve suggested and get the
> following:
>
>
>
> / # varnishstat -1 | grep -e g_space -e g_bytes
>
> SMA.s0.g_bytes 0 . Bytes
> outstanding
>
> SMA.s0.g_space 104857600 . Bytes
> available
>
> SMA.Transient.g_bytes 0 . Bytes
> outstanding
>
> SMA.Transient.g_space 0 . Bytes
> available
>
>
>
> The Varnish container was redeployed this afternoon and currently shows
> memory utilisation around 3% so probably not illustrating the problem very
> well right now.
>
>
>
> Is there a way to limit the amount of transient storage and clear when hit
> without effecting performance? Given that we aren’t caching are there any
> other settings we should look at to improve memory utilisation?
>
>
>
> Kind regards,
>
> David
>
>
>
>
>
>
>
> *From: *Guillaume Quintard <guillaume@varnish-software.com>
> *Date: *Wednesday, 25 July 2018 at 16:00
> *To: *"FULLER, David" <FULLERD@parliament.uk>
> *Cc: *"varnish-misc@varnish-cache.org" <varnish-misc@varnish-cache.org>
> *Subject: *Re: Memory utilisation gradually increasing
>
>
>
> Hello David,
>
>
>
> Have a look at varnishstat ("varnishstat -1 | grep -e g_space -e
> g_bytes"). When you are passing, varnish is going to consume Transient
> storage.
>
>
> --
>
> Guillaume Quintard
>
>
>
> On Wed, Jul 25, 2018 at 7:19 AM, FULLER, David <FULLERD@parliament.uk>
> wrote:
>
> We currently have an issue with memory utilisation in Varnish 5.2.1, we
> are only using Reverse Proxy not the caching functionality.
>
>
>
> We are running it in an AWS ECS Docker container, with 1GB of memory
> allocated. Memory increases daily by around 8% until it tops out and site
> connectivity problems occur. Redeploying the container resolves the
> problem and the cycle starts again.
>
>
>
> When Varnish starts we have ‘malloc’ set at 100MB, from my understanding
> this setting is only relevant if caching is being used, which in our case
> it isn’t.
>
>
>
> Has anyone seen a similar problem?
>
>
>
> Thanks
>
>
>
> UK Parliament Disclaimer: This e-mail is confidential to the intended
> recipient. If you have received it in error, please notify the sender and
> delete it from your system. Any unauthorised use, disclosure, or copying is
> not permitted. This e-mail has been checked for viruses, but no liability
> is accepted for any damage caused by any virus transmitted by this e-mail.
> This e-mail address is not secure, is not encrypted and should not be used
> for sensitive data.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> UK Parliament Disclaimer: This e-mail is confidential to the intended
> recipient. If you have received it in error, please notify the sender and
> delete it from your system. Any unauthorised use, disclosure, or copying is
> not permitted. This e-mail has been checked for viruses, but no liability
> is accepted for any damage caused by any virus transmitted by this e-mail.
> This e-mail address is not secure, is not encrypted and should not be used
> for sensitive data.
>

Thanks for your reply.

Am I correct in thinking that even though we’re not using Varnish for caching, objects are still stored using malloc? We have malloc set at 100MB, with 1GB allocated to the Varnish container. Based on the docs you’ve linked to should we set malloc at 750MB (75% of the container memory) and could this be the cause of the memory problem we’re seeing?

Regards,
David

From: Guillaume Quintard <guillaume@varnish-software.com>
Date: Wednesday, 25 July 2018 at 18:30
To: "FULLER, David" <FULLERD@parliament.uk>, varnish-misc <varnish-misc@varnish-cache.org>
Subject: Re: Memory utilisation gradually increasing

Let's keep the mailing list in CC :-)

http://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage<http://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage>

You also have Reza's post: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage<https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage>

Finally, memory is will also be consumed by workspaces (one per thread).

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 8:32 AM, FULLER, David <FULLERD@parliament.uk<mailto:FULLERD@parliament.uk>> wrote:
Hi Guillaume,

Thanks for the response, I’ve run the command you’ve suggested and get the following:

/ # varnishstat -1 | grep -e g_space -e g_bytes
SMA.s0.g_bytes 0 . Bytes outstanding
SMA.s0.g_space 104857600 . Bytes available
SMA.Transient.g_bytes 0 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available

The Varnish container was redeployed this afternoon and currently shows memory utilisation around 3% so probably not illustrating the problem very well right now.

Is there a way to limit the amount of transient storage and clear when hit without effecting performance? Given that we aren’t caching are there any other settings we should look at to improve memory utilisation?

Kind regards,
David



From: Guillaume Quintard <guillaume@varnish-software.com<mailto:guillaume@varnish-software.com>>
Date: Wednesday, 25 July 2018 at 16:00
To: "FULLER, David" <FULLERD@parliament.uk<mailto:FULLERD@parliament.uk>>
Cc: "varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>" <varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>>
Subject: Re: Memory utilisation gradually increasing

Hello David,

Have a look at varnishstat ("varnishstat -1 | grep -e g_space -e g_bytes"). When you are passing, varnish is going to consume Transient storage.

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 7:19 AM, FULLER, David <FULLERD@parliament.uk<mailto:FULLERD@parliament.uk>> wrote:
We currently have an issue with memory utilisation in Varnish 5.2.1, we are only using Reverse Proxy not the caching functionality.

We are running it in an AWS ECS Docker container, with 1GB of memory allocated. Memory increases daily by around 8% until it tops out and site connectivity problems occur. Redeploying the container resolves the problem and the cycle starts again.

When Varnish starts we have ‘malloc’ set at 100MB, from my understanding this setting is only relevant if caching is being used, which in our case it isn’t.

Has anyone seen a similar problem?

Thanks

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

On Thu, Jul 26, 2018 at 5:05 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks for your reply.
>
> Am I correct in thinking that even though we’re not using Varnish for
> caching, objects are still stored using malloc? We have malloc set at
> 100MB, with 1GB allocated to the Varnish container. Based on the docs
> you’ve linked to should we set malloc at 750MB (75% of the container memory)
> and could this be the cause of the memory problem we’re seeing?

Are you reloading VCLs in your container? Loaded VCLs contribute to
the memory footprint but that's usually negligible. However we've seen
setups where cron jobs or similar means would schedule reloads on a
regular basis whether or not the VCL actually changed on disk, leading
to thousands of loaded VCL. And that could look like a leak with the
memory footprint gradually increasing.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

Thank you for your response, we do not have any cron jobs or schedules set up. Is there a way to check the number of VCLs currently loaded?

thanks

?On 26/07/2018, 16:26, "Dridi Boukelmoune" <dridi@varni.sh> wrote:

On Thu, Jul 26, 2018 at 5:05 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks for your reply.
>
> Am I correct in thinking that even though we’re not using Varnish for
> caching, objects are still stored using malloc? We have malloc set at
> 100MB, with 1GB allocated to the Varnish container. Based on the docs
> you’ve linked to should we set malloc at 750MB (75% of the container memory)
> and could this be the cause of the memory problem we’re seeing?

Are you reloading VCLs in your container? Loaded VCLs contribute to
the memory footprint but that's usually negligible. However we've seen
setups where cron jobs or similar means would schedule reloads on a
regular basis whether or not the VCL actually changed on disk, leading
to thousands of loaded VCL. And that could look like a leak with the
memory footprint gradually increasing.

Dridi



UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Fri, Jul 27, 2018 at 3:04 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Hi Dridi,
>
> Thank you for your response, we do not have any cron jobs or schedules set up. Is there a way to check the number of VCLs currently loaded?

Something like varnishadm vcl.list | wc -l
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks, we have 1 active VCL and around 35 available (of these 3 are auto/warm and the remainder cold/cold). The containers running Varnish had to be rebuilt a couple of hours ago, so I'll check varnishadm again on Monday to see whether the numbers have increased. Are the cold/cold VCLs considered loaded in terms of memory usage?


?On 27/07/2018, 14:36, "Dridi Boukelmoune" <dridi@varni.sh> wrote:

On Fri, Jul 27, 2018 at 3:04 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Hi Dridi,
>
> Thank you for your response, we do not have any cron jobs or schedules set up. Is there a way to check the number of VCLs currently loaded?

Something like varnishadm vcl.list | wc -l



UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Fri, Jul 27, 2018 at 4:41 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks, we have 1 active VCL and around 35 available (of these 3 are auto/warm and the remainder cold/cold). The containers running Varnish had to be rebuilt a couple of hours ago, so I'll check varnishadm again on Monday to see whether the numbers have increased. Are the cold/cold VCLs considered loaded in terms of memory usage?

Yes, cold VCLs are considered loaded in terms of memory usage,
but in a cold state.

It means that they have a lower footprint, but an overhead still
exists. To lower the footprint, varnishd and (well-behaved) VMODs
release any resources that can be acquired again once the VCL is
warmed up prior its use. Just telling varnish to `vcl.use` a cold VCL
will automatically go through the warm up phase, and set the VCL as
active once it reaches the warm state.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks Dridi,

After further investigation I found that we do have a python/cron job running that checks for backend changes and if so does a vcl.load. This has resulted in a growing number of VCLs being loaded, as you suspected. After a redeploying the Varnish container this morning and then monitoring varnishadm we have gone from around 30 VCLs loaded to over 200 in a few hours. Is there a way to limit the number of VCLs that can be loaded, with older ones being dropped as new ones are loaded?



?On 28/07/2018, 18:57, "Dridi Boukelmoune" <dridi@varni.sh> wrote:

On Fri, Jul 27, 2018 at 4:41 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks, we have 1 active VCL and around 35 available (of these 3 are auto/warm and the remainder cold/cold). The containers running Varnish had to be rebuilt a couple of hours ago, so I'll check varnishadm again on Monday to see whether the numbers have increased. Are the cold/cold VCLs considered loaded in terms of memory usage?

Yes, cold VCLs are considered loaded in terms of memory usage,
but in a cold state.

It means that they have a lower footprint, but an overhead still
exists. To lower the footprint, varnishd and (well-behaved) VMODs
release any resources that can be acquired again once the VCL is
warmed up prior its use. Just telling varnish to `vcl.use` a cold VCL
will automatically go through the warm up phase, and set the VCL as
active once it reaches the warm state.

Dridi



UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Mon, Jul 30, 2018 at 2:50 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks Dridi,
>
> After further investigation I found that we do have a python/cron job running that checks for backend changes and if so does a vcl.load. This has resulted in a growing number of VCLs being loaded, as you suspected. After a redeploying the Varnish container this morning and then monitoring varnishadm we have gone from around 30 VCLs loaded to over 200 in a few hours. Is there a way to limit the number of VCLs that can be loaded, with older ones being dropped as new ones are loaded?

For Varnish 6.0 we released a new varnishreload script, see its usage:

https://github.com/varnishcache/pkg-varnish-cache/blob/0ad2f22629c4a368959c423a19e352c9c6c79682/systemd/varnishreload#L46-L74

The main goals were to unify the reload script on all the platforms we
support and simplify the logic by only supporting the privileged
"service manager" use case. As a result it was also easy to add the -m
option to discard old reloads if have more than "maximum" reload VCLs.

In other words, it's up to the python script scheduled by your cron
job to keep track of reloads and discard older ones according to your
policy.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

That’s great, thanks. Does the first VCL listed (labelled boot in column 4 – ‘available cold/cold 0 boot’) need to stay loaded and is there a recommended amount of VCLs to keep loaded?
From: Dridi Boukelmoune <dridi@varni.sh>
Date: Tuesday, 31 July 2018 at 10:03
To: "FULLER, David" <FULLERD@parliament.uk>
Cc: Guillaume Quintard <guillaume@varnish-software.com>, varnish-misc <varnish-misc@varnish-cache.org>
Subject: Re: Memory utilisation gradually increasing

On Mon, Jul 30, 2018 at 2:50 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> Thanks Dridi,
>
> After further investigation I found that we do have a python/cron job running that checks for backend changes and if so does a vcl.load. This has resulted in a growing number of VCLs being loaded, as you suspected. After a redeploying the Varnish container this morning and then monitoring varnishadm we have gone from around 30 VCLs loaded to over 200 in a few hours. Is there a way to limit the number of VCLs that can be loaded, with older ones being dropped as new ones are loaded?

For Varnish 6.0 we released a new varnishreload script, see its usage:

https://github.com/varnishcache/pkg-varnish-cache/blob/0ad2f22629c4a368959c423a19e352c9c6c79682/systemd/varnishreload#L46-L74<https://github.com/varnishcache/pkg-varnish-cache/blob/0ad2f22629c4a368959c423a19e352c9c6c79682/systemd/varnishreload#L46-L74>

The main goals were to unify the reload script on all the platforms we
support and simplify the logic by only supporting the privileged
"service manager" use case. As a result it was also easy to add the -m
option to discard old reloads if have more than "maximum" reload VCLs.

In other words, it's up to the python script scheduled by your cron
job to keep track of reloads and discard older ones according to your
policy.

Dridi

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

On Wed, Aug 1, 2018 at 1:05 PM, FULLER, David <FULLERD@parliament.uk> wrote:
> That’s great, thanks. Does the first VCL listed (labelled boot in column 4
> – ‘available cold/cold 0 boot’) need to stay loaded and is there
> a recommended amount of VCLs to keep loaded?

There's nothing special about "boot", it's just the VCL name given to
the one loaded at launch time using the -f or -b option.

For the amount of VCLs to keep loaded, I'd say enough to feel
confident about rollbacks. If it takes you 30m to link a problem to a
VCL change, then keep 1h worth of reloads for example. Again, your
policy, your rules :)

Cheers
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc