Mailing List Archive

high memory usage with malloc and file backends configured
Hi,

There is a large discrepancy between Varnish resident memory reported by top vs reported by varnishstat. Varnish is configured with both malloc and file storage; 20G malloc and 75G file storage.

These are the startup parameters-

VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varnish/varnish_storage.bin,75G"

After running for a few days top is reporting more than twice amount of memory used for varnishd process than varnishstat.

From top-

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07 varnishd

From varnishstat-

SMA.memcache.g_bytes 5.20G 39.65K . 5.20G 5.20G 5.20G
SMA.memcache.g_space 14.80G -39.65K . 14.80G 14.80G 14.80G
SMF.filecache.g_bytes 7.78G 27.97K . 7.78G 7.78G 7.78G
SMF.filecache.g_space 67.22G -27.97K . 67.22G 67.22G 67.22G

This is the relevant part of the VCL regarding storage backend selection-

sub vcl_backend_response {
# define separate cache storage groups
if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)product\-image\.aspx") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} else {
set beresp.storage_hint = "memcache";
set beresp.http.X-Cache-Storage = "memory";
}

Can post entire VCL if needed.

Would think that since varnishstat reports 5.20G RAM used the resident memory should be around 6-7G, 14G seems excessively high. File storage should use minimal resident memory, correct?

Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except std and directors. Using latest 4.1.7

Anything else you need please let me know.
Re: high memory usage with malloc and file backends configured [ In reply to ]
Hi Charles,

So, if I'm reading this right, there's no discrepancy. Varnish will malloc
the full storage (malloc), and will mmap a file the size of the full
storage (file). So even though the storage is not used, it's allocated.

Does it make sense, or did I miss something?

--
Guillaume Quintard

On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <charles@beachcamera.com>
wrote:

> Hi,
>
>
>
> There is a large discrepancy between Varnish resident memory reported by
> top vs reported by varnishstat. Varnish is configured with both malloc and
> file storage; 20G malloc and 75G file storage.
>
>
>
> These are the startup parameters-
>
>
>
> VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/
> varnish/varnish_storage.bin,75G"
>
>
>
> After running for a few days top is reporting more than twice amount of
> memory used for varnishd process than varnishstat.
>
>
>
> From top-
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
>
> 1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07
> varnishd
>
>
>
>
> From varnishstat-
>
>
>
> SMA.memcache.g_bytes
> 5.20G 39.65K . 5.20G 5.20G 5.20G
>
> SMA.memcache.g_space
> 14.80G -39.65K . 14.80G 14.80G
> 14.80G
>
> SMF.filecache.g_bytes
> 7.78G 27.97K . 7.78G 7.78G
> 7.78G
>
> SMF.filecache.g_space
> 67.22G -27.97K . 67.22G 67.22G 67.22G
>
>
>
> This is the relevant part of the VCL regarding storage backend selection-
>
>
>
> sub vcl_backend_response {
>
> # define separate cache storage groups
>
> if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com")
> {
>
> set beresp.storage_hint = "filecache";
>
> set beresp.http.X-Cache-Storage = "disk";
>
> } elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)")
> {
>
> set beresp.storage_hint = "filecache";
>
> set beresp.http.X-Cache-Storage = "disk";
>
> } elsif (bereq.url ~ "(?i)product\-image\.aspx") {
>
> set beresp.storage_hint = "filecache";
>
> set beresp.http.X-Cache-Storage = "disk";
>
> } else {
>
> set beresp.storage_hint = "memcache";
>
> set beresp.http.X-Cache-Storage =
> "memory";
>
> }
>
>
>
> Can post entire VCL if needed.
>
>
>
> Would think that since varnishstat reports 5.20G RAM used the resident
> memory should be around 6-7G, 14G seems excessively high. File storage
> should use minimal resident memory, correct?
>
>
>
> Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except
> std and directors. Using latest 4.1.7
>
>
>
> Anything else you need please let me know.
>
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: high memory usage with malloc and file backends configured [ In reply to ]
Hi Guillaume,

Thank you for replying so quickly. I think i'm misunderstanding what file storage does. My goal is to have some objects stored in memory (malloc) and others stored on disk (file)

From what you're saying file method still uses resident memory for each object, so with my configuration (20G malloc, 75G file) i would need 95G RAM if all storage is used? (without swap being used)

If this is the case i'm curious what the use cases are for file vs only malloc.

On Jul 10, 2017, at 1:55 PM, Guillaume Quintard <guillaume@varnish-software.com<mailto:guillaume@varnish-software.com>> wrote:

Hi Charles,

So, if I'm reading this right, there's no discrepancy. Varnish will malloc the full storage (malloc), and will mmap a file the size of the full storage (file). So even though the storage is not used, it's allocated.

Does it make sense, or did I miss something?

--
Guillaume Quintard

On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <charles@beachcamera.com<mailto:charles@beachcamera.com>> wrote:
Hi,

There is a large discrepancy between Varnish resident memory reported by top vs reported by varnishstat. Varnish is configured with both malloc and file storage; 20G malloc and 75G file storage.

These are the startup parameters-

VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varnish/varnish_storage.bin,75G"

After running for a few days top is reporting more than twice amount of memory used for varnishd process than varnishstat.

From top-

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07 varnishd

From varnishstat-

SMA.memcache.g_bytes 5.20G 39.65K . 5.20G 5.20G 5.20G
SMA.memcache.g_space 14.80G -39.65K . 14.80G 14.80G 14.80G
SMF.filecache.g_bytes 7.78G 27.97K . 7.78G 7.78G 7.78G
SMF.filecache.g_space 67.22G -27.97K . 67.22G 67.22G 67.22G

This is the relevant part of the VCL regarding storage backend selection-

sub vcl_backend_response {
# define separate cache storage groups
if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)product\-image\.aspx") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} else {
set beresp.storage_hint = "memcache";
set beresp.http.X-Cache-Storage = "memory";
}

Can post entire VCL if needed.

Would think that since varnishstat reports 5.20G RAM used the resident memory should be around 6-7G, 14G seems excessively high. File storage should use minimal resident memory, correct?

Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except std and directors. Using latest 4.1.7

Anything else you need please let me know.



_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: high memory usage with malloc and file backends configured [ In reply to ]
You misundestood me, but that's probably my fault :-)

The file used by the file storage will be fully allocated, ie. you;ll have
a 75G file on your disk. Then varnish will mmap it to memory, which means
the kernel will give Varnish a memory space corresponding to the file
content. The trick is that the whole file doesn't need to be in memory,
only the "active" parts are.

What happens is that we let the kernel manage that space, and it will
leverage the unused memory to do so. So, true, it you had 200G of RAM, the
file storage would effectively take 75G, because they would be available.
That's not your case, so the kernel will only use whatever amount is unused.

That being said, outside of storage, Varnish uses roughly 1K per object
stored, that's probably not impacting right now, but that's good to keep in
mind.

--
Guillaume Quintard

On Mon, Jul 10, 2017 at 2:42 PM, Bender, Charles <charles@beachcamera.com>
wrote:

> Hi Guillaume,
>
> Thank you for replying so quickly. I think i'm misunderstanding what file
> storage does. My goal is to have some objects stored in memory (malloc) and
> others stored on disk (file)
>
> From what you're saying file method still uses resident memory for each
> object, so with my configuration (20G malloc, 75G file) i would need 95G
> RAM if all storage is used? (without swap being used)
>
> If this is the case i'm curious what the use cases are for file vs only
> malloc.
>
> On Jul 10, 2017, at 1:55 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
> Hi Charles,
>
> So, if I'm reading this right, there's no discrepancy. Varnish will malloc
> the full storage (malloc), and will mmap a file the size of the full
> storage (file). So even though the storage is not used, it's allocated.
>
> Does it make sense, or did I miss something?
>
> --
> Guillaume Quintard
>
> On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <charles@beachcamera.com>
> wrote:
>
>> Hi,
>>
>>
>>
>> There is a large discrepancy between Varnish resident memory reported by
>> top vs reported by varnishstat. Varnish is configured with both malloc and
>> file storage; 20G malloc and 75G file storage.
>>
>>
>>
>> These are the startup parameters-
>>
>>
>>
>> VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varn
>> ish/varnish_storage.bin,75G"
>>
>>
>>
>> After running for a few days top is reporting more than twice amount of
>> memory used for varnishd process than varnishstat.
>>
>>
>>
>> From top-
>>
>>
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>>
>>
>> 1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07
>> varnishd
>>
>>
>>
>>
>> From varnishstat-
>>
>>
>>
>> SMA.memcache.g_bytes
>> 5.20G 39.65K . 5.20G 5.20G 5.20G
>>
>> SMA.memcache.g_space
>> 14.80G -39.65K . 14.80G 14.80G
>> 14.80G
>>
>> SMF.filecache.g_bytes
>> 7.78G 27.97K . 7.78G 7.78G
>> 7.78G
>>
>> SMF.filecache.g_space
>> 67.22G -27.97K . 67.22G 67.22G 67.22G
>>
>>
>>
>> This is the relevant part of the VCL regarding storage backend selection-
>>
>>
>>
>> sub vcl_backend_response {
>>
>> # define separate cache storage groups
>>
>> if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com")
>> {
>>
>> set beresp.storage_hint = "filecache";
>>
>> set beresp.http.X-Cache-Storage = "disk";
>>
>> } elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)")
>> {
>>
>> set beresp.storage_hint = "filecache";
>>
>> set beresp.http.X-Cache-Storage = "disk";
>>
>> } elsif (bereq.url ~ "(?i)product\-image\.aspx") {
>>
>> set beresp.storage_hint = "filecache";
>>
>> set beresp.http.X-Cache-Storage = "disk";
>>
>> } else {
>>
>> set beresp.storage_hint = "memcache";
>>
>> set beresp.http.X-Cache-Storage =
>> "memory";
>>
>> }
>>
>>
>>
>> Can post entire VCL if needed.
>>
>>
>>
>> Would think that since varnishstat reports 5.20G RAM used the resident
>> memory should be around 6-7G, 14G seems excessively high. File storage
>> should use minimal resident memory, correct?
>>
>>
>>
>> Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except
>> std and directors. Using latest 4.1.7
>>
>>
>>
>> Anything else you need please let me know.
>>
>>
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
Re: high memory usage with malloc and file backends configured [ In reply to ]
On Mon, Jul 10, 2017 at 1:55 PM, Guillaume Quintard
<guillaume@varnish-software.com> wrote:
> Hi Charles,
>
> So, if I'm reading this right, there's no discrepancy. Varnish will malloc
> the full storage (malloc), and will mmap a file the size of the full storage
> (file). So even though the storage is not used, it's allocated.
>
> Does it make sense, or did I miss something?

Technically malloc storage makes individual allocations, so only file
storage will pre-allocate the full storage.

Allocations also come with an overhead (house-keeping, alignment
requirements...) not reported by Varnish since we can't technically
tell what's happening under our feet (for instance, jemalloc). Also
storage size is only for... storage. So while we have a rule of thumb
of 1kB of overhead per object, that's not the sole non-storage memory
footprint.

Dridi

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: high memory usage with malloc and file backends configured [ In reply to ]
> From what you're saying file method still uses resident memory for each
> object, so with my configuration (20G malloc, 75G file) i would need 95G RAM
> if all storage is used? (without swap being used)

Not RAM, virtual memory. But yes, this is the idea.

> If this is the case i'm curious what the use cases are for file vs only
> malloc.

Basically, malloc storage relies on the underlying allocator (preferably
jemalloc) to acquire memory, whereas file storage has its own allocator
looking for space inside the designated "area" (the 75GB file that was
mapped in memory). If you over-commit memory, it may be subject to
swap for malloc, and the OS page cache/disk buffer for file.

Depending on your workloads or the shape and distribution of your
responses, one might perform better than the other. You can also have
more than one (like you did) for various reasons. And finally, you may
get more memory from the Transient storage (you didn't paste the
stats) that is used for uncacheable or shortlived objects.

Dridi

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: high memory usage with malloc and file backends configured [ In reply to ]
Thanks so much Guillaume for clearly explaining how the file storage works. Have a good understanding of how both storage engines work so will plan accordingly.

Thanks again for your help and responding so quickly.

Charles

On Jul 10, 2017, at 2:54 PM, Guillaume Quintard <guillaume@varnish-software.com<mailto:guillaume@varnish-software.com>> wrote:

You misundestood me, but that's probably my fault :-)

The file used by the file storage will be fully allocated, ie. you;ll have a 75G file on your disk. Then varnish will mmap it to memory, which means the kernel will give Varnish a memory space corresponding to the file content. The trick is that the whole file doesn't need to be in memory, only the "active" parts are.

What happens is that we let the kernel manage that space, and it will leverage the unused memory to do so. So, true, it you had 200G of RAM, the file storage would effectively take 75G, because they would be available. That's not your case, so the kernel will only use whatever amount is unused.

That being said, outside of storage, Varnish uses roughly 1K per object stored, that's probably not impacting right now, but that's good to keep in mind.

--
Guillaume Quintard

On Mon, Jul 10, 2017 at 2:42 PM, Bender, Charles <charles@beachcamera.com<mailto:charles@beachcamera.com>> wrote:
Hi Guillaume,

Thank you for replying so quickly. I think i'm misunderstanding what file storage does. My goal is to have some objects stored in memory (malloc) and others stored on disk (file)

From what you're saying file method still uses resident memory for each object, so with my configuration (20G malloc, 75G file) i would need 95G RAM if all storage is used? (without swap being used)

If this is the case i'm curious what the use cases are for file vs only malloc.

On Jul 10, 2017, at 1:55 PM, Guillaume Quintard <guillaume@varnish-software.com<mailto:guillaume@varnish-software.com>> wrote:

Hi Charles,

So, if I'm reading this right, there's no discrepancy. Varnish will malloc the full storage (malloc), and will mmap a file the size of the full storage (file). So even though the storage is not used, it's allocated.

Does it make sense, or did I miss something?

--
Guillaume Quintard

On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <charles@beachcamera.com<mailto:charles@beachcamera.com>> wrote:
Hi,

There is a large discrepancy between Varnish resident memory reported by top vs reported by varnishstat. Varnish is configured with both malloc and file storage; 20G malloc and 75G file storage.

These are the startup parameters-

VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varnish/varnish_storage.bin,75G"

After running for a few days top is reporting more than twice amount of memory used for varnishd process than varnishstat.

From top-

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07 varnishd

From varnishstat-

SMA.memcache.g_bytes 5.20G 39.65K . 5.20G 5.20G 5.20G
SMA.memcache.g_space 14.80G -39.65K . 14.80G 14.80G 14.80G
SMF.filecache.g_bytes 7.78G 27.97K . 7.78G 7.78G 7.78G
SMF.filecache.g_space 67.22G -27.97K . 67.22G 67.22G 67.22G

This is the relevant part of the VCL regarding storage backend selection-

sub vcl_backend_response {
# define separate cache storage groups
if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} elsif (bereq.url ~ "(?i)product\-image\.aspx") {
set beresp.storage_hint = "filecache";
set beresp.http.X-Cache-Storage = "disk";
} else {
set beresp.storage_hint = "memcache";
set beresp.http.X-Cache-Storage = "memory";
}

Can post entire VCL if needed.

Would think that since varnishstat reports 5.20G RAM used the resident memory should be around 6-7G, 14G seems excessively high. File storage should use minimal resident memory, correct?

Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except std and directors. Using latest 4.1.7

Anything else you need please let me know.



_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc