Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Varnish>
  3. Bugs
#1754: n_objecthead increase just before oomkill
varnish-bugs at varnish-cache
Jun 29, 2015, 5:24 AM
Post #1 of 11 (2200 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+-------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: trunk
Severity: normal | Keywords:
----------------------+-------------------
In 4.0.3 we've seen on some occasions that systems running with long TTLs
+ LRU to clean up, with storage tuned to size of physical memory, have
been subject to the out of memory handling (oomkiller) in Linux.

A rise in n_objecthead, not correlated with n_object or n_objectcore, was
seen in the hours before on some varnishstat plots.

Assumption that something starts to leak that is not accounted as normal
storage, and the system (tuned just to the edge of storage) goes over the
safe limit after a while.

We don't have the full picture here, making this ticket to have somewhere
to keep it.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Jul 6, 2015, 4:20 AM
Post #2 of 11 (2178 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------
Changes (by lkarsten):

* version: trunk => 4.0.3


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Jul 6, 2015, 4:22 AM
Post #3 of 11 (2171 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------
Changes (by slink):

* cc: nils.goroll@… (added)


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Jul 6, 2015, 4:31 AM
Post #4 of 11 (2174 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by slink):

Out of curiosity, is this on bare metal or virtualized? If yes, please
name the hypervisor.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Jul 6, 2015, 4:31 AM
Post #5 of 11 (2172 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by slink):

Also, following a suspicion: Does this also happen with khugepaged
disabled?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:4>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 4, 2016, 2:31 AM
Post #6 of 11 (1972 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.0.3
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by dmz):

We're seeing a similar issue on 4.1.1 running on bare metal, khugepaged
disabled. High throughput configured for 56G is slowly leaking up to a 90G
commit over a week, fortunately we monitor and usually manage to restart
before OOM.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:5>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 22, 2016, 4:43 AM
Post #7 of 11 (1934 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version:
Severity: normal | Resolution:
Keywords: |
----------------------+------------------
Changes (by lkarsten):

* version: 4.0.3 =>


Comment:

dmz: can you please supply a varnishstat -1 output of 1) stable state, and
2) just before you restart it?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 22, 2016, 4:51 AM
Post #8 of 11 (1934 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version:
Severity: normal | Resolution:
Keywords: |
----------------------+------------------

Comment (by slink):

Also it would be good to know if you could lower the malloc stevedore size
to a safe value (like 1/3 of your available memory) and see if this issue
still arises.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:7>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 22, 2016, 5:26 AM
Post #9 of 11 (1934 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version:
Severity: normal | Resolution:
Keywords: |
----------------------+------------------

Comment (by dmz):

I'm very sorry to have got your hopes up on tracking this one down, if it
exists. We currently suspect our own increase in useage was as simple as
large TTL objects slowly filling the cache, coupled with an overly
optimistic malloc size. We've imposed a maximum TTL, but due to several
recent unscheduled restarts won't know for around another week if it is
now genuinely stable or not (enclosed a graph of memory use showing two
weeks of "leaking" followed by reconfiguration with max TTL, initially
with a much reduced malloc size). I won't update further unless we see
further "leaking".

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:8>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 22, 2016, 5:38 AM
Post #10 of 11 (1935 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version:
Severity: normal | Resolution:
Keywords: |
----------------------+------------------

Comment (by slink):

Just in case this helps, here's some background and a hypothesis: The
malloc stevedore size you configure is a net value, limiting the total sum
of all allocations which varnishd may issue to the malloc implementation,
usually jemalloc. Because varnishd requests allocations of different
sizes, memory freed by expiring objects or lru nuking them may not be
usable, by the malloc implementation, to satisfy new allocation requests,
so the gross amount of memory required may be significantly larger than
the net value configure. My personal ballpark figure from real life
experience is a factor of 2-3.
My hypothesis on what you're seeing is that allocations are satisfied
until your cache is forced to nuke objects, from which point on you'd see
the gross/net ratio rise.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:9>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1754: n_objecthead increase just before oomkill [ In reply to ]
varnish-bugs at varnish-cache
Feb 22, 2016, 6:05 AM
Post #11 of 11 (1934 views)
Permalink
#1754: n_objecthead increase just before oomkill
----------------------+-----------------------
Reporter: lkarsten | Owner:
Type: defect | Status: needinfo
Priority: normal | Milestone:
Component: varnishd | Version:
Severity: normal | Resolution:
Keywords: |
----------------------+-----------------------
Changes (by slink):

* status: new => needinfo


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1754#comment:10>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal