#1738: Large number of ExpKills and nuking issues
----------------------+--------------------
Reporter: coredump | Type: defect
Status: new | Priority: normal
Milestone: | Component: build
Version: unknown | Severity: normal
Keywords: |
----------------------+--------------------
Last Friday we had a problem that may be a bug.
Our setup uses two varnish servers, each one using malloc,65G storage.
Same hardware/config with 96GB total memory. No swap used. There's a load
balancer in front of the varnish servers doing consistent hashing. Using
varnish-trunk revision 6f53e59
One of our servers started increasing number of objects, expires stopped
happening and we had a giant number of lru_nukes (those eventually went to
zero) and worker thread going to the max number. Those are some graphs
showing how it behaved:
https://dl.dropboxusercontent.com/1/view/z9xmpv99p7oi6ws/Captured/3gD2H.png
http://dsl.so/1B8AtVA
Varnishlog showed requests going normally, except with some that had 10k+
ExpKill messages, like this one:
https://gist.github.com/coredump/6710e57aaf3dff7788e1
And those are the params being used at the moment:
https://gist.github.com/coredump/8a1c4d56ed79adb90956
As a result, the load balancers started to get timeouts from that varnish
server.
--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1738>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator
_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
----------------------+--------------------
Reporter: coredump | Type: defect
Status: new | Priority: normal
Milestone: | Component: build
Version: unknown | Severity: normal
Keywords: |
----------------------+--------------------
Last Friday we had a problem that may be a bug.
Our setup uses two varnish servers, each one using malloc,65G storage.
Same hardware/config with 96GB total memory. No swap used. There's a load
balancer in front of the varnish servers doing consistent hashing. Using
varnish-trunk revision 6f53e59
One of our servers started increasing number of objects, expires stopped
happening and we had a giant number of lru_nukes (those eventually went to
zero) and worker thread going to the max number. Those are some graphs
showing how it behaved:
https://dl.dropboxusercontent.com/1/view/z9xmpv99p7oi6ws/Captured/3gD2H.png
http://dsl.so/1B8AtVA
Varnishlog showed requests going normally, except with some that had 10k+
ExpKill messages, like this one:
https://gist.github.com/coredump/6710e57aaf3dff7788e1
And those are the params being used at the moment:
https://gist.github.com/coredump/8a1c4d56ed79adb90956
As a result, the load balancers started to get timeouts from that varnish
server.
--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1738>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator
_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs