Mailing List Archive

Gracefully stopping varnish
Hi there,

I've been looking around and I can't find a documented way of gracefully
shutting down varnishd, and by gracefully I mean tell varnish "stop
accepting connections, but finish what you were doing, then shutdown".

I did find something in the "first varnish design notes" (
https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed to
indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
doesn't seem to work, and TERM, well... terminates but not gracefully.

I also tried using "varnishadm stop", which also doesn't gracefully stops
connection.

Is there anyway to achieve this?

Thanks!
-Hugues
Re: Gracefully stopping varnish [ In reply to ]
Hi,

That's not possible. However, what you really want, I think, is not sending
new requests to Varnish. That's usually done at the loa-bbalancing level.
If your LB use probes, you can tell Varnish to stop honoring them, drain
the connections, then kill it.

--
Guillaume Quintard

On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:

> Hi there,
>
> I've been looking around and I can't find a documented way of gracefully
> shutting down varnishd, and by gracefully I mean tell varnish "stop
> accepting connections, but finish what you were doing, then shutdown".
>
> I did find something in the "first varnish design notes" (
> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed to
> indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
> doesn't seem to work, and TERM, well... terminates but not gracefully.
>
> I also tried using "varnishadm stop", which also doesn't gracefully stops
> connection.
>
> Is there anyway to achieve this?
>
> Thanks!
> -Hugues
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Gracefully stopping varnish [ In reply to ]
Hi,

That is indeed what I want to do. Draining connections at the LB level just
involves more work ;).

For context: my entire stack runs on a kubernetes cluster.

I sometime need to replace a running instance of a server process [be it
nginx apache php-fpm varnish whatever] with a new instance.

Taking Apache as an example, I simply create the new apache instance (a new
pod in kubernetes speak), which immediately starts getting some traffic
(without changing the load balancer configuration at all, a kubernetes
"service" automatically detects new pods), then I gracefully shutdown the
old instance (kubernetes actually automatically tells the pod to shutdown),
by issuing a "apachectl -k graceful-stop" (kubernetes is configured to
issue this command for me), which instructs apache to stop accepting
connections, finish, then shutdown.

It's really great because instead of having to push a new config refusing
probes and reload it, I (/kubernetes) simply gracefully stops apache and
the traffic flows to the new instance. nginx and php-fpm also handle things
this way.

At any rate, thanks for the advice, I will start using probes!

Cheers,
-Hugues



On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Hi,
>
> That's not possible. However, what you really want, I think, is not
> sending new requests to Varnish. That's usually done at the loa-bbalancing
> level. If your LB use probes, you can tell Varnish to stop honoring them,
> drain the connections, then kill it.
>
> --
> Guillaume Quintard
>
> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>
>> Hi there,
>>
>> I've been looking around and I can't find a documented way of gracefully
>> shutting down varnishd, and by gracefully I mean tell varnish "stop
>> accepting connections, but finish what you were doing, then shutdown".
>>
>> I did find something in the "first varnish design notes" (
>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed to
>> indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
>> doesn't seem to work, and TERM, well... terminates but not gracefully.
>>
>> I also tried using "varnishadm stop", which also doesn't gracefully stops
>> connection.
>>
>> Is there anyway to achieve this?
>>
>> Thanks!
>> -Hugues
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
Re: Gracefully stopping varnish [ In reply to ]
Hello,

So, there must be a misconception on my, because to me, just refusing new
connections isn't graceful. If the LB is sending you new connections, you
should honor them.

So, from what I get, k8s stops sending new connections to Varnish, and what
you want is is for Varnish to shutdown once there's no more active
connections, which is, I would argue, kinda different from refusing new
connections. But, let's stop being a pompous jerk obsessed with semantics
and let's try to work on a solution.

From what I get, we can kill varnish as soon as the number of active
connections drops to 0, so your apachectl command would be equal to this in
Varnish language:

if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print
a}'` != 0 ]; then
sleep 1
fi
killall varnishd


There's probably a shorter version ,but that's the gist of it.

Would that do?

--
Guillaume Quintard

On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com> wrote:

> Hi,
>
> That is indeed what I want to do. Draining connections at the LB level
> just involves more work ;).
>
> For context: my entire stack runs on a kubernetes cluster.
>
> I sometime need to replace a running instance of a server process [be it
> nginx apache php-fpm varnish whatever] with a new instance.
>
> Taking Apache as an example, I simply create the new apache instance (a
> new pod in kubernetes speak), which immediately starts getting some traffic
> (without changing the load balancer configuration at all, a kubernetes
> "service" automatically detects new pods), then I gracefully shutdown the
> old instance (kubernetes actually automatically tells the pod to shutdown),
> by issuing a "apachectl -k graceful-stop" (kubernetes is configured to
> issue this command for me), which instructs apache to stop accepting
> connections, finish, then shutdown.
>
> It's really great because instead of having to push a new config refusing
> probes and reload it, I (/kubernetes) simply gracefully stops apache and
> the traffic flows to the new instance. nginx and php-fpm also handle things
> this way.
>
> At any rate, thanks for the advice, I will start using probes!
>
> Cheers,
> -Hugues
>
>
>
> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> Hi,
>>
>> That's not possible. However, what you really want, I think, is not
>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>> drain the connections, then kill it.
>>
>> --
>> Guillaume Quintard
>>
>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>
>>> Hi there,
>>>
>>> I've been looking around and I can't find a documented way of gracefully
>>> shutting down varnishd, and by gracefully I mean tell varnish "stop
>>> accepting connections, but finish what you were doing, then shutdown".
>>>
>>> I did find something in the "first varnish design notes" (
>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed
>>> to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
>>> doesn't seem to work, and TERM, well... terminates but not gracefully.
>>>
>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>> stops connection.
>>>
>>> Is there anyway to achieve this?
>>>
>>> Thanks!
>>> -Hugues
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>
Re: Gracefully stopping varnish [ In reply to ]
> So, there must be a misconception on my, because to me, just refusing new
connections isn't graceful. If the LB is sending you new connections, you
should honor them.

You are right. I took some shortcuts in my explanation.

> So, from what I get, k8s stops sending new connections to Varnish, and
what you want is is for Varnish to shutdown once there's no more active
connections.

That is exactly what happens, indeed. How it happens is irrelevant to our
conversation, but, for the sake of being on the same page it's important to
assume: k8s stopped sending new connections to the old instance and started
sending them to the new instance, and I want old varnish to shutdown once
there's no more active connections.

> But, let's stop being a pompous jerk obsessed with semantics and let's
try to work on a solution.

All good, it's actually important ;)

I will try your solution right away and let you know, in theory it seems
like it should work. Thanks for trying to find a solution!

-Hugues


On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Hello,
>
> So, there must be a misconception on my, because to me, just refusing new
> connections isn't graceful. If the LB is sending you new connections, you
> should honor them.
>
> So, from what I get, k8s stops sending new connections to Varnish, and
> what you want is is for Varnish to shutdown once there's no more active
> connections, which is, I would argue, kinda different from refusing new
> connections. But, let's stop being a pompous jerk obsessed with semantics
> and let's try to work on a solution.
>
> From what I get, we can kill varnish as soon as the number of active
> connections drops to 0, so your apachectl command would be equal to this in
> Varnish language:
>
> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print
> a}'` != 0 ]; then
> sleep 1
> fi
> killall varnishd
>
>
> There's probably a shorter version ,but that's the gist of it.
>
> Would that do?
>
> --
> Guillaume Quintard
>
> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com>
> wrote:
>
>> Hi,
>>
>> That is indeed what I want to do. Draining connections at the LB level
>> just involves more work ;).
>>
>> For context: my entire stack runs on a kubernetes cluster.
>>
>> I sometime need to replace a running instance of a server process [be it
>> nginx apache php-fpm varnish whatever] with a new instance.
>>
>> Taking Apache as an example, I simply create the new apache instance (a
>> new pod in kubernetes speak), which immediately starts getting some traffic
>> (without changing the load balancer configuration at all, a kubernetes
>> "service" automatically detects new pods), then I gracefully shutdown the
>> old instance (kubernetes actually automatically tells the pod to shutdown),
>> by issuing a "apachectl -k graceful-stop" (kubernetes is configured to
>> issue this command for me), which instructs apache to stop accepting
>> connections, finish, then shutdown.
>>
>> It's really great because instead of having to push a new config refusing
>> probes and reload it, I (/kubernetes) simply gracefully stops apache and
>> the traffic flows to the new instance. nginx and php-fpm also handle things
>> this way.
>>
>> At any rate, thanks for the advice, I will start using probes!
>>
>> Cheers,
>> -Hugues
>>
>>
>>
>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>> guillaume@varnish-software.com> wrote:
>>
>>> Hi,
>>>
>>> That's not possible. However, what you really want, I think, is not
>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>> drain the connections, then kill it.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> I've been looking around and I can't find a documented way of
>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>
>>>> I did find something in the "first varnish design notes" (
>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed
>>>> to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
>>>> doesn't seem to work, and TERM, well... terminates but not gracefully.
>>>>
>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>> stops connection.
>>>>
>>>> Is there anyway to achieve this?
>>>>
>>>> Thanks!
>>>> -Hugues
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>
>>
>
Re: Gracefully stopping varnish [ In reply to ]
Your solution works, thanks much.

Cheers,
-Hugues

On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues@betabrand.com> wrote:

> > So, there must be a misconception on my, because to me, just refusing
> new connections isn't graceful. If the LB is sending you new connections,
> you should honor them.
>
> You are right. I took some shortcuts in my explanation.
>
> > So, from what I get, k8s stops sending new connections to Varnish, and
> what you want is is for Varnish to shutdown once there's no more active
> connections.
>
> That is exactly what happens, indeed. How it happens is irrelevant to our
> conversation, but, for the sake of being on the same page it's important to
> assume: k8s stopped sending new connections to the old instance and started
> sending them to the new instance, and I want old varnish to shutdown once
> there's no more active connections.
>
> > But, let's stop being a pompous jerk obsessed with semantics and let's
> try to work on a solution.
>
> All good, it's actually important ;)
>
> I will try your solution right away and let you know, in theory it seems
> like it should work. Thanks for trying to find a solution!
>
> -Hugues
>
>
> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> Hello,
>>
>> So, there must be a misconception on my, because to me, just refusing new
>> connections isn't graceful. If the LB is sending you new connections, you
>> should honor them.
>>
>> So, from what I get, k8s stops sending new connections to Varnish, and
>> what you want is is for Varnish to shutdown once there's no more active
>> connections, which is, I would argue, kinda different from refusing new
>> connections. But, let's stop being a pompous jerk obsessed with semantics
>> and let's try to work on a solution.
>>
>> From what I get, we can kill varnish as soon as the number of active
>> connections drops to 0, so your apachectl command would be equal to this in
>> Varnish language:
>>
>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print
>> a}'` != 0 ]; then
>> sleep 1
>> fi
>> killall varnishd
>>
>>
>> There's probably a shorter version ,but that's the gist of it.
>>
>> Would that do?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com>
>> wrote:
>>
>>> Hi,
>>>
>>> That is indeed what I want to do. Draining connections at the LB level
>>> just involves more work ;).
>>>
>>> For context: my entire stack runs on a kubernetes cluster.
>>>
>>> I sometime need to replace a running instance of a server process [be it
>>> nginx apache php-fpm varnish whatever] with a new instance.
>>>
>>> Taking Apache as an example, I simply create the new apache instance (a
>>> new pod in kubernetes speak), which immediately starts getting some traffic
>>> (without changing the load balancer configuration at all, a kubernetes
>>> "service" automatically detects new pods), then I gracefully shutdown the
>>> old instance (kubernetes actually automatically tells the pod to shutdown),
>>> by issuing a "apachectl -k graceful-stop" (kubernetes is configured to
>>> issue this command for me), which instructs apache to stop accepting
>>> connections, finish, then shutdown.
>>>
>>> It's really great because instead of having to push a new config
>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>> handle things this way.
>>>
>>> At any rate, thanks for the advice, I will start using probes!
>>>
>>> Cheers,
>>> -Hugues
>>>
>>>
>>>
>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>> guillaume@varnish-software.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> That's not possible. However, what you really want, I think, is not
>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>> drain the connections, then kill it.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> I've been looking around and I can't find a documented way of
>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>
>>>>> I did find something in the "first varnish design notes" (
>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which seemed
>>>>> to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL
>>>>> doesn't seem to work, and TERM, well... terminates but not gracefully.
>>>>>
>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>> stops connection.
>>>>>
>>>>> Is there anyway to achieve this?
>>>>>
>>>>> Thanks!
>>>>> -Hugues
>>>>>
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc@varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>
>>>
>>
>
Re: Gracefully stopping varnish [ In reply to ]
Glad to hear it! Out of curiosity, do you run that script manually on
container you want to kill, or is there a way to do it automatically?

--
Guillaume Quintard

On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <hugues@betabrand.com> wrote:

> Your solution works, thanks much.
>
> Cheers,
> -Hugues
>
> On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues@betabrand.com>
> wrote:
>
>> > So, there must be a misconception on my, because to me, just refusing
>> new connections isn't graceful. If the LB is sending you new connections,
>> you should honor them.
>>
>> You are right. I took some shortcuts in my explanation.
>>
>> > So, from what I get, k8s stops sending new connections to Varnish, and
>> what you want is is for Varnish to shutdown once there's no more active
>> connections.
>>
>> That is exactly what happens, indeed. How it happens is irrelevant to our
>> conversation, but, for the sake of being on the same page it's important to
>> assume: k8s stopped sending new connections to the old instance and started
>> sending them to the new instance, and I want old varnish to shutdown once
>> there's no more active connections.
>>
>> > But, let's stop being a pompous jerk obsessed with semantics and let's
>> try to work on a solution.
>>
>> All good, it's actually important ;)
>>
>> I will try your solution right away and let you know, in theory it seems
>> like it should work. Thanks for trying to find a solution!
>>
>> -Hugues
>>
>>
>> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
>> guillaume@varnish-software.com> wrote:
>>
>>> Hello,
>>>
>>> So, there must be a misconception on my, because to me, just refusing
>>> new connections isn't graceful. If the LB is sending you new connections,
>>> you should honor them.
>>>
>>> So, from what I get, k8s stops sending new connections to Varnish, and
>>> what you want is is for Varnish to shutdown once there's no more active
>>> connections, which is, I would argue, kinda different from refusing new
>>> connections. But, let's stop being a pompous jerk obsessed with semantics
>>> and let's try to work on a solution.
>>>
>>> From what I get, we can kill varnish as soon as the number of active
>>> connections drops to 0, so your apachectl command would be equal to this in
>>> Varnish language:
>>>
>>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print
>>> a}'` != 0 ]; then
>>> sleep 1
>>> fi
>>> killall varnishd
>>>
>>>
>>> There's probably a shorter version ,but that's the gist of it.
>>>
>>> Would that do?
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> That is indeed what I want to do. Draining connections at the LB level
>>>> just involves more work ;).
>>>>
>>>> For context: my entire stack runs on a kubernetes cluster.
>>>>
>>>> I sometime need to replace a running instance of a server process [be
>>>> it nginx apache php-fpm varnish whatever] with a new instance.
>>>>
>>>> Taking Apache as an example, I simply create the new apache instance (a
>>>> new pod in kubernetes speak), which immediately starts getting some traffic
>>>> (without changing the load balancer configuration at all, a kubernetes
>>>> "service" automatically detects new pods), then I gracefully shutdown the
>>>> old instance (kubernetes actually automatically tells the pod to shutdown),
>>>> by issuing a "apachectl -k graceful-stop" (kubernetes is configured to
>>>> issue this command for me), which instructs apache to stop accepting
>>>> connections, finish, then shutdown.
>>>>
>>>> It's really great because instead of having to push a new config
>>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>>> handle things this way.
>>>>
>>>> At any rate, thanks for the advice, I will start using probes!
>>>>
>>>> Cheers,
>>>> -Hugues
>>>>
>>>>
>>>>
>>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>>> guillaume@varnish-software.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> That's not possible. However, what you really want, I think, is not
>>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>>> drain the connections, then kill it.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I've been looking around and I can't find a documented way of
>>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>>
>>>>>> I did find something in the "first varnish design notes" (
>>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which
>>>>>> seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop"
>>>>>> but KILL doesn't seem to work, and TERM, well... terminates but not
>>>>>> gracefully.
>>>>>>
>>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>>> stops connection.
>>>>>>
>>>>>> Is there anyway to achieve this?
>>>>>>
>>>>>> Thanks!
>>>>>> -Hugues
>>>>>>
>>>>>> _______________________________________________
>>>>>> varnish-misc mailing list
>>>>>> varnish-misc@varnish-cache.org
>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Gracefully stopping varnish [ In reply to ]
Since this could be useful to some other people, here's a bit more details
on how it's implemented on my end.

TL;DR: it's automatic. I embedded the script in my docker image and it gets
run by k8s when needed.


*My container image looks like this:*

```
FROM bb:base
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update
RUN apt-get install -y --force-yes varnish

COPY files/graceful-shutdown.sh /

CMD our-long-varnishd-cmd
```

*graceful-shutdown.sh looks like this:*

```
CONNECTIONS_REMAINING=1
while [ $CONNECTIONS_REMAINING != 0 ]; do
CONNECTIONS_REMAINING=`varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/
{a+=$2} END {print a}'`
echo "$CONNECTIONS_REMAINING remaining, waiting..."
sleep 1
done
killall varnishd
```

*I then a have a k8s deployment file:*

```
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: varnish
labels:
app: varnish
version: "5.2"
spec:
replicas: 1
template:
metadata:
labels:
app: varnish
spec:
terminationGracePeriodSeconds: 300
containers:
- name: varnish
image: bb/bb:varnish
lifecycle:
preStop:
exec:
command:
- "sh"
- "/graceful-shutdown.sh"
imagePullPolicy: Always
command: ["our-long-varnishd-cmd"]
```

The preStop command is run by k8s (when needed**), which will wait until
the command exits (up to 300s in this config). Once the preStop command
exits, k8s will send a SIGTERM to varnishd (which should already be gone by
that time).

(** `kubectl delete my-varnish-pod` for example, or, if I upload a new
deployment file or other conditions triggering a shutdown of the pod.)

-H

On Wed, Oct 18, 2017 at 1:00 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Glad to hear it! Out of curiosity, do you run that script manually on
> container you want to kill, or is there a way to do it automatically?
>
> --
> Guillaume Quintard
>
> On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <hugues@betabrand.com>
> wrote:
>
>> Your solution works, thanks much.
>>
>> Cheers,
>> -Hugues
>>
>> On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues@betabrand.com>
>> wrote:
>>
>>> > So, there must be a misconception on my, because to me, just refusing
>>> new connections isn't graceful. If the LB is sending you new connections,
>>> you should honor them.
>>>
>>> You are right. I took some shortcuts in my explanation.
>>>
>>> > So, from what I get, k8s stops sending new connections to Varnish,
>>> and what you want is is for Varnish to shutdown once there's no more active
>>> connections.
>>>
>>> That is exactly what happens, indeed. How it happens is irrelevant to
>>> our conversation, but, for the sake of being on the same page it's
>>> important to assume: k8s stopped sending new connections to the old
>>> instance and started sending them to the new instance, and I want old
>>> varnish to shutdown once there's no more active connections.
>>>
>>> > But, let's stop being a pompous jerk obsessed with semantics and let's
>>> try to work on a solution.
>>>
>>> All good, it's actually important ;)
>>>
>>> I will try your solution right away and let you know, in theory it seems
>>> like it should work. Thanks for trying to find a solution!
>>>
>>> -Hugues
>>>
>>>
>>> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
>>> guillaume@varnish-software.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> So, there must be a misconception on my, because to me, just refusing
>>>> new connections isn't graceful. If the LB is sending you new connections,
>>>> you should honor them.
>>>>
>>>> So, from what I get, k8s stops sending new connections to Varnish, and
>>>> what you want is is for Varnish to shutdown once there's no more active
>>>> connections, which is, I would argue, kinda different from refusing new
>>>> connections. But, let's stop being a pompous jerk obsessed with semantics
>>>> and let's try to work on a solution.
>>>>
>>>> From what I get, we can kill varnish as soon as the number of active
>>>> connections drops to 0, so your apachectl command would be equal to this in
>>>> Varnish language:
>>>>
>>>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END
>>>> {print a}'` != 0 ]; then
>>>> sleep 1
>>>> fi
>>>> killall varnishd
>>>>
>>>>
>>>> There's probably a shorter version ,but that's the gist of it.
>>>>
>>>> Would that do?
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> That is indeed what I want to do. Draining connections at the LB level
>>>>> just involves more work ;).
>>>>>
>>>>> For context: my entire stack runs on a kubernetes cluster.
>>>>>
>>>>> I sometime need to replace a running instance of a server process [be
>>>>> it nginx apache php-fpm varnish whatever] with a new instance.
>>>>>
>>>>> Taking Apache as an example, I simply create the new apache instance
>>>>> (a new pod in kubernetes speak), which immediately starts getting some
>>>>> traffic (without changing the load balancer configuration at all, a
>>>>> kubernetes "service" automatically detects new pods), then I gracefully
>>>>> shutdown the old instance (kubernetes actually automatically tells the pod
>>>>> to shutdown), by issuing a "apachectl -k graceful-stop" (kubernetes is
>>>>> configured to issue this command for me), which instructs apache to stop
>>>>> accepting connections, finish, then shutdown.
>>>>>
>>>>> It's really great because instead of having to push a new config
>>>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>>>> handle things this way.
>>>>>
>>>>> At any rate, thanks for the advice, I will start using probes!
>>>>>
>>>>> Cheers,
>>>>> -Hugues
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>>>> guillaume@varnish-software.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> That's not possible. However, what you really want, I think, is not
>>>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>>>> drain the connections, then kill it.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I've been looking around and I can't find a documented way of
>>>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>>>
>>>>>>> I did find something in the "first varnish design notes" (
>>>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which
>>>>>>> seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop"
>>>>>>> but KILL doesn't seem to work, and TERM, well... terminates but not
>>>>>>> gracefully.
>>>>>>>
>>>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>>>> stops connection.
>>>>>>>
>>>>>>> Is there anyway to achieve this?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Hugues
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> varnish-misc mailing list
>>>>>>> varnish-misc@varnish-cache.org
>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Gracefully stopping varnish [ In reply to ]
Thanks for sharing!

On Wed, Oct 18, 2017 at 3:35 PM, Hugues Alary <hugues@betabrand.com> wrote:

> Since this could be useful to some other people, here's a bit more details
> on how it's implemented on my end.
>
> TL;DR: it's automatic. I embedded the script in my docker image and it
> gets run by k8s when needed.
>
>
> *My container image looks like this:*
>
> ```
> FROM bb:base
> ENV DEBIAN_FRONTEND noninteractive
>
> RUN apt-get update
> RUN apt-get install -y --force-yes varnish
>
> COPY files/graceful-shutdown.sh /
>
> CMD our-long-varnishd-cmd
> ```
>
> *graceful-shutdown.sh looks like this:*
>
> ```
> CONNECTIONS_REMAINING=1
> while [ $CONNECTIONS_REMAINING != 0 ]; do
> CONNECTIONS_REMAINING=`varnishstat -1 | awk
> '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print a}'`
> echo "$CONNECTIONS_REMAINING remaining, waiting..."
> sleep 1
> done
> killall varnishd
> ```
>
> *I then a have a k8s deployment file:*
>
> ```
> ---
> apiVersion: extensions/v1beta1
> kind: Deployment
> metadata:
> name: varnish
> labels:
> app: varnish
> version: "5.2"
> spec:
> replicas: 1
> template:
> metadata:
> labels:
> app: varnish
> spec:
> terminationGracePeriodSeconds: 300
> containers:
> - name: varnish
> image: bb/bb:varnish
> lifecycle:
> preStop:
> exec:
> command:
> - "sh"
> - "/graceful-shutdown.sh"
> imagePullPolicy: Always
> command: ["our-long-varnishd-cmd"]
> ```
>
> The preStop command is run by k8s (when needed**), which will wait until
> the command exits (up to 300s in this config). Once the preStop command
> exits, k8s will send a SIGTERM to varnishd (which should already be gone by
> that time).
>
> (** `kubectl delete my-varnish-pod` for example, or, if I upload a new
> deployment file or other conditions triggering a shutdown of the pod.)
>
> -H
>
> On Wed, Oct 18, 2017 at 1:00 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> Glad to hear it! Out of curiosity, do you run that script manually on
>> container you want to kill, or is there a way to do it automatically?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <hugues@betabrand.com>
>> wrote:
>>
>>> Your solution works, thanks much.
>>>
>>> Cheers,
>>> -Hugues
>>>
>>> On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues@betabrand.com>
>>> wrote:
>>>
>>>> > So, there must be a misconception on my, because to me, just
>>>> refusing new connections isn't graceful. If the LB is sending you new
>>>> connections, you should honor them.
>>>>
>>>> You are right. I took some shortcuts in my explanation.
>>>>
>>>> > So, from what I get, k8s stops sending new connections to Varnish,
>>>> and what you want is is for Varnish to shutdown once there's no more active
>>>> connections.
>>>>
>>>> That is exactly what happens, indeed. How it happens is irrelevant to
>>>> our conversation, but, for the sake of being on the same page it's
>>>> important to assume: k8s stopped sending new connections to the old
>>>> instance and started sending them to the new instance, and I want old
>>>> varnish to shutdown once there's no more active connections.
>>>>
>>>> > But, let's stop being a pompous jerk obsessed with semantics and
>>>> let's try to work on a solution.
>>>>
>>>> All good, it's actually important ;)
>>>>
>>>> I will try your solution right away and let you know, in theory it
>>>> seems like it should work. Thanks for trying to find a solution!
>>>>
>>>> -Hugues
>>>>
>>>>
>>>> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
>>>> guillaume@varnish-software.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> So, there must be a misconception on my, because to me, just refusing
>>>>> new connections isn't graceful. If the LB is sending you new connections,
>>>>> you should honor them.
>>>>>
>>>>> So, from what I get, k8s stops sending new connections to Varnish, and
>>>>> what you want is is for Varnish to shutdown once there's no more active
>>>>> connections, which is, I would argue, kinda different from refusing new
>>>>> connections. But, let's stop being a pompous jerk obsessed with semantics
>>>>> and let's try to work on a solution.
>>>>>
>>>>> From what I get, we can kill varnish as soon as the number of active
>>>>> connections drops to 0, so your apachectl command would be equal to this in
>>>>> Varnish language:
>>>>>
>>>>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END
>>>>> {print a}'` != 0 ]; then
>>>>> sleep 1
>>>>> fi
>>>>> killall varnishd
>>>>>
>>>>>
>>>>> There's probably a shorter version ,but that's the gist of it.
>>>>>
>>>>> Would that do?
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues@betabrand.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> That is indeed what I want to do. Draining connections at the LB
>>>>>> level just involves more work ;).
>>>>>>
>>>>>> For context: my entire stack runs on a kubernetes cluster.
>>>>>>
>>>>>> I sometime need to replace a running instance of a server process [be
>>>>>> it nginx apache php-fpm varnish whatever] with a new instance.
>>>>>>
>>>>>> Taking Apache as an example, I simply create the new apache instance
>>>>>> (a new pod in kubernetes speak), which immediately starts getting some
>>>>>> traffic (without changing the load balancer configuration at all, a
>>>>>> kubernetes "service" automatically detects new pods), then I gracefully
>>>>>> shutdown the old instance (kubernetes actually automatically tells the pod
>>>>>> to shutdown), by issuing a "apachectl -k graceful-stop" (kubernetes is
>>>>>> configured to issue this command for me), which instructs apache to stop
>>>>>> accepting connections, finish, then shutdown.
>>>>>>
>>>>>> It's really great because instead of having to push a new config
>>>>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>>>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>>>>> handle things this way.
>>>>>>
>>>>>> At any rate, thanks for the advice, I will start using probes!
>>>>>>
>>>>>> Cheers,
>>>>>> -Hugues
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>>>>> guillaume@varnish-software.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> That's not possible. However, what you really want, I think, is not
>>>>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>>>>> drain the connections, then kill it.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues@betabrand.com> wrote:
>>>>>>>
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> I've been looking around and I can't find a documented way of
>>>>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>>>>
>>>>>>>> I did find something in the "first varnish design notes" (
>>>>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which
>>>>>>>> seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop"
>>>>>>>> but KILL doesn't seem to work, and TERM, well... terminates but not
>>>>>>>> gracefully.
>>>>>>>>
>>>>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>>>>> stops connection.
>>>>>>>>
>>>>>>>> Is there anyway to achieve this?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -Hugues
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> varnish-misc mailing list
>>>>>>>> varnish-misc@varnish-cache.org
>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>