Mailing List Archive

Best practice for caching scenario with different backend servers but same content
Hi,
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish,
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:

```
In varnish-1:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b10());



In varnish-2:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b10());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());


In varnish-3:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b10());

```
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,
What is varnish recommendation for this scenario?
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,

There are a lot of things to unpack here.

> if a varnish for specific file requests to one backend server and for the
same file but to another backend server it would cache that file again
because of different Host headers ! so my solution is using fallback
director instead of round-robin

The two aren't related, if you have a hashing problem causing you to cache
the same object twice, changing the directors isn't going to save you.
Ideally, the requests will get normalized (host header and path) in
vcl_recv{} so that they will be properly hashed in vcl_hash{}.

The backend resolution only happens after you have exited
vcl_backend_fetch{}, long after you have (not) found the object in the
cache, and the best solution for video is usually to use
consistent-hashing. In open-source this means vmod_shard (
https://varnish-cache.org/docs/trunk/reference/vmod_directors.html#directors-shard),
in Enterprise, it'll be udo (
https://docs.varnish-software.com/varnish-cache-plus/vmods/udo/#set-hash),
they are going to handle about the same except udo makes it easier to set
the hash, which may be important for live (more info below).

With consistent hashing, you can configure all Varnish servers the same,
and they will determine which backend to use based on the request.
Typically, the same request will always go to the same backend. This
provides pretty good load-balancing over time, and additionally it
leverages the internally caching that most video origins have.

If you are serving VOD, that is all you need, but if you are serving Live,
you need to care about one other thing: you want consistent hashing not per
request, but per stream. Because the origins may be slightly out of sync,
you may get a manifest on origin A which will advertise a chunk that isn't
available anywhere yet, and if you don't fetch the new chunk from origin A,
you'll get a 404 or a 412.
So, for live, you will need to use shard's key() (
https://varnish-cache.org/docs/trunk/reference/vmod_directors.html#int-xshard-key-string)
or udo's set_hash() (
https://docs.varnish-software.com/varnish-cache-plus/vmods/udo/#set-hash)
to create a hash based on the stream path.

For example, consider these paths:
- /live/australia/Channel5/480p/manifest.m3u8 and
/live/australia/Channel5/480p/chunk_43212123.ts: the stream path is
/live/australia/Channel5/480p/
- /video/live/52342645323/manifest.dash and
/video/live/52342645323/manifest.dash?time=4216432432&bitrate=80000000&audio=523453:
the stream path is /video/live/52342645323/manifest.dash

On top of all this, if you start having more than 5 Varnish servers, you
might want to consider adding an extra layer of caching between the
client-facing Varnish nodes and the origins (origin shields) to reduce
the load on the origins. In that case, the shields would be the one
handling the consistent hashing.

Hope this helps

--
Guillaume Quintard


On Sun, Aug 1, 2021 at 4:18 AM Hamidreza Hosseini <hrhosseini@hotmail.com>
wrote:

> Hi,
> I want to use varnish in my scenario as cache service, I have about 10
> http servers that serve Hls fragments as the backend servers and about 5
> varnish servers for caching purpose, the problem comes in when I use
> round-robin director for backend servers in varnish,
> if a varnish for specific file requests to one backend server and for the
> same file but to another backend server it would cache that file again
> because of different Host headers ! so my solution is using fallback
> director instead of round-robin as follow:
>
> ```
> In varnish-1:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b9());
> hls_cluster.add_backend(b10());
>
>
>
> In varnish-2:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b10());
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b9());
>
>
> In varnish-3:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b9());
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b10());
>
> ```
> But I think this is not the best solution, because there is no load
> balancing despite, I used different backend for the first argument of
> fallback directive,
> What is varnish recommendation for this scenario?
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,
1.

Is there any way to normalize host headers and other things to say to varnish not to cache the same content for different backend?
I want to use round robin director but after fetching the content I want to normalize the header and cache the content,
I would appreciate if you give me an example about this and how I can do it.

2.
I couldn't find any good example for directors-shard and xshard-key-string, I would appreciate if you could give example about this too.

Many Thanks
________________________________
From: varnish-misc <varnish-misc-bounces+hrhosseini=hotmail.com@varnish-cache.org> on behalf of Hamidreza Hosseini <hrhosseini@hotmail.com>
Sent: Sunday, August 1, 2021 4:17 AM
To: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Best practice for caching scenario with different backend servers but same content

Hi,
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish,
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:

```
In varnish-1:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b10());



In varnish-2:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b10());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());


In varnish-3:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b10());

```
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,
What is varnish recommendation for this scenario?
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,

I'm pretty sure there's a confusion with the sequence of actions here.
Normalization happen *before* you look into the cache, so that way before
you fetch anything from the backend. By the time you cache the data
(vcl_backend_response), the hash key has already been set (vcl_hash), it's
way too late to normalize the request.

As to normalization, it's usually done in vcl_recv, and it can range from
just setting the host header to a static string to using std.tolower() and
removing the host port.

for the sake of the example:

sub vcl_vcl {
set req.http.host = "myvideoservice.com";
}

For shard example, look at the VCTs, for example:
https://github.com/varnishcache/varnish-cache/blob/6.6/bin/varnishtest/tests/d00029.vtc#L66

import directors;

sub vcl_init {
new shard_dir = directors.shard();
shard_dir.add_backend(be1);
shard_dir.add_backend(be2);
shard_dir.add_backend(be3;

new p = directors.shard_param();
vd.associate(p.use());

vd.reconfigure(replicas=25);}
sub vcl_backend_fetch {
p.set(by=KEY, key=bereq.url);
set bereq.backend_hint = shard_dir.backend(resolve=LAZY);}

For udo:

import crypto;
import udo;

sub vcl_init {
new udo_dir = udo.director();
udo_dir.set_type(random);
udo_dir.add_backend(be1);
udo_dir.add_backend(be2);
udo_dir.add_backend(be3);
udo_dir.set_type(hash);}
sub vcl_backend_fetch {
set bereq.backend_hint = udo_dir.backend();
udo_dir.set_hash(crypto.hash(sha256, bereq.url));}


These have been written without testing, so don't put them straight into
production.

--
Guillaume Quintard


On Thu, Aug 5, 2021 at 3:33 AM Hamidreza Hosseini <hrhosseini@hotmail.com>
wrote:

> Hi,
> 1.
>
> Is there any way to normalize host headers and other things to say to
> varnish not to cache the same content for different backend?
> I want to use round robin director but after fetching the content I want
> to normalize the header and cache the content,
> I would appreciate if you give me an example about this and how I can do
> it.
>
> 2.
> I couldn't find any good example for directors-shard and
> xshard-key-string, I would appreciate if you could give example about this
> too.
>
> Many Thanks
> ------------------------------
> *From:* varnish-misc <varnish-misc-bounces+hrhosseini=
> hotmail.com@varnish-cache.org> on behalf of Hamidreza Hosseini <
> hrhosseini@hotmail.com>
> *Sent:* Sunday, August 1, 2021 4:17 AM
> *To:* varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
> *Subject:* Best practice for caching scenario with different backend
> servers but same content
>
> Hi,
> I want to use varnish in my scenario as cache service, I have about 10
> http servers that serve Hls fragments as the backend servers and about 5
> varnish servers for caching purpose, the problem comes in when I use
> round-robin director for backend servers in varnish,
> if a varnish for specific file requests to one backend server and for the
> same file but to another backend server it would cache that file again
> because of different Host headers ! so my solution is using fallback
> director instead of round-robin as follow:
>
> ```
> In varnish-1:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b9());
> hls_cluster.add_backend(b10());
>
>
>
> In varnish-2:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b10());
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b9());
>
>
> In varnish-3:
> new hls_cluster = directors.fallback();
> hls_cluster.add_backend(b9());
> hls_cluster.add_backend(b1());
> hls_cluster.add_backend(b2());
> hls_cluster.add_backend(b3());
> hls_cluster.add_backend(b4());
> hls_cluster.add_backend(b5());
> hls_cluster.add_backend(b6());
> hls_cluster.add_backend(b7());
> hls_cluster.add_backend(b8());
> hls_cluster.add_backend(b10());
>
> ```
> But I think this is not the best solution, because there is no load
> balancing despite, I used different backend for the first argument of
> fallback directive,
> What is varnish recommendation for this scenario?
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,
I read the sample config that you sent but Can I use "bereq.url" in this way:
for example I want to shard my requests for live streams based on the url's that clients enter, for example if the following url's are different live streams (stream1 and stream2 are the name of different streams ):
```
mydomain.com/live/australia/stream1/chunk_43212123.ts
mydomain.com/live/australia/stream2/chunk_43212123.ts
mydomain.com/live/australia/stream3/chunk_43212123.ts
```
Now think I want just the url excluded with chunk file becomes hashed and sharded:
Just this part: "/live/australia/stream{1,2,3}/"
Not :"/live/australia/stream{1,2,3}/chunk_43212123.ts"

So by adjusting " p.set(by=KEY, key=bereq.url) " it would shard "bereq.url", it means "/live/australia/stream{1,2,3}/chunk_43212123.ts"


________________________________
From: varnish-misc <varnish-misc-bounces+hrhosseini=hotmail.com@varnish-cache.org> on behalf of Hamidreza Hosseini <hrhosseini@hotmail.com>
Sent: Sunday, August 1, 2021 4:17 AM
To: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Best practice for caching scenario with different backend servers but same content

Hi,
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish,
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:

```
In varnish-1:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b10());



In varnish-2:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b10());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());


In varnish-3:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b10());

```
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,
What is varnish recommendation for this scenario?
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,
This is my configuration based on Doc and sample configuration, I would appreciate if you answer my questions:

```
probe myprobe {
.request =
"HEAD / HTTP/1.1"
"Connection: close"
"User-Agent: Varnish Health Probe";
.timeout = 1s;
.interval = 5s;
.window = 5;
.threshold = 3;
}


backend B1 { .host = "B1.mydomain.local"; .port = "80"; .probe = myprobe; }
backend B2 { .host = "B2.mydomain.local"; .port = "80"; .probe = myprobe; }
backend B3 { .host = "B3.mydomain.local"; .port = "80"; .probe = myprobe; }
backend B4 { .host = "B4.mydomain.local"; .port = "80"; .probe = myprobe; }
backend B5 { .host = "B5.mydomain.local"; .port = "80"; .probe = myprobe; }



sub vcl_init {

new hls_cluster = directors.shard();
hls_cluster.add_backend(B1);
hls_cluster.add_backend(B2);
hls_cluster.add_backend(B3);
hls_cluster.add_backend(B4);
hls_cluster.add_backend(B5);


new p = directors.shard_param();
hls_cluster.reconfigure(p, replicas=25);
hls_cluster.associate(p.use());


}


sub vcl_backend_fetch {
p.set(by=KEY, key=bereq.url);
set bereq.backend_hint = hls_cluster.backend(resolve=LAZY, healthy=ALL);}

}


```


1. First of all I think following section is not true, Do we need define both shard parameter(p) and replicas in the reconfigure directive?
"hls_cluster.reconfigure(p, replicas=25);" or just "hls_cluster.reconfigure(replicas=25);"



2. What does "replicas=25" means in the sample configuration?
Doc says:
<ident><n> (default ident being the backend name) for each backend and for a running number n from 1 to replicas

this is what I found out:
varnish will choose one number for a backend randomly from 1 to "replicas" number and then will combine it with name of backend and then will hash all for circular ring,
But when we have backends with different names this hash would be different for each backend because they have different name!
why is this neccessary?

`https://varnish-cache.org/docs/6.2/reference/vmod_directors.html#bool-xshard-reconfigure-int-replicas-67`


3. In the shard.backend(...) section and About "resolve=LAZY":
I couldn't understand what does LAZY resolve mean?
DOC:
```
LAZY: return an instance of this director for later backend resolution.
LAZY mode is required for referencing shard director instances, for example as backends for other directors (director layering).
```
https://varnish-cache.org/docs/6.1/reference/vmod_directors.generated.html#shard-backend


4. For returning healthy backend besides defining probes as I adjust, should I configure healthy=ALL as follow?

```
set bereq.backend_hint = hls_cluster.backend(resolve=LAZY, healthy=ALL);}
```
DOC:

```
ALL: Check health state also for alternative backend selection
```

5. About rampup and warmup :
rampup: I understand that if a backend goes down and become healthy again if we defined a rampup period for it, it would wait till this period passed and then varnish will send the request to that backend for this fraction of time it will return alternative backend
warmup: for a choosen backend for specific key it will spread request between two backend (the original backend and its alternative if we define 0.5 for warmup)

Please correct me if I said anything wrong. I would apprecate if you could explain about the functionality of this two parameter.


6. conceptual question:

1.What's the exact difference between hash and shard directive and when should we use which one?
the Doc says when the backend changes shard is more consistent than hash but how?

2.What will it happen when I'm using shard director based on "key=bereq.url" if I add/delete one backend from backend lists? will it change the consistent hashing ring for requests?

Thanks for your answers in advance
Best regards, Hamidreza





________________________________
From: Hamidreza Hosseini
Sent: Sunday, August 1, 2021 4:17 AM
To: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Best practice for caching scenario with different backend servers but same content

Hi,
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish,
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:

```
In varnish-1:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b10());



In varnish-2:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b10());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());


In varnish-3:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b10());

```
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,
What is varnish recommendation for this scenario?
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hello,

The best way to answer these questions is to start with the last one:

On 8/9/21 10:50, Hamidreza Hosseini wrote:
>
> 6. conceptual question:
>
> 1.What's the exact difference between hash and shard directive and when
> should we use which one?
> the Doc says when the backend changes shard is more consistent than hash
> but how?

It's not "more consistent", it's "consistent hashing", which is the name
of a hashing algorithm intended for load balancing:

https://en.wikipedia.org/wiki/Consistent_hashing

The hash director uses a more traditional kind of hashing algorithm to
map requests to backends. Consistent hashing, implemented by the shard
director, is intended to mitigate problems that can arise when backends
become unhealthy and then healthy again. The focus is mainly on backends
that do something expensive for new requests, such as their own caching,
but can work faster for requests that they've seen before.

A hash algorithm computes a number h(b) for a backend b, whose value is
in a large range, say 32 bit. Then if you have N backends, the
traditional algorithm indexes them from 0 to N-1, and picks h(b) mod N.
Varnish's hash director is something like that.

Say you have N=10 backends, so the traditional algorithm picks h(b) mod
10. Then a backend goes unhealthy, so now N=9. Since x mod 10 is unequal
to x mod 9 for all x, the mapping of requests to backends shifts
completely. This can be painful for backends that benefit from getting
mostly the same requests, for example due to caching.

After a while, the unhealthy backend becomes healthy again, so we go
from N=9 back to N=10. If in the meantime the backends had "gotten used
to" the changed distribution of requests, say by filling their local
caches, then they get the pain all over again.

Consistent hashing attempts to lessen the pain. If a backend drops out,
then the mapping of requests to that backend must change, but the
mapping stays the same for all other backends. So the distribution to
backends changes only as much as it has to.

Disclaimer: the original version of the shard director was developed at
my company. Some lines of code that I contributed are still in there.

> 2.What will it happen when I'm using shard director based on
> "key=bereq.url" if I add/delete one backend from backend lists? will it
> change the consistent hashing ring for requests?

That's what consistent hashing is about. It only changes for the
backends that were added or deleted, not for the rest.

> 1. First of all I think following section is not true, Do we need define
> both shard parameter(p) and replicas in the reconfigure directive?
> "hls_cluster.reconfigure(p, replicas=25);" or just
> "hls_cluster.reconfigure(replicas=25);"

There is no 2-argument form of reconfigure(). It has one optional
argument (replicas) with a default value.

> 2. What does "replicas=25" means in the sample configuration?
>
> why is this neccessary?

The short answer is that if you have to ask, then use the default. Since
the argument is optional, and set to the default if you leave it out,
just leave it out: reconfigure()

You should NOT set it to 25.

replicas is an internal parameter of the algorithm, and there aren't
many guidelines as how it should be set, so we wanted to be able to
experiment with it. It's really there so that developers of Varnish and
the director can test it. Most users of Varnish don't need to worry
about it.

It turns out that the value doesn't matter all that much, as long as it
isn't too low. 25 is too low. I advise against setting it lower than the
default (67).

What replicas does and why it's necessary gets into details of the
algorithm. We can say more about that if there's interest, but this
email is getting pretty long as it is. (The Wikipedia article gets into
this in the section "Practical Extensions".)

> 3. In the shard.backend(...) section and About "resolve=LAZY":
> I couldn't understand what does LAZY resolve mean?

When you set bereq.backend to a director, most of the Varnish directors
do not immediately execute the algorithm to choose a backend, not until
it's time to actually send the backend request. This has some
advantages, for example if the VCL logic after setting bereq.backend
results in not going to that backend after all. resolve=LAZY works this way.

The alternative resolve=NOW is for contexts where you return the backend
(read the return value of shard.backend()) and need to know which
backend it's going to be. Then the backend is chosen right away, and
stays that way when the backend request is sent.

> 4. For returning healthy backend besides defining probes as I adjust,
> should I configure healthy=ALL as follow?

The parameters alt, healthy, rampup and warmup give you some control
over what happens when one or more backends drop out.

Say your request ordinarily maps to b1, but b1 is unhealthy; then there
is a specific backend b2 that is chosen next. If b2 is also unhealthy,
then there is a specific alternative b3, and so on. These are always the
same. In other words, the order of alternatives chosen for unhealthy
backends is constant for a given configuration.

If you set alt=N with N > 0, then the Nth backend in that order is
chosen. This is mainly for retries -- by setting alt=bereq.retries, you
try a different backend on each retry, in the order established by b1,
b2, b3 ... etc.

The healthy parameter controls what happens when the director searches
down the list due to unhealthy backends. healthy=ALL means that it
continues searching, starting at alt=N (from the start when alt=0),
until a healthy backend is found (or fail if they're all unhealthy).

healthy=CHOSEN means don't skip ahead due to alt, just search for a
healthy backend starting from the beginning of the list.

healthy=IGNORE means don't consider the health status, and just choose
the backend at alt=N no matter what.

> 5. About rampup and warmup :
> rampup: I understand that if a backend goes down and become healthy
> again if we defined a rampup period for it, it would wait till this
> period passed and then varnish will send the request to that backend for
> this fraction of time it will return alternative backend

Not quite. When a backend is added and rampup is set, then the
probability of choosing that backend increases during the time set by
rampup. Say for rampup=60s, the probability is very low just after the
backend goes healthy, 25% after 15 seconds, 50% after 30 seconds, and so on.

The idea is for backends that do expensive operations such as caching
for "new" requests. Such a backend could be overwhelmed if Varnish sends
all of the new requests at once when the backend becomes healthy, so
rampup increases the load slowly.

> warmup: for a choosen backend for specific key it will spread request
> between two backend (the original backend and its alternative if we
> define 0.5 for warmup)

Yes. warmup is the probability that the next backend down the line is
chosen, even if the first backend that would have been chosen is healthy.

This is also for backends that would suffer under heavy load if another
backend goes unhealthy, due to new requests that would have gone to the
unhealthy backend. With warmup, the backend gets a portion of those
requests even when the other backend is healthy, so that it's partially
prepared for the additional load if the other backend drops out.


HTH,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
On 8/9/21 14:49, Geoff Simmons wrote:
>
> A hash algorithm computes a number h(b) for a backend b, ...

Sorry, this should have been more like h(bereq), meaning that the number
is computed from features of the request. From that you get to the
choice of a backend.

The Varnish hash director uses the same hash value that was computed for
caching.

The shard director also does that by default, but some other choices can
be set with the parameter by in shard.backend(): by=HASH (default),
by=URL, by=KEY, by=BLOB


Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
On Sat, Aug 7, 2021 at 9:21 AM Hamidreza Hosseini
<hrhosseini@hotmail.com> wrote:
>
> Hi,
> I read the sample config that you sent but Can I use "bereq.url" in this way:
> for example I want to shard my requests for live streams based on the url's that clients enter, for example if the following url's are different live streams (stream1 and stream2 are the name of different streams ):
> ```
> mydomain.com/live/australia/stream1/chunk_43212123.ts
> mydomain.com/live/australia/stream2/chunk_43212123.ts
> mydomain.com/live/australia/stream3/chunk_43212123.ts
> ```
> Now think I want just the url excluded with chunk file becomes hashed and sharded:
> Just this part: "/live/australia/stream{1,2,3}/"
> Not :"/live/australia/stream{1,2,3}/chunk_43212123.ts"
>
> So by adjusting " p.set(by=KEY, key=bereq.url) " it would shard "bereq.url", it means "/live/australia/stream{1,2,3}/chunk_43212123.ts"

If you want to do that, you need to extract the relevant part and
compute a key from it:

> set bereq.http.my-key = bereq.http.host + regsub(bereq.url, "/chunk_.*", "");
> set bereq.backend = dir.backend(by=KEY, key=dir.key(bereq.http.my-key));

Below is a test case where both requests go through the same
backend:

---
varnishtest "shard by subset of url"

server s1 {
loop 2 {
rxreq
txresp
}
} -start

server s2 {
loop 2 {
rxreq
txresp
}
} -start

varnish v1 -vcl+backend {
import directors;

sub vcl_init {
new dir = directors.shard();
dir.add_backend(s1);
dir.add_backend(s2);
}

sub vcl_backend_fetch {
set bereq.http.shard-key = bereq.http.host +
regsub(bereq.url, "/chunk_.*", "");
set bereq.backend = dir.backend(by=KEY,
key=dir.key(bereq.http.shard-key));
}

sub vcl_backend_response {
set beresp.http.backend = beresp.backend;
}
} -start

client c1 {
txreq -url "/live/australia/stream1/chunk_123.ts"
rxresp
expect resp.http.backend == s1

txreq -url "/live/australia/stream1/chunk_456.ts"
rxresp
expect resp.http.backend == s1
} -run
---

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
Hi,
Thanks to you and all varnish team for such answers that helped me alot,
I read the default varnish cache configuration again:
https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl
and find out vcl_hash as follow:

```
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}

```
So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:)

```

sub vcl_hash {
hash_data(req.url);
return (lookup);
}

```

By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration?

________________________________
From: varnish-misc <varnish-misc-bounces+hrhosseini=hotmail.com@varnish-cache.org> on behalf of Hamidreza Hosseini <hrhosseini@hotmail.com>
Sent: Sunday, August 1, 2021 4:17 AM
To: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Best practice for caching scenario with different backend servers but same content

Hi,
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish,
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:

```
In varnish-1:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b10());



In varnish-2:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b10());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b9());


In varnish-3:
new hls_cluster = directors.fallback();
hls_cluster.add_backend(b9());
hls_cluster.add_backend(b1());
hls_cluster.add_backend(b2());
hls_cluster.add_backend(b3());
hls_cluster.add_backend(b4());
hls_cluster.add_backend(b5());
hls_cluster.add_backend(b6());
hls_cluster.add_backend(b7());
hls_cluster.add_backend(b8());
hls_cluster.add_backend(b10());

```
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,
What is varnish recommendation for this scenario?
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
On Sat, Aug 14, 2021 at 10:54 AM Hamidreza Hosseini
<hrhosseini@hotmail.com> wrote:
>
> Hi,
> Thanks to you and all varnish team for such answers that helped me alot,
> I read the default varnish cache configuration again:
> https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl
> and find out vcl_hash as follow:
>
> ```
> sub vcl_hash {
> hash_data(req.url);
> if (req.http.host) {
> hash_data(req.http.host);
> } else {
> hash_data(server.ip);
> }
> return (lookup);
> }
>
> ```
> So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:)
>
> ```
>
> sub vcl_hash {
> hash_data(req.url);
> return (lookup);
> }
>
> ```
>
> By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration?

In this case req.http.host usually refers to the the domain end users
resolve to find your varnish server (or other hops in front of it). It
is usually the same for every client, let's take www.myapp.com as an
example. If your varnish server is in front of multiple services, you
should be handling the different host headers explicitly. For exampe
if you have exactly two domains you should normalize them to some
canonical form. Using the same example domain that could be
www.myapp.com and static.myapp.com for instance.

In that case hashing the URL only would prevent you from adding new
domains through your Varnish server. It won't hurt if you know you
will only ever have one domain to deal with, but hashing the host will
also not hurt as long as you normalize it to a unique value.

You are correct that by default hashing the request appropriately will
help the shard director do the right thing out of the box. I remember
however that you only wanted to hash a subset of the URL for video
segments, so hashing the URL as-is won't provide the behavior you are
looking for.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
> In that case, hashing the URL only would prevent you from adding new
domains through your Varnish server. It won't hurt if you know you
will only ever have one domain to deal with, but hashing the host will
also not hurt as long as you normalize it to a unique value.

Hi,
Let me elaborate my architecture more:
I have some backend servers to serve hls fragments for video live stream,e.g:

```

hls_backend_01
hls_backend_02
hls_backend_03
hls_backend_04
hls_backend_05
hls_backend_06
hls_backend_07
hls_backend_08
hls_backend_09
hls_backend_10

```

There is same content on all hls backend servers, there are 5 varnish in front of them for caching
Now If I use round-robin director on Varnishes, because varnish would cache " req.http.host + req.url ", so for the same content but from different backends it would cache double! for example:
if varnish for the first request and "test.ts" file goes to "hls_backend_01" backend server, would cache it and
for the next request from other clients because it is using round-robin director
it goes to "hls_backend_02" and would cache the same file again due to different "req.http.host"
So now I have a solution to use Shard director based on "key=req.url" instead of round robin
another way is to use round robin but adjusting the hash vcl to something like bellow:

```

sub vcl_hash {
hash_data(req.url);
return (lookup);
}

```

In this way varnish just hash the "req.url" not "req.http.host"
So, Varnish would cache the content based on the content uniqueness not based on the difference between backends.
1. At first, I asked how I can normalize it, Is it possible at all according to what I said!?
Would you please explain it more with an example?

2. You give an example about other domains, In this case I do not understand what it has to do with the domain?

3.Maybe I'm thinking in wrong way because if varnish hash the data based on req.url : 'hash_data(req.url)' It shouldn't cache the same content but different backends again!
for example my request is :

http://varnish-01:/hls/test.ts
for first request it goes to "hls_backend_01" backend and cache it and for next request it goes to "hls_backend_02" backend,
so for each request it caches it again because backends are different?

Many Thanks, Hamidreza

________________________________
From: varnish-misc <varnish-misc-bounces+hrhosseini=hotmail.com@varnish-cache.org> on behalf of Dridi Boukelmoune <dridi@varni.sh>
Sent: Sunday, August 15, 2021 10:30 PM
To: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Re: Best practice for caching scenario with different backend servers but same content

On Sat, Aug 14, 2021 at 10:54 AM Hamidreza Hosseini
<hrhosseini@hotmail.com> wrote:
>
> Hi,
> Thanks to you and all varnish team for such answers that helped me alot,
> I read the default varnish cache configuration again:
> https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl
> and find out vcl_hash as follow:
>
> ```
> sub vcl_hash {
> hash_data(req.url);
> if (req.http.host) {
> hash_data(req.http.host);
> } else {
> hash_data(server.ip);
> }
> return (lookup);
> }
>
> ```
> So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:)
>
> ```
>
> sub vcl_hash {
> hash_data(req.url);
> return (lookup);
> }
>
> ```
>
> By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration?

In this case req.http.host usually refers to the the domain end users
resolve to find your varnish server (or other hops in front of it). It
is usually the same for every client, let's take www.myapp.com<http://www.myapp.com> as an
example. If your varnish server is in front of multiple services, you
should be handling the different host headers explicitly. For exampe
if you have exactly two domains you should normalize them to some
canonical form. Using the same example domain that could be
www.myapp.com<http://www.myapp.com> and static.myapp.com for instance.

In that case hashing the URL only would prevent you from adding new
domains through your Varnish server. It won't hurt if you know you
will only ever have one domain to deal with, but hashing the host will
also not hurt as long as you normalize it to a unique value.

You are correct that by default hashing the request appropriately will
help the shard director do the right thing out of the box. I remember
however that you only wanted to hash a subset of the URL for video
segments, so hashing the URL as-is won't provide the behavior you are
looking for.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Best practice for caching scenario with different backend servers but same content [ In reply to ]
On Mon, Aug 16, 2021 at 1:34 PM Hamidreza Hosseini
<hrhosseini@hotmail.com> wrote:
>
> > In that case, hashing the URL only would prevent you from adding new
> domains through your Varnish server. It won't hurt if you know you
> will only ever have one domain to deal with, but hashing the host will
> also not hurt as long as you normalize it to a unique value.
>
> Hi,
> Let me elaborate my architecture more:
> I have some backend servers to serve hls fragments for video live stream,e.g:
>
> ```
>
> hls_backend_01
> hls_backend_02
> hls_backend_03
> hls_backend_04
> hls_backend_05
> hls_backend_06
> hls_backend_07
> hls_backend_08
> hls_backend_09
> hls_backend_10
>
> ```
>
> There is same content on all hls backend servers, there are 5 varnish in front of them for caching
> Now If I use round-robin director on Varnishes, because varnish would cache " req.http.host + req.url ", so for the same content but from different backends it would cache double! for example:
> if varnish for the first request and "test.ts" file goes to "hls_backend_01" backend server, would cache it and
> for the next request from other clients because it is using round-robin director
> it goes to "hls_backend_02" and would cache the same file again due to different "req.http.host"
> So now I have a solution to use Shard director based on "key=req.url" instead of round robin
> another way is to use round robin but adjusting the hash vcl to something like bellow:
>
> ```
>
> sub vcl_hash {
> hash_data(req.url);
> return (lookup);
> }
>
> ```
>
> In this way varnish just hash the "req.url" not "req.http.host"
> So, Varnish would cache the content based on the content uniqueness not based on the difference between backends.
> 1. At first, I asked how I can normalize it, Is it possible at all according to what I said!?
> Would you please explain it more with an example?

In this case I think you are confusing "req.http.host" (host header)
with the backend host name.

For example, if you reach one of your 5 Varnish servers via
www.example.com that's what clients will use and that's what
req.http.host will contain.

Your backends FQDNs could be something like this:

- hls01.internal.example.com
- hls02.internal.example.com
- hls03.internal.example.com
- ...
- hls10.internal.example.com

As the example suggests, these domains should not be directly reached
by clients if your goal is to proxy them with Varnish. Those internal
FQDNs should have no effect on the cache key populated with
hash_data(...).

> 2. You give an example about other domains, In this case I do not understand what it has to do with the domain

Let's say your clients can reach either example.com or www.example.com
for the same service, or tomorrow you add more than your HLS service
behind Varnish you may very well receive multiple host headers.

> 3.Maybe I'm thinking in wrong way because if varnish hash the data based on req.url : 'hash_data(req.url)' It shouldn't cache the same content but different backends again!
> for example my request is :

In this case you are "hashing" the client request with hash_data(...)
and it has nothing to do with backend selection. The fallback director
will precisely not do any kind of traffic balancing since its purpose
is to always select the first healthy backend in the insertion order.
The shard director may rely on the request hash or other criteria as
we already covered.

> http://varnish-01:/hls/test.ts
> for first request it goes to "hls_backend_01" backend and cache it and for next request it goes to "hls_backend_02" backend,
> so for each request it caches it again because backends are different?

All subsequent requests to http://varnish-01:/hls/test.ts should go to
the same hls_backend_01 backend with the shard director. As long as
there are no other criteria than the ones we already discussed. If you
want consistency across all your Varnish servers, you should configure
your shard directors identically, with the backends added in the same
order (unlike your initial VCL example using the fallback director).

Best,
Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc