Mailing List Archive

Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs
All of the sudden Varnish fails to start in my development environment, and gives me the following error message:

Message from VCC-compiler:
Backend host "redacted-hostname": resolves to too many addresses.
Only one IPv4 and one IPv6 are allowed.
Please specify which exact address you want to use, we found all of these:
555.123.123.3:80
555.123.123.3:80
555.123.123.3:80

I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail.

The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs.
Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs [ In reply to ]
The fact the IPs are identical is weird, but I wouldn't be surprised if the
dns entry actually contained 3 identical IPs.

> Shouldn't Varnish be able to figure out that in that case it can just
choose any one and it will work as expected?

Shouldn't your DNS entries be clean? ;-)

Honestly, if the IP(s) behind the service name is liable to change, you
shouldn't use a dynamic backend because Varnish resolves the IP when the
VCL is loaded, so if the IP changes behind your back, Varnish won't follow
it, and you'll be screwed.
Instead, you should use dynamic backends, of which there are a handful:
- dynamic <https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's
been around for ages, it's battle-tested, and it's included in the oficial
Varnish Docker image <https://hub.docker.com/_/varnish>
- udo+activedns
<https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>,
by Varnish Software: the design is slightly different and allows you to
specify pretty much any load-balancing policy you might need. You'll need a
subscription but you'll get excellent support (disclaimer, I'm an ex
employee)
- reqwest
<https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>,
by yours truly: the interface focuses on providing a simple experience and
a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)

As you can see, the static backend's reluctance to fully handle DNS has
been a fertile ground for vmods :-)

--
Guillaume Quintard


On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun@hotmail.com> wrote:

> All of the sudden Varnish fails to start in my development environment,
> and gives me the following error message:
>
> Message from VCC-compiler:
> Backend host "redacted-hostname": resolves to too many addresses.
> Only one IPv4 and one IPv6 are allowed.
> Please specify which exact address you want to use, we found all of these:
> 555.123.123.3:80
> 555.123.123.3:80
> 555.123.123.3:80
>
> I have changed the hostname and the IP above to not expose our server, but
> all three IP numbers are 100% identical. Shouldn't Varnish be able to
> figure out that in that case it can just choose any one and it will work as
> expected? It really should remove duplicates, and only if there are more
> than one non-duplicate IP then it should fail.
>
> The problem is that the backend host is a so called "app service" in
> Microsoft Azure, which is basically a platform as a service (PaaS), where
> Microsoft handles the networking including the domain name (no user access
> it directly). I have no idea why it suddenly resolves to multiple duplicate
> IPs.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs [ In reply to ]
On Wed, Apr 19, 2023 at 2:44?PM Guillaume Quintard
<guillaume.quintard@gmail.com> wrote:
>
> The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs.
>
> > Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected?
>
> Shouldn't your DNS entries be clean? ;-)

It should, but Varnish could probably make an effort here.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs [ In reply to ]
> Shouldn't your DNS entries be clean? ;-)

Preferably, but I blame Microsoft here ????

The problem went away by itself when I tried starting again like half an hour later or so, so I guess it was a temporary glitch in the matrix.

As far as I understand it, the IPs of these machines only change if they are deleted and created again. We do it occasionally in test/staging, and there we can live with Varnish needing to be restarted. In production we don't really delete them once they are properly setup, unless there is some major problem and then a restart of the load balanced varnish servers should not be a concern.

Thanks for your vmod suggestions! I will check them out. The dynamic one seems like the only one that supports community edition LTS 6.0. The documentation seems a bit lacking (no full VCL example), but I guess I could use their test cases as examples.

________________________________
From: Guillaume Quintard <guillaume.quintard@gmail.com>
Sent: Wednesday, April 19, 2023 4:42 PM
To: Batanun B <batanun@hotmail.com>
Cc: varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
Subject: Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs

The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs.

> Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected?

Shouldn't your DNS entries be clean? ;-)

Honestly, if the IP(s) behind the service name is liable to change, you shouldn't use a dynamic backend because Varnish resolves the IP when the VCL is loaded, so if the IP changes behind your back, Varnish won't follow it, and you'll be screwed.
Instead, you should use dynamic backends, of which there are a handful:
- dynamic<https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's been around for ages, it's battle-tested, and it's included in the oficial Varnish Docker image<https://hub.docker.com/_/varnish>
- udo+activedns<https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>, by Varnish Software: the design is slightly different and allows you to specify pretty much any load-balancing policy you might need. You'll need a subscription but you'll get excellent support (disclaimer, I'm an ex employee)
- reqwest<https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>, by yours truly: the interface focuses on providing a simple experience and a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)

As you can see, the static backend's reluctance to fully handle DNS has been a fertile ground for vmods :-)

--
Guillaume Quintard


On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun@hotmail.com<mailto:batanun@hotmail.com>> wrote:
All of the sudden Varnish fails to start in my development environment, and gives me the following error message:

Message from VCC-compiler:
Backend host "redacted-hostname": resolves to too many addresses.
Only one IPv4 and one IPv6 are allowed.
Please specify which exact address you want to use, we found all of these:
555.123.123.3:80
555.123.123.3:80
555.123.123.3:80

I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail.

The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs [ In reply to ]
> The documentation seems a bit lacking (no full VCL example), but I guess
I could use their test cases as examples.

https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583
maybe?
I'm sure Nils will pipe up here if you need help, and if you want more
synchronous assistance, there's always the discord channel
<https://varnish-cache.org/support/>.

> The dynamic one seems like the only one that supports community edition
LTS 6.0.

Yes, of the three, that's the only one that will support that one (VS is
focused on the Enterprise version, and I lack the time to port vmods on 6.0
(but I'll welcome the help)).

Cheers,

--
Guillaume Quintard


On Wed, Apr 19, 2023 at 9:02?AM Batanun B <batanun@hotmail.com> wrote:

> > Shouldn't your DNS entries be clean? ;-)
>
> Preferably, but I blame Microsoft here ????
>
> The problem went away by itself when I tried starting again like half an
> hour later or so, so I guess it was a temporary glitch in the matrix.
>
> As far as I understand it, the IPs of these machines only change if they
> are deleted and created again. We do it occasionally in test/staging, and
> there we can live with Varnish needing to be restarted. In production we
> don't really delete them once they are properly setup, unless there is some
> major problem and then a restart of the load balanced varnish servers
> should not be a concern.
>
> Thanks for your vmod suggestions! I will check them out. The dynamic one
> seems like the only one that supports community edition LTS 6.0. The
> documentation seems a bit lacking (no full VCL example), but I guess I
> could use their test cases as examples.
>
> ------------------------------
> *From:* Guillaume Quintard <guillaume.quintard@gmail.com>
> *Sent:* Wednesday, April 19, 2023 4:42 PM
> *To:* Batanun B <batanun@hotmail.com>
> *Cc:* varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org>
> *Subject:* Re: Varnish won't start because backend host resolves to too
> many addresses, but they are all identical IPs
>
> The fact the IPs are identical is weird, but I wouldn't be surprised if
> the dns entry actually contained 3 identical IPs.
>
> > Shouldn't Varnish be able to figure out that in that case it can just
> choose any one and it will work as expected?
>
> Shouldn't your DNS entries be clean? ;-)
>
> Honestly, if the IP(s) behind the service name is liable to change, you
> shouldn't use a dynamic backend because Varnish resolves the IP when the
> VCL is loaded, so if the IP changes behind your back, Varnish won't follow
> it, and you'll be screwed.
> Instead, you should use dynamic backends, of which there are a handful:
> - dynamic <https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's
> been around for ages, it's battle-tested, and it's included in the oficial
> Varnish Docker image <https://hub.docker.com/_/varnish>
> - udo+activedns
> <https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>,
> by Varnish Software: the design is slightly different and allows you to
> specify pretty much any load-balancing policy you might need. You'll need a
> subscription but you'll get excellent support (disclaimer, I'm an ex
> employee)
> - reqwest
> <https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>,
> by yours truly: the interface focuses on providing a simple experience and
> a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)
>
> As you can see, the static backend's reluctance to fully handle DNS has
> been a fertile ground for vmods :-)
>
> --
> Guillaume Quintard
>
>
> On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun@hotmail.com> wrote:
>
> All of the sudden Varnish fails to start in my development environment,
> and gives me the following error message:
>
> Message from VCC-compiler:
> Backend host "redacted-hostname": resolves to too many addresses.
> Only one IPv4 and one IPv6 are allowed.
> Please specify which exact address you want to use, we found all of these:
> 555.123.123.3:80
> 555.123.123.3:80
> 555.123.123.3:80
>
> I have changed the hostname and the IP above to not expose our server, but
> all three IP numbers are 100% identical. Shouldn't Varnish be able to
> figure out that in that case it can just choose any one and it will work as
> expected? It really should remove duplicates, and only if there are more
> than one non-duplicate IP then it should fail.
>
> The problem is that the backend host is a so called "app service" in
> Microsoft Azure, which is basically a platform as a service (PaaS), where
> Microsoft handles the networking including the domain name (no user access
> it directly). I have no idea why it suddenly resolves to multiple duplicate
> IPs.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs [ In reply to ]
> https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583
> maybe?
>

> I'm sure Nils will pipe up here if you need help, and if you want more
> synchronous assistance, there's always the discord channel
> <https://varnish-cache.org/support/>.

Thanks! :)