Mailing List Archive

Hi Jérôme

VMs are no go for big content companies except Microsoft. You can run
Microsoft CDN on VM but rest of the content will need to be cached. You can
actually install this yourself

Depending on how much traffic do you have , you may be able to get
facebook, youtube, netflix caches i think minimum bw requirement changes
per region

Good luck

On Thu, Jan 18, 2024 at 06:53 Jérôme Nicolle <jerome@ceriz.fr> wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

On Thu, Jan 18, 2024 at 12:53:19PM +0100,
Jérôme Nicolle <jerome@ceriz.fr> wrote
a message of 36 lines which said:

> - Low redundancy of old cables (2)
> - Total service loss when both cables are down because of congestion on
> satelite backups

A problem which is not often mentioned is that most (all?) "local
caches" (CDN, DNS resolvers, etc) do not have an "offline mode" (or
"disconnected-from-master mode"). During an outage, they continue to
work for some time then break suddenly, in a not-friendly way, serving
various error messages instead of old data and/or useful
messages. (For instance, the DNS resolver may not be able to serve
stale answers.)

The time during which they can continue to work when they are
disconnected from their master is typically undocumented (except for
the DNS), and discovered only when there is a long outage.

Making the Internet work better with sometimes-broken connectivity is
still an area of research.

Many CDNs have hardware options for self hosted caches. I think there would
likely be concerns about <20G of connectivity to those caches though. With
an incorrect setup, you could end up maxing out those links just with cache
fill traffic itself.


On Thu, Jan 18, 2024 at 6:54?AM Jérôme Nicolle <jerome@ceriz.fr> wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Hello Mehmet,

Le 18/01/2024 à 12:58, Mehmet a écrit :
> VMs are no go for big content companies except Microsoft. You can run
> Microsoft CDN on VM but rest of the content will need to be cached. You
> can actually install this yourself

I've already read most docs for caching servers provided by major
actors. What I'm mostly concerned about is their ability to peer with
multiple AS on the local IXP, as to not over-replicate them.

Should I establish a dedicated network peering on the IXP ? Or will they
come with their own ASNs ? The peering case is quite not documented on
publicly available specs, if even possible.

> Depending on how much traffic do you have , you may be able to get
> facebook, youtube, netflix caches i think minimum bw requirement changes
> per region

Those I'm nearly sure I could get, if I can pool caches amongst ISPs.
The current constraints are issues to any content provider, not just for
local ISPs.

Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Hi Gael,

Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
> Friends from PCH (www.pch.net <http://www.pch.net>) operate backend
> services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They
> would be very happy to help.

I'm sure they would, I'm a big fan of their work BTW. Though hosting
them in a densely connected area isn't the same as it will in remote
locations, I guess there could be some work to be done to get it running
properly, as Stephane wrote.

How would you think we could work on that ? I mean, disconnected or
extremely high latency scenarii should be on a research roadmap by
SpaceX' standards, right ? ;-)

Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Some will work directly on the IX via BGP. Others have to go behind a member of the IX.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Jérôme Nicolle" <jerome@ceriz.fr>
To: "Mehmet" <mehmet@akcin.net>
Cc: nanog@nanog.org
Sent: Thursday, January 18, 2024 8:38:31 AM
Subject: Re: Shared cache servers on an island's IXP

Hello Mehmet,

Le 18/01/2024 à 12:58, Mehmet a écrit :
> VMs are no go for big content companies except Microsoft. You can run
> Microsoft CDN on VM but rest of the content will need to be cached. You
> can actually install this yourself

I've already read most docs for caching servers provided by major
actors. What I'm mostly concerned about is their ability to peer with
multiple AS on the local IXP, as to not over-replicate them.

Should I establish a dedicated network peering on the IXP ? Or will they
come with their own ASNs ? The peering case is quite not documented on
publicly available specs, if even possible.

> Depending on how much traffic do you have , you may be able to get
> facebook, youtube, netflix caches i think minimum bw requirement changes
> per region

Those I'm nearly sure I could get, if I can pool caches amongst ISPs.
The current constraints are issues to any content provider, not just for
local ISPs.

Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Hi Tom,

Le 18/01/2024 à 15:20, Tom Beecher a écrit :
> Many CDNs have hardware options for self hosted caches. I think there
> would likely be concerns about <20G of connectivity to those caches
> though. With an incorrect setup, you could end up maxing out those links
> just with cache fill traffic itself.

In a case where these servers are on a dedicated network peering with
the ISPs, I think it would be safe to throttle the sync feeds to not
saturate actual uplinks.

At least, that we can do, but throttling uncached content to customers
is forbidden (net neutrality).

Though Netflix is supposedly sending pre-loaded servers, and I think
that - in this location - it's gonna mean a lot already. The quastion is
: how would the servers peer with local ISPs.

Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Hosting authoritative and recursive dns servers at the IXP would
drastically improve the experience of users most of the time.

Of course, Stephane considerations are correct and there’s no solution for
when global connectivity is lost and responses will stop being sent.

Gaël


On Thu 18 Jan 2024 at 14:42, Jérôme Nicolle <jerome@ceriz.fr> wrote:

> Hi Gael,
>
> Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
> > Friends from PCH (www.pch.net <http://www.pch.net>) operate backend
> > services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They
> > would be very happy to help.
>
> I'm sure they would, I'm a big fan of their work BTW. Though hosting
> them in a densely connected area isn't the same as it will in remote
> locations, I guess there could be some work to be done to get it running
> properly, as Stephane wrote.
>
> How would you think we could work on that ? I mean, disconnected or
> extremely high latency scenarii should be on a research roadmap by
> SpaceX' standards, right ? ;-)
>
> Best regards,
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Jérôme Nicolle wrote on 18/01/2024 14:38:
> Those I'm nearly sure I could get, if I can pool caches amongst ISPs.
> The current constraints are issues to any content provider, not just for
> local ISPs.

two issues here: the smaller issue is that CDNs sometimes want their own
routable IP address blocks, especially if they're connecting directly to
the IXP, which usually means /24 in practice. It doesn't always happen,
and sometimes the CDN is happy to use provider address space (i.e. IXP),
or smaller address blocks. But it's something to note.

The bigger issue is: who pays the transit costs for the CDN's cache-fill
requirements? CDNs typically won't pay for cache-fill for installations
like this, and if one local ISP is pulling disproportionate quantities
of data compared to other ISPs at the IXP, then this can cause problems
unless there's an shared billing mechanism built in.

Nick

Hi Nick,

Thanks for your remarks. It's actually an ongoing discussion.

Le 18/01/2024 à 18:24, Nick Hilliard a écrit :
> two issues here: the smaller issue is that CDNs sometimes want their own
> routable IP address blocks, especially if they're connecting directly to
> the IXP, which usually means /24 in practice. It doesn't always happen,
> and sometimes the CDN is happy to use provider address space (i.e. IXP),
> or smaller address blocks. But it's something to note.

I'd rather have CDN use some of their anycast /24 to peer with the IX,
with a back-end connectivity for their control-plane and back-feeding.

> The bigger issue is: who pays the transit costs for the CDN's cache-fill
> requirements? CDNs typically won't pay for cache-fill for installations
> like this, and if one local ISP is pulling disproportionate quantities
> of data compared to other ISPs at the IXP, then this can cause problems
> unless there's an shared billing mechanism built in.

We're willing to provide a dedicated LAN, with routed access, to fill
caches and administer the machines. It would be fully dissociated from
the IXP though, unless we could find a way to make it work and as to
meet extra requirements upon redundancy.

Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14