Mailing List Archive

Marco Davids via NANOG <nanog@nanog.org> writes:

> It turns out that there underlying CDN's with domain names such as
> ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net',
> that reside on authoritative name servers that *only* have an IPv4
> address.

Fastly does have IPv6 enabled authoritative DNS server but it looks like
it's not the default.

I ran into this some time ago with deb.debian.org on an IPv6 only Debian
VM with a locally installed resolver. I opened a ticket which was closed
in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296

After some ranting and shouting it now works but a couple of days
ago I ran in the same problem while trying to install something via
pip. fles.pythonhosted.org is also using fastly.

> I guess my question is simple: Why?

I'm asking myself the same question.

> Are there good architectural reason for this? Or is it just something
> that is overlooked and forgotten about?

I don't think it was overlooked or forgotten. More along

"We have always done it this way", "We had problems enabling IPv6 (ages
ago)" or something else you can find on https://ipv6excuses.com/.

Jens
--
----------------------------------------------------------------------------
| Delbrueckstr. 41 | 12051 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@quux.de | --------------- |
----------------------------------------------------------------------------

Hi Jens,

Op 22-10-21 om 14:03 schreef Jens Link:

> I ran into this some time ago with deb.debian.org on an IPv6 only Debian
> VM with a locally installed resolver. I opened a ticket which was closed
> in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296

Just for the record; your issue is slightly different:

You wrote:

"deb.debian.org is a CNAME for debian.map.fastly.net. There are no AAAA
records for fastly.net so any DNS querys from an IPv6 only resolver will
not work."

At the moment debian.map.fastly.net has an AAAA-record though.

The thing is; the authoritative name servers of fastly.net are only
willing to hand out that AAAA-record via IPv4. So it still doesn't work
with the (locally installed) IPv6-only resolver ;-)

Cheers,

--
Marco

On 10/22/21 14:03, Jens Link wrote:

> I don't think it was overlooked or forgotten. More along
>
> "We have always done it this way", "We had problems enabling IPv6 (ages
> ago)" or something else you can find on https://ipv6excuses.com/.

I think it's a combination of both... they tried back in the day, it
broke, and they "parked" it for later.

When Marketing and The World were happy to see that
www.insert-favorite-content-url-here.com had AAAA and IPv6 PTR records,
who cared whether boring, little-known FQDN's were remembered or not.

And then, all the engineers moved on to some other gig, where they did
better because, why not?

Mark.

On second thoughts...

I seem to have been confused by the 'no AAAA records for fastly.net' (as
a DNS-purist: that should have said "ns[1234].fastly.net" instead, to
make it relevant). ;-)

>> I ran into this some time ago with deb.debian.org

Right.

So please ignore:

> Just for the record; your issue is slightly different:
>
> You wrote:
>
> "deb.debian.org is a CNAME for debian.map.fastly.net. There are no AAAA
> records for fastly.net so any DNS querys from an IPv6 only resolver will
> not work."


--
Marco

Hello,

client side IPv6-only is one thing, but IPv6-only recursive DNS
resolution is probably so niche that content providers and CDN's do
not particularly care at this point in time.

On the other hand, there is probably no good reason to run
authoritative DNS servers without IPv6 connectivity.


Lukas

On Fri, 22 Oct 2021, 13:03 Jens Link, <lists@quux.de> wrote:

> I ran into this some time ago with deb.debian.org on an IPv6 only Debian
> VM with a locally installed resolver. I opened a ticket which was closed
> in record time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961296
>
> After some ranting and shouting it now works
>

I'm going to post the relevant message here:

> Sometimes I wonder why I report bugs..
>
> But your answer was the answer I was expecting. Thanks for noting.
>
> So I can summarize this as "The Debian Project doesn't care if IPv6 is
working"?

Jens, you went into that ticket looking for a fight, in a place staffed by
largely unpaid volunteers, choosing to belittle their efforts and then
attempting to shame them into action.

You even chose to mark the bug severity higher than the default, despite
you having chosen that mirror for your install.
https://www.debian.org/Bugs/Developer#severities

Marco explained to you that the mirror network has plenty of selections to
make, but you choose to make a fuss about one supported option not working
to a standard the rest of the community pretty much agrees is nowhere near
attainable at this time. Using netselect
https://packages.debian.org/stable/net/netselect to choose a reachable
mirror with the lowest latency would have easily mitigated this issue for
you.

DNS64 and NAT64 are going to be with us for a very long time, and if you
refuse to support IPv4 even through a translation layer then it is clear
you are acting against the interests of further IPv6 adoption by
associating IPv6 issues with zealotry. The apathy sometimes associated with
IPv6 support today is because of this perceived high effort low reward
nature of confrontation.

I would strongly advise you apologise to Marco for your grandstanding, and
adopt a more constructive way of furthering your ideology. The NANOG code
of conduct clearly states:

https://www.nanog.org/about/code-conduct/

> In the spirit of mutual respect and collaboration, NANOG does not
tolerate any unwelcome behavior, including but not limited to:
> * Aggressively pushing your own services, products, or causes.
> [...]

Please join the rest of us in advocating for IPv6 adoption, rather than the
current bullying tactics you seem to be choosing that wins the battle and
loses the war. We are all friends* here. You can be a great asset in this
effort we should all seek.

M


*FSVO friends, obvs.

Hi everyone, goedenmiddag Marco!

On Fri, Oct 22, 2021 at 01:40:42PM +0200, Marco Davids via NANOG wrote:
> We currently live in times where is actually fun to go IPv6-only. In my
> case, as in: running a FreeBSD kernel compiled without the IPv4-stack.

Indeed, this is fun experimentation. Shaking the (source code) trees
through excercises like these is a valuable way to identify gaps.

> It turns out that there underlying CDN's with domain names such as
> ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
> reside on authoritative name servers that *only* have an IPv4 address.

As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
Fastly is working hard with select customers and friends to support IPv6
for everyone.

> I guess my question is simple: Why?
>
> Are there good architectural reason for this? Or is it just something
> that is overlooked and forgotten about?

The universal deployment of IPv6 appears to be a multi-decennial
multigenerational project. Allow me to shed some light on various
aspects.

One of the challenges faced by those wishing to deploy IPv6 (compared to
IPv4) is how from a BGP Default-Free Zone perspective, IPv4 and IPv6 are
not alike at all! The Internet's IPv6 routing topology is vastly
different from the IPv4 topology.

The above phenomenon is perfectly understandable following from the fact
that IPv4 predates IPv6 - and IP networks grow as they grow. In a
perfect world the IPv6 network would grow perfectly congruent alongside
the global IPv4 network. In this perfect world indeed IPv6 can "just be
enabled", and used whenever available!

Unfortunately the reality of the situation is far more chaotic! For
example if you look at PeeringDB's 'netixlan' table, large discrepancies
between the number of absent IPv4 entries and absent IPv6 entries are
visible:

$ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c '"ipaddr4": null'
1286

$ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c '"ipaddr6": null'
8160

From the above it's implied that the density of the 'IPv4 mesh' is much
higher than the density and diversity of the 'IPv6 mesh', simply because
more operators present more IPv4 traffic-exchange opportunities to other
operators - compared to IPv6. This has performance implications.

Another aspect that flabbergasts me anno 2021 is how there *still* are
BGP peering disputes between (more than two) major global internet service
providers in which IPv6 is 'held hostage' as part of slow commercial
negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Anyway, back to your question: content delivery networks who leverage
all possible technical knobs and buttons to increase performance (such
as BGP traffic engineering) might be reluctant to offer IPv6 services
"as if they are the same as IPv4". More study is required.

Tl;DR - work in progress! :-)

Kind regards,

Job

ps. Have you tried running an IPv6-only RPKI validator? About 1.4% of
RPKI VRPs appears to be 'missing' in IPv6-only environments :-/

On 10/22/21 11:13 AM, Job Snijders via NANOG wrote:
> Another aspect that flabbergasts me anno 2021 is how there *still* are
> BGP peering disputes between (more than two) major global internet service
> providers in which IPv6 is 'held hostage' as part of slow commercial
> negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Even the DNS root servers are not 100% reachable via IPv6. I would think IANA
would have some standard about reachability for root operators.

FWIW, I just was able to change my home office internet (I reside in the most
densely populated county of Florida). The new provider sold me a dual stack
connection, however when they came to deliver it, there was no IPv6 as
promised. After spending almost a week playing phone tag, I finally got some
one with clue. I was told they have no support if IPv6 and no plans to ever
support IPv6 as there is no way to monetize it.

This leaves me in the same position as my prior circuit via the local cable
co. (no plans to offer IPv6) but at least it's faster than the 2 meg up cable
service.

Until IPv6 becomes provides a way to make money for the ISP, I don't see it
being offered outside of the datacenter.
--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

On Friday, 22 October, 2021 16:45, "Bryan Fields" <Bryan@bryanfields.net> said:

> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
> being offered outside of the datacenter.

I don't think it'll ever make money, but I think it will reduce costs. CGNAT boxes cost money, operating them costs money, dealing with the support fallout from them costs money. Especially in the residential space, where essentially if the customer calls you, ever, you just blew years' worth of margin.

My residential ISP here in the UK routes me (and every other subscriber) a /56 without being asked. (Their supplied CPE router just puts the first /64 on the LAN and refuses to process PD requests to hand out any of the other /64s, but baby steps...)

Cheers,
Tim.

Hi again,

Op 22-10-21 om 17:13 schreef Job Snijders:

> Tl;DR

Not at all. This was a very interesting read! Thank you.

While pondering over it, I noticed that the ns[1234].fastly.net servers
are nicely anycasted throughout the globe. If anyone could turn on IPv6
on their authoritatives without therisk of loosing too much performance,
I reckon it would be them... our Cloudflare. But they already did it. ;-).

> work in progress!

I have good hopes. Rumour has it that Fastly employs some very smart
people. I'm sure we'll see nice things happening when the time is right.

--
Marco

On 10/22/21 17:45, Bryan Fields wrote:

>
> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
> being offered outside of the datacenter.

It is being offered, it's just not being adopted.

We deliver an IPv6 /126 p2p and /56 or /48 onward assignment to all our
DIA customers. No interest.

We deliver an IPv6 /125 p2p and eBGP session to all our IP Transit
customers. 5/10 are interested.

You can only do what you can only do.

Mark.

On 10/22/21 18:08, tim@pelican.org wrote:

> I don't think it'll ever make money, but I think it will reduce costs. CGNAT boxes cost money, operating them costs money, dealing with the support fallout from them costs money. Especially in the residential space, where essentially if the customer calls you, ever, you just blew years' worth of margin.

The problem is accurately modelling cost reduction using native IPv6 in
lieu of CG-NAT is hard when the folk that need convincing are the CFO's.

They are more used to "spend 1 to get 2". Convincing them to "save 2 by
spending 1" - not as easy as one may think.

Mark.

Bryan,

On Oct 22, 2021, at 11:45 AM, Bryan Fields <Bryan@bryanfields.net> wrote:
> On 10/22/21 11:13 AM, Job Snijders via NANOG wrote:
>> Another aspect that flabbergasts me anno 2021 is how there *still* are
>> BGP peering disputes between (more than two) major global internet service
>> providers in which IPv6 is 'held hostage' as part of slow commercial
>> negotiations. Surely end-to-end IPv6 connectivity should be a priority?
>
> Even the DNS root servers are not 100% reachable via IPv6.

Excepting temporary failures, they are as far as I am aware. Why do you think they aren’t?

> I would think IANA
> would have some standard about reachability for root operators.

I think you might misunderstand relationships here.

The IANA team’s standards are what the community defines. In the case of the root operators, RFC 7720 says “root service” must be available via IPv6 and RSSAC-001 (“Service Expectations of Root Servers”, https://www.icann.org/en/system/files/files/rssac-001-root-service-expectations-04dec15-en.pdf) says:

"[E.3.1-B] Individual Root Servers will deliver the service in conformance to IETF standards and requirements as described in RFC 7720 [4] and any other IETF standards-defined Internet Protocol as deemed appropriate."

So, in theory, all the root servers should be available via IPv6 and, as far as I can tell, they are.

However, the IANA team is not the enforcement arm of the Internet. If a root server operator chooses to not abide by RFC 7720, there is nothing the IANA team can do unilaterally other than make the root server operator aware of the fact.

> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
> being offered outside of the datacenter.

Different markets, different approaches. In the areas I’ve lived in Los Angeles, commodity residential service via AT&T (1 Gbps up/down fiber) and Spectrum (varying speeds) is dual stack by default (as far as I can tell). I suspect all it would take would be one of the providers in your area to offer IPv6 and advertise the fact in their marketing to cause the others to fall into line.

Regards,
-drc

On Fri, Oct 22, 2021 at 8:48 AM Bryan Fields <Bryan@bryanfields.net> wrote:

> On 10/22/21 11:13 AM, Job Snijders via NANOG wrote:
> > Another aspect that flabbergasts me anno 2021 is how there *still* are
> > BGP peering disputes between (more than two) major global internet
> service
> > providers in which IPv6 is 'held hostage' as part of slow commercial
> > negotiations. Surely end-to-end IPv6 connectivity should be a priority?
>
> Even the DNS root servers are not 100% reachable via IPv6. I would think
> IANA
> would have some standard about reachability for root operators.
>
> FWIW, I just was able to change my home office internet (I reside in the
> most
> densely populated county of Florida). The new provider sold me a dual
> stack
> connection, however when they came to deliver it, there was no IPv6 as
> promised. After spending almost a week playing phone tag, I finally got
> some
> one with clue. I was told they have no support if IPv6 and no plans to
> ever
> support IPv6 as there is no way to monetize it.
>
> This leaves me in the same position as my prior circuit via the local cable
> co. (no plans to offer IPv6) but at least it's faster than the 2 meg up
> cable
> service.
>
> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
> being offered outside of the datacenter.


87% of mobiles in the usa are ipv6

https://www.worldipv6launch.org/measurements/





> --
> Bryan Fields
>
> 727-409-1194 - Voice
> http://bryanfields.net
>

And ipv4 I presume so there is still easier and cost less money to just go with that.
From our point as an MSP no customer has a requirement that they want to be able to be reached via IPV6 so it’s still cheaper to buy up IPV4 address space and do a lot of nat than to convert all our services to function properly with IPV6.
Sure one could argue that they should have been made that way from the beginning but without customer demand why would we spend the money?

//Gustav


> 23 okt. 2021 kl. 15:33 skrev Ca By <cb.list6@gmail.com>:
>
> ?
>
>
>> On Fri, Oct 22, 2021 at 8:48 AM Bryan Fields <Bryan@bryanfields.net> wrote:
>> On 10/22/21 11:13 AM, Job Snijders via NANOG wrote:
>> > Another aspect that flabbergasts me anno 2021 is how there *still* are
>> > BGP peering disputes between (more than two) major global internet service
>> > providers in which IPv6 is 'held hostage' as part of slow commercial
>> > negotiations. Surely end-to-end IPv6 connectivity should be a priority?
>>
>> Even the DNS root servers are not 100% reachable via IPv6. I would think IANA
>> would have some standard about reachability for root operators.
>>
>> FWIW, I just was able to change my home office internet (I reside in the most
>> densely populated county of Florida). The new provider sold me a dual stack
>> connection, however when they came to deliver it, there was no IPv6 as
>> promised. After spending almost a week playing phone tag, I finally got some
>> one with clue. I was told they have no support if IPv6 and no plans to ever
>> support IPv6 as there is no way to monetize it.
>>
>> This leaves me in the same position as my prior circuit via the local cable
>> co. (no plans to offer IPv6) but at least it's faster than the 2 meg up cable
>> service.
>>
>> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
>> being offered outside of the datacenter.
>
> 87% of mobiles in the usa are ipv6
>
> https://www.worldipv6launch.org/measurements/
>
>
>
>
>>
>> --
>> Bryan Fields
>>
>> 727-409-1194 - Voice
>> http://bryanfields.net

> On Oct 23, 2021, at 8:30 AM, Ca By <cb.list6@gmail.com> wrote:
>
> 87% of mobiles in the usa are ipv6
>
> https://www.worldipv6launch.org/measurements/ <https://www.worldipv6launch.org/measurements/>
>

Agreed. When they have to connect to an IPv4 only host, they do some type of AFTR. These devices have never known a world outside of this situation. That is a major difference.


>
>
>
> --
> Bryan Fields
>
> 727-409-1194 - Voice
> http://bryanfields.net <http://bryanfields.net/>

I think you will find that there are some places in which getting IPv6 network service has been difficult, and as a result even IPv6-capable equipment is not reachable by IPv6. Those are, however, few and far between.

Sent using a machine that autocorrects in interesting ways...

> On Oct 23, 2021, at 6:04 AM, David Conrad <drc@virtualized.org> wrote:
>
> So, in theory, all the root servers should be available via IPv6 and, as far as I can tell, they are.

On Sat, Oct 23, 2021, 15:17 Fred Baker <fredbaker.ietf@gmail.com> wrote:

> I think you will find that there are some places in which getting IPv6
> network service has been difficult, and as a result even IPv6-


Fred, do you mean places like, all of Verizon FiOS?


capable equipment is not reachable by IPv6. Those are, however, few and far
> between.
>
> Sent using a machine that autocorrects in interesting ways...
>
> > On Oct 23, 2021, at 6:04 AM, David Conrad <drc@virtualized.org> wrote:
> >
> > So, in theory, all the root servers should be available via IPv6 and, as
> far as I can tell, they are.
>

On 10/23/21 9:03 AM, David Conrad wrote:
> Bryan,
>
>> Even the DNS root servers are not 100% reachable via IPv6.
>
> Excepting temporary failures, they are as far as I am aware. Why do you
> think they aren’t?

I can't reach C, 2001:500:2::c, from many places in v6 land. My home and
secondary data center can't reach it, but my backup VM's at another data
center can.

<snip>

> However, the IANA team is not the enforcement arm of the Internet. If a
> root server operator chooses to not abide by RFC 7720, there is nothing the
> IANA team can do unilaterally other than make the root server operator
> aware of the fact.

Surely IANA has the power to compel a root server operator to abide by policy
or they lose the right to be a root server?

>> Until IPv6 becomes provides a way to make money for the ISP, I don't see
>> it being offered outside of the datacenter.
>
> Different markets, different approaches. In the areas I’ve lived in Los
> Angeles, commodity residential service via AT&T (1 Gbps up/down fiber) and
> Spectrum (varying speeds) is dual stack by default (as far as I can tell).
> I suspect all it would take would be one of the providers in your area to
> offer IPv6 and advertise the fact in their marketing to cause the others to
> fall into line.

Prior ISP charged me $15/month per IPv4 address and a mandatory router rent of
$10/month. New one gets $5/month per IPv4 address. The reason for this is IP
scarcity. They have plenty of v4 space, so this allows them to charge for it.
v6 isn't going to make them any more money as they can't charge for it.


--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

On 10/23/21 9:30 AM, Ca By wrote:
>> Until IPv6 becomes provides a way to make money for the ISP, I don't see it
>> being offered outside of the datacenter.
>
> 87% of mobiles in the usa are ipv6
>
> https://www.worldipv6launch.org/measurements/

Mobile is different, v6 makes financial sense as CG NAT doesn't scale to 400m
cell phones in north America. (does NANOG scope include Mexico?)

That said most (all) IPv6 cellular providers still don't use it for end to end
connectivity, as inbound connections are silently dropped. In the US if you
want inbound connectivity to work via cellular, you must to buy the static IP
service from Verizon, and it has no IPv6 support, and no plans for it in the
future.

Oddly enough the MVNO services over T-Mobile seem to allow inbound IPv6, but
TMO proper doesn't. V6 that works everywhere would simplify a _huge_
connectivity problem for me.


--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Sent using a machine that autocorrects in interesting ways...

> On Oct 23, 2021, at 1:55 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
>
>> On Sat, Oct 23, 2021, 15:17 Fred Baker <fredbaker.ietf@gmail.com> wrote:
>> I think you will find that there are some places in which getting IPv6 network service has been difficult, and as a result even IPv6-
>
>
> Fred, do you mean places like, all of Verizon FiOS?

That would be an example. If I traceroute to an ipv6 address, the fact that I get a response is proof that the path works. F root has some servers for which, MOU be damned, there is no working IPv6 path. Mumble…

>> capable equipment is not reachable by IPv6. Those are, however, few and far between.
>>
>> Sent using a machine that autocorrects in interesting ways...
>>
>> > On Oct 23, 2021, at 6:04 AM, David Conrad <drc@virtualized.org> wrote:
>> >
>> > So, in theory, all the root servers should be available via IPv6 and, as far as I can tell, they are.

On Fri, Oct 22, 2021 at 05:13:09PM +0200, Job Snijders via NANOG wrote:
> Hi everyone, goedenmiddag Marco!
>
> On Fri, Oct 22, 2021 at 01:40:42PM +0200, Marco Davids via NANOG wrote:
> > We currently live in times where is actually fun to go IPv6-only. In my
> > case, as in: running a FreeBSD kernel compiled without the IPv4-stack.
>
> Indeed, this is fun experimentation. Shaking the (source code) trees
> through excercises like these is a valuable way to identify gaps.
>
> > It turns out that there underlying CDN's with domain names such as
> > ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
> > reside on authoritative name servers that *only* have an IPv4 address.
>
> As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
> Fastly is working hard with select customers and friends to support IPv6
> for everyone.
>
> ** SNIP **
>
> as BGP traffic engineering) might be reluctant to offer IPv6 services
> "as if they are the same as IPv4". More study is required.
>
> Tl;DR - work in progress! :-)

Some of the other CDNs do have IPv6 on the authorities and
should work without issues.

eg:

dig -6 +trace www.akamai.com.

- Jared

--
Jared Mauch | pgp key available via finger from jared@puck.nether.net
clue++; | http://puck.nether.net/~jared/ My statements are only mine.

On Mon, Oct 25, 2021 at 04:20:28PM -0400, Jared Mauch wrote:
> Some of the other CDNs do have IPv6 on the authorities and
> should work without issues.
>
> eg:
>
> dig -6 +trace www.akamai.com.

Yes of course :-)

dig -6 +trace www.fastly.com.

Kind regards,

Job

Bryan,

On Oct 23, 2021, at 5:56 PM, Bryan Fields <Bryan@bryanfields.net> wrote:
>> Excepting temporary failures, they are as far as I am aware. Why do you
>> think they aren’t?
>
> I can't reach C, 2001:500:2::c, from many places in v6 land. My home and

> secondary data center can't reach it, but my backup VM's at another data
> center can.

Ah. Cogent. I suspect IPv6 peering policies. Somebody should bake a cake.

>> However, the IANA team is not the enforcement arm of the Internet. If a
>> root server operator chooses to not abide by RFC 7720, there is nothing the
>> IANA team can do unilaterally other than make the root server operator
>> aware of the fact.
>
> Surely IANA has the power to compel a root server operator to abide by policy
> or they lose the right to be a root server?

To compel? No. Not in the slightest. That is not how the root server system works. This is a (very) common misconception.

There has been some effort to create a governance model for the root server system (see https://www.icann.org/en/system/files/files/rssac-037-15jun18-en.pdf) but I believe it has gotten bogged down in the question of “what do you do when a root server operator isn’t doing the job ‘right’ (whatever that means and after figuring out who decides) but doesn’t want to give up being a root server operator?”. It’s a hard question, but it isn’t the folks at IANA who answer it.

Regards,
-drc

On Tue, 26 Oct 2021, David Conrad wrote:

> Ah. Cogent. I suspect IPv6 peering policies. Somebody should bake a
> cake.

According to https://twitter.com/Benjojo12/status/1452673637606166536
Cogent<->Google IPv6 now works. A cake is in order, but perhaps a
celebratory one!?

--
Mikael Abrahamsson email: swmike@swm.pp.se

On 22/10/2021 17:08, tim@pelican.org wrote:
> I don't think it'll ever make money, but I think it will reduce
> costs. CGNAT boxes cost money, operating them costs money, dealing
> with the support fallout from them costs money. Especially in the
> residential space, where essentially if the customer calls you, ever,
> you just blew years' worth of margin.

There aren't enough folk thinking along these lines, so thank you for
writing it.

Every flow you can route exclusively with 6, is one flow you aren't
having to pay extra for so it can sit in a CGNAT state table.

... And that's before they call you, as Tim also rightly points out.

--
Tom

Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
traffic flow through CGNAT, so cheaper hardware and less IPv4 address
space. Isn't it?

22.10.21 20:19, Mark Tinka ????:
>
>
> On 10/22/21 18:08, tim@pelican.org wrote:
>
>> I don't think it'll ever make money, but I think it will reduce
>> costs.  CGNAT boxes cost money, operating them costs money, dealing
>> with the support fallout from them costs money.  Especially in the
>> residential space, where essentially if the customer calls you, ever,
>> you just blew years' worth of margin.
>
> The problem is accurately modelling cost reduction using native IPv6 in
> lieu of CG-NAT is hard when the folk that need convincing are the CFO's.
>
> They are more used to "spend 1 to get 2". Convincing them to "save 2 by
> spending 1" - not as easy as one may think.
>
> Mark.
>

On 11/3/21 22:13, Max Tulyev wrote:

> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
> traffic flow through CGNAT, so cheaper hardware and less IPv4 address
> space. Isn't it?

How to express that in numbers CFO can take to the bank?

Mark.

With a kicking ass pitch

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mark Tinka
Sent: November 26, 2021 5:52 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/3/21 22:13, Max Tulyev wrote:

> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
> traffic flow through CGNAT, so cheaper hardware and less IPv4 address
> space. Isn't it?

How to express that in numbers CFO can take to the bank?

Mark.

On 26/11/2021 13:52, Mark Tinka wrote:
> On 11/3/21 22:13, Max Tulyev wrote:
>> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
>> traffic flow through CGNAT, so cheaper hardware and less IPv4 address
>> space. Isn't it?
>
> How to express that in numbers CFO can take to the bank?

"want to buy 5 of those shiny new CGNAT boxes or only 2 ?"

Frank

Well … YMMV. We’ve been running v6 for years, and it didn’t really make a dent in spend or boxes or rate of v4 depletion. Big part of the problem in our neck of the woods is millions of v4-only terminals … as well as large customer/gov bids requiring tons of v4 address space.

> On Nov 26, 2021, at 07:04, Jean St-Laurent via NANOG <nanog@nanog.org> wrote:
>
> ?With a kicking ass pitch
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mark Tinka
> Sent: November 26, 2021 5:52 AM
> To: nanog@nanog.org
> Subject: Re: IPv6 and CDN's
>
>
>
>> On 11/3/21 22:13, Max Tulyev wrote:
>>
>> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
>> traffic flow through CGNAT, so cheaper hardware and less IPv4 address
>> space. Isn't it?
>
> How to express that in numbers CFO can take to the bank?
>
> Mark.
>

Care to explain because the alternative seems pretty self-evident.




-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: "Jose Luis Rodriguez" <jlrodriguez@gmail.com>
To: "Jean St-Laurent" <jean@ddostest.me>
Cc: nanog@nanog.org
Sent: Friday, November 26, 2021 8:16:53 AM
Subject: Re: IPv6 and CDN's

Well … YMMV. We’ve been running v6 for years, and it didn’t really make a dent in spend or boxes or rate of v4 depletion. Big part of the problem in our neck of the woods is millions of v4-only terminals … as well as large customer/gov bids requiring tons of v4 address space.

> On Nov 26, 2021, at 07:04, Jean St-Laurent via NANOG <nanog@nanog.org> wrote:
>
> With a kicking ass pitch
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mark Tinka
> Sent: November 26, 2021 5:52 AM
> To: nanog@nanog.org
> Subject: Re: IPv6 and CDN's
>
>
>
>> On 11/3/21 22:13, Max Tulyev wrote:
>>
>> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less
>> traffic flow through CGNAT, so cheaper hardware and less IPv4 address
>> space. Isn't it?
>
> How to express that in numbers CFO can take to the bank?
>
> Mark.
>

Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/



Build around that maybe?



Jean



From: Mike Hammett <nanog@ics-il.net>
Sent: November 26, 2021 11:56 AM
To: Jose Luis Rodriguez <jlrodriguez@gmail.com>
Cc: nanog@nanog.org; Jean St-Laurent <jean@ddostest.me>
Subject: Re: IPv6 and CDN's



Care to explain because the alternative seems pretty self-evident.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com



_____

With that specific line directly from Apple:



"And when IPv6 is in use, the median connection setup is 1.4 times faster than IPv4. This is primarily due to reduced NAT usage and improved routing."



There it is, Improved routing.



Jean



From: Jean St-Laurent <jean@ddostest.me>
Sent: November 26, 2021 4:44 PM
To: 'Mike Hammett' <nanog@ics-il.net>; 'Jose Luis Rodriguez' <jlrodriguez@gmail.com>
Cc: 'nanog@nanog.org' <nanog@nanog.org>
Subject: RE: IPv6 and CDN's



Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/



Build around that maybe?



Jean



From: Mike Hammett <nanog@ics-il.net <mailto:nanog@ics-il.net> >
Sent: November 26, 2021 11:56 AM
To: Jose Luis Rodriguez <jlrodriguez@gmail.com <mailto:jlrodriguez@gmail.com> >
Cc: nanog@nanog.org <mailto:nanog@nanog.org> ; Jean St-Laurent <jean@ddostest.me <mailto:jean@ddostest.me> >
Subject: Re: IPv6 and CDN's



Care to explain because the alternative seems pretty self-evident.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com



_____

On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
>
> Here are some maths and 1 argument kicking ass pitch for CFO’s that
> use iphones.
>
> *Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4*
>
> https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/
>
> Build around that maybe?
>
>
This really hits my bs meter big time. I can't see how nat'ing is going
to cause a 40% performance hit during connections. The article also
mentions http2 (and later v3) which definitely make big improvements so
I'm suspecting that the author is conflating them.

Mike

On Fri, Nov 26, 2021 at 6:07 PM Michael Thomas <mike@mtcc.com> wrote:

>
> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
>
> Here are some maths and 1 argument kicking ass pitch for CFO’s that use
> iphones.
>
> *Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4*
>
>
> https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/
>
>
>
> Build around that maybe?
>
> This really hits my bs meter big time. I can't see how nat'ing is going to
> cause a 40% performance hit during connections. The article also mentions
> http2 (and later v3) which definitely make big improvements so I'm
> suspecting that the author is conflating them.
>
> Mike
>

Ok, take the same ipv6 is faster claim from facebook

https://www.internetsociety.org/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/



>

On 11/26/21 3:11 PM, Ca By wrote:
>
>
> On Fri, Nov 26, 2021 at 6:07 PM Michael Thomas <mike@mtcc.com> wrote:
>
>
> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
>>
>> Here are some maths and 1 argument kicking ass pitch for CFO’s
>> that use iphones.
>>
>> *Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4*
>>
>> https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/
>>
>> Build around that maybe?
>>
>>
> This really hits my bs meter big time. I can't see how nat'ing is
> going to cause a 40% performance hit during connections. The
> article also mentions http2 (and later v3) which definitely make
> big improvements so I'm suspecting that the author is conflating them.
>
> Mike
>
>
> Ok, take the same ipv6 is faster claim from facebook
>
> https://www.internetsociety.org/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/
>
>
Still really thin with details of why. At least this says that they are
NAT'ing v4 at *their* edge. But 99% of the lag of filling your newsfeed
is their backend and transport, not connection times so who knows what
they are actually measuring. Most NAT'ing is done at the consumer end by
your home router in any case.

Mike

Might also be due to Happy Eyeballs 2 artificial IPv4 A resolution delay.

https://www.researchgate.net/publication/343467289_Reducing_User_Perceived_Latency_in_Smart_Phones_Exploiting_IP_Network_Diversity/fulltext/5f2b6bdaa6fdcccc43ac7b99/Reducing-User-Perceived-Latency-in-Smart-Phones-Exploiting-IP-Network-Diversity.pdf

I don’t know the current Apple IOS HE2 postpone delay.

JC Bisecco

De : NANOG <nanog-bounces+jc=jclb.net@nanog.org> De la part de Michael Thomas
Envoyé : samedi 27 novembre 2021 00:20
À : Ca By <cb.list6@gmail.com>
Cc : nanog@nanog.org
Objet : Re: IPv6 and CDN's



On 11/26/21 3:11 PM, Ca By wrote:


On Fri, Nov 26, 2021 at 6:07 PM Michael Thomas <mike@mtcc.com<mailto:mike@mtcc.com>> wrote:


On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.
Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4
https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/

Build around that maybe?


This really hits my bs meter big time. I can't see how nat'ing is going to cause a 40% performance hit during connections. The article also mentions http2 (and later v3) which definitely make big improvements so I'm suspecting that the author is conflating them.

Mike

Ok, take the same ipv6 is faster claim from facebook

https://www.internetsociety.org/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/



Still really thin with details of why. At least this says that they are NAT'ing v4 at *their* edge. But 99% of the lag of filling your newsfeed is their backend and transport, not connection times so who knows what they are actually measuring. Most NAT'ing is done at the consumer end by your home router in any case.

Mike

We now have apple and fb saying ipv6 is faster than ipv4.



If we can onboard Amazon, Netflix, Google and some others, then it is a done deal that ipv6 is indeed faster than ipv4.



Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.



Xmas is coming so the budget season. Who knows. You might get lucky this year.



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Michael Thomas
Sent: November 26, 2021 6:20 PM
To: Ca By <cb.list6@gmail.com>
Cc: nanog@nanog.org
Subject: Re: IPv6 and CDN's





On 11/26/21 3:11 PM, Ca By wrote:





On Fri, Nov 26, 2021 at 6:07 PM Michael Thomas <mike@mtcc.com <mailto:mike@mtcc.com> > wrote:



On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:

Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/



Build around that maybe?



This really hits my bs meter big time. I can't see how nat'ing is going to cause a 40% performance hit during connections. The article also mentions http2 (and later v3) which definitely make big improvements so I'm suspecting that the author is conflating them.

Mike



Ok, take the same ipv6 is faster claim from facebook



https://www.internetsociety.org/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/





Still really thin with details of why. At least this says that they are NAT'ing v4 at *their* edge. But 99% of the lag of filling your newsfeed is their backend and transport, not connection times so who knows what they are actually measuring. Most NAT'ing is done at the consumer end by your home router in any case.

Mike

On 11/26/21 4:15 PM, Jean St-Laurent wrote:
>
> We now have apple and fb saying ipv6 is faster than ipv4.
>
> If we can onboard Amazon, Netflix, Google and some others, then it is
> a done deal that ipv6 is indeed faster than ipv4.
>
> Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.
>
Netflix is already v6 ready. The biggest obstacle is probably aws
because that's where a lot of the long tail of the internet resides.
Lobbying them would get the most bang for the buck.

Mike

AWS has been gradually improving support and adding features. They just
announced this service, which might help with adoption:

https://aws.amazon.com/about-aws/whats-new/2021/11/aws-nat64-dns64-communication-ipv6-ipv4-services/


On Fri., Nov. 26, 2021, 19:28 Michael Thomas, <mike@mtcc.com> wrote:

>
> On 11/26/21 4:15 PM, Jean St-Laurent wrote:
>
> We now have apple and fb saying ipv6 is faster than ipv4.
>
>
>
> If we can onboard Amazon, Netflix, Google and some others, then it is a
> done deal that ipv6 is indeed faster than ipv4.
>
>
>
> Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.
>
> Netflix is already v6 ready. The biggest obstacle is probably aws because
> that's where a lot of the long tail of the internet resides. Lobbying them
> would get the most bang for the buck.
>
> Mike
>
>

On 11/26/21 4:30 PM, Oliver O'Boyle wrote:
> AWS has been gradually improving support and adding features. They
> just announced this service, which might help with adoption:
>
> https://aws.amazon.com/about-aws/whats-new/2021/11/aws-nat64-dns64-communication-ipv6-ipv4-services/

That's a start, I guess. Before all they had was some weird VPN
something or other. Let me guess though: they are monetizing their
market failure.

Mike


>
>
> On Fri., Nov. 26, 2021, 19:28 Michael Thomas, <mike@mtcc.com> wrote:
>
>
> On 11/26/21 4:15 PM, Jean St-Laurent wrote:
>>
>> We now have apple and fb saying ipv6 is faster than ipv4.
>>
>> If we can onboard Amazon, Netflix, Google and some others, then
>> it is a done deal that ipv6 is indeed faster than ipv4.
>>
>> Hence, an easy argument to tell your CFO that you need IPv6 for
>> your CDN.
>>
> Netflix is already v6 ready. The biggest obstacle is probably aws
> because that's where a lot of the long tail of the internet
> resides. Lobbying them would get the most bang for the buck.
>
> Mike
>
>

But CFOs like monetization. Was that thread about IPv6 or CFO?



From: Michael Thomas <mike@mtcc.com>
Sent: November 26, 2021 7:37 PM
To: Oliver O'Boyle <oliver.oboyle@gmail.com>
Cc: Jean St-Laurent <jean@ddostest.me>; Ca By <cb.list6@gmail.com>; North American Network Operators' Group <nanog@nanog.org>
Subject: Re: IPv6 and CDN's



That's a start, I guess. Before all they had was some weird VPN something or other. Let me guess though: they are monetizing their market failure.

On 11/26/21 4:39 PM, Jean St-Laurent wrote:
>
> But CFOs like monetization. Was that thread about IPv6 or CFO?
>

Amazon's in this case. They are monetizing their lack of v6 support
requiring you go through all kinds of expensive hoops instead of doing
the obvious and routing v6 packets.

Mike

> *From:*Michael Thomas <mike@mtcc.com>
> *Sent:* November 26, 2021 7:37 PM
> *To:* Oliver O'Boyle <oliver.oboyle@gmail.com>
> *Cc:* Jean St-Laurent <jean@ddostest.me>; Ca By <cb.list6@gmail.com>;
> North American Network Operators' Group <nanog@nanog.org>
> *Subject:* Re: IPv6 and CDN's
>
> That's a start, I guess. Before all they had was some weird VPN
> something or other. Let me guess though: they are monetizing their
> market failure.
>

On Fri., Nov. 26, 2021, 19:41 Michael Thomas, <mike@mtcc.com> wrote:

>
> On 11/26/21 4:39 PM, Jean St-Laurent wrote:
>
> But CFOs like monetization. Was that thread about IPv6 or CFO?
>
>
> Amazon's in this case. They are monetizing their lack of v6 support
> requiring you go through all kinds of expensive hoops instead of doing the
> obvious and routing v6 packets.
>

They're getting better at it, at least. They also recently added v6 support
in their NLBs and you can get a /56 for every VPC for direct access. I
don't think they offer BYO v6 yet, as they do for v4, but it will come.


Mike
>
>
>
> *From:* Michael Thomas <mike@mtcc.com> <mike@mtcc.com>
> *Sent:* November 26, 2021 7:37 PM
> *To:* Oliver O'Boyle <oliver.oboyle@gmail.com> <oliver.oboyle@gmail.com>
> *Cc:* Jean St-Laurent <jean@ddostest.me> <jean@ddostest.me>; Ca By
> <cb.list6@gmail.com> <cb.list6@gmail.com>; North American Network
> Operators' Group <nanog@nanog.org> <nanog@nanog.org>
> *Subject:* Re: IPv6 and CDN's
>
>
>
> That's a start, I guess. Before all they had was some weird VPN something
> or other. Let me guess though: they are monetizing their market failure.
>
>

> On Oct 26, 2021, at 9:11 AM, David Conrad <drc@virtualized.org> wrote:
>
> There has been some effort to create a governance model for the root server system (see https://www.icann.org/en/system/files/files/rssac-037-15jun18-en.pdf) but I believe it has gotten bogged down in the question of “what do you do when a root server operator isn’t doing the job ‘right’ (whatever that means and after figuring out who decides) but doesn’t want to give up being a root server operator?”.

Unless you actually read the document. The process is that the fact is recognized and documented, the Designation and Removal function advises the ICANN board, they adopt a resolution, and instruct the IANA to remove the addresses from the relevant files, and from that point on nobody NEW tries to usevtheRSO. If someone does ask the company a question, they might or might not respond, and it might even have correct data, but then again might not. We can’t control who sends us requests, and we don’t have a black list.

On 11/26/21 15:00, Jean St-Laurent wrote:

> With a kicking ass pitch

Can I take your CFO :-)...

Mark.

On 11/26/21 15:47, Frank Habicht wrote:

>
> "want to buy 5 of those shiny new CGNAT boxes or only 2 ?"

To which she will respond, "2 or 5, what do I make :-)?"

Mark.

On 11/26/21 16:16, Jose Luis Rodriguez wrote:

> Well … YMMV. We’ve been running v6 for years, and it didn’t really make a dent in spend or boxes or rate of v4 depletion. Big part of the problem in our neck of the woods is millions of v4-only terminals … as well as large customer/gov bids requiring tons of v4 address space.

I can very easily see why "IPv6 saves you on CG-NAT capex might not be
entirely true" in cases such as these.

On paper, it all adds up.

Mark.

On 11/27/21 02:41, Michael Thomas wrote:

> Amazon's in this case. They are monetizing their lack of v6 support
> requiring you go through all kinds of expensive hoops instead of doing
> the obvious and routing v6 packets.
>

Individual CDN's and content providers have better control over how they
deploy IPv6, vs. ISP's who have far less capital, warm bodies and
innovation DNA.

I'm arguing for the latter.

Mark.

On 11/26/21 23:47, Jean St-Laurent via NANOG wrote:

> With that specific line directly from Apple:
>
> "And when IPv6 is in use, the median connection setup is 1.4 times
> faster than IPv4. This is primarily due to reduced NAT usage and
> improved routing."
>
> There it is, Improved routing.
>

Perhaps you mean "improved forwarding".

In an environment that is heavily peered, what is the visual difference
in experience for a customer connecting to a site at 1ms vs. 1.4ms?

Across the sea, assume 140ms between Cape Town - London (Omicron,
anyone?), what is the visual difference between 140ms vs. 196ms? Okay,
bad example, but I can probably get an MS-MPC from Juniper that can claw
back 1.3X of that 1.4X advantage :-). Besides, locally-peered traffic is
likely to exceed long-haul traffic, in many markets.

Mark.

On 11/27/21 02:15, Jean St-Laurent via NANOG wrote:

> We now have apple and fb saying ipv6 is faster than ipv4.
>
> If we can onboard Amazon, Netflix, Google and some others, then it is
> a done deal that ipv6 is indeed faster than ipv4.
>
> Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.
>

Could work for CDN's... but what about the CFO of an ISP or MNO?

Mark.

On 11/27/21 02:39, Jean St-Laurent via NANOG wrote:

> But CFOs like monetization. Was that thread about IPv6 or CFO?
>

In 2021, what's the difference?

Mark.

Mark Tinka wrote:

> I can very easily see why "IPv6 saves you on CG-NAT capex might not
> be entirely true" in cases such as these.

Because lengthy IPv6 addresses mean a lot more opex than IPv4.

> On paper, it all adds up.

With IPv6, you need 4 times more paper.

Masataka Ohta

On Fri, Nov 26, 2021 at 6:51 PM Oliver O'Boyle <oliver.oboyle@gmail.com>
wrote:

> They're getting better at it, at least. They also recently added v6
> support in their NLBs and you can get a /56 for every VPC for direct
> access. I don't think they offer BYO v6 yet, as they do for v4, but it will
> come.
>

Since we are deploying BYO IPv6 in AWS, I can assure you they do offer it
now. That was a blocker for us.

Scott

On 11/27/21 17:07, Masataka Ohta wrote:

> Because lengthy IPv6 addresses mean a lot more opex than IPv4.

I disagree - it can be more opex if you want to run both together, but
less so if you choose one; largely IPv6, but also largely IPv4 if you
don't intend to be in the game for the rest of your life.

My point was that there might not obviously be a linear relationship
between less CG-NAT and more native IPv6, that makes a material
difference to the CFO's Excel spreadsheet, off the bat.

Mark.

On Sat., Nov. 27, 2021, 10:46 Scott Morizot, <tmorizot@gmail.com> wrote:

> On Fri, Nov 26, 2021 at 6:51 PM Oliver O'Boyle <oliver.oboyle@gmail.com>
> wrote:
>
>> They're getting better at it, at least. They also recently added v6
>> support in their NLBs and you can get a /56 for every VPC for direct
>> access. I don't think they offer BYO v6 yet, as they do for v4, but it will
>> come.
>>
>
> Since we are deploying BYO IPv6 in AWS, I can assure you they do offer it
> now. That was a blocker for us.
>

Wonderful! When did they start offering that?



> Scott
>

On Sat, Nov 27, 2021 at 5:05 PM Oliver O'Boyle <oliver.oboyle@gmail.com> wrote:

> On Sat., Nov. 27, 2021, 10:46 Scott Morizot, <tmorizot@gmail.com> wrote:
>> Since we are deploying BYO IPv6 in AWS, I can assure you they do offer it now. That was a blocker for us.

> Wonderful! When did they start offering that?

I believe it was announced back in the first half
of 2020.

As I recall it was limited to certain regions at the
time of the original announcement (and being
AWS it probably still has some region and/or
resource specific availability limitations).

On 11/27/21 7:46 AM, Scott Morizot wrote:
> On Fri, Nov 26, 2021 at 6:51 PM Oliver O'Boyle
> <oliver.oboyle@gmail.com> wrote:
>
> They're getting better at it, at least. They also recently added
> v6 support in their NLBs and you can get a /56 for every VPC for
> direct access. I don't think they offer BYO v6 yet, as they do for
> v4, but it will come.
>
>
> Since we are deploying BYO IPv6 in AWS, I can assure you they do offer
> it now. That was a blocker for us.
>
I thought it had to be some virtual private cloud setup? To get the long
tail it needs to be a lot more simple. Like "here is the AAAA record"
after autoconf.

Mike

> On Nov 27, 2021, at 06:04 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/27/21 02:41, Michael Thomas wrote:
>
>> Amazon's in this case. They are monetizing their lack of v6 support requiring you go through all kinds of expensive hoops instead of doing the obvious and routing v6 packets.
>>
>
> Individual CDN's and content providers have better control over how they deploy IPv6, vs. ISP's who have far less capital, warm bodies and innovation DNA.
>
> I'm arguing for the latter.
>
> Mark.

I honestly think that in Amazon’s case, it’s because they’ve cocked up v4 so badly in their attempts to squeeze every drop of life out of every v4 address they have that they have built a nightmare network that makes it utterly difficult to do the obvious and simply route v6 packets because they can’t even do that with v4 if they wanted to.

Admittedly, this is based only on comments and descriptions I’ve heard from others about the Amazon network, including customers, Amazon SEs, Amazon staff, etc., but it does seem to fit the available observations rather well.

Owen

> On Nov 27, 2021, at 06:05 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/27/21 02:15, Jean St-Laurent via NANOG wrote:
>
>> We now have apple and fb saying ipv6 is faster than ipv4.
>>
>> If we can onboard Amazon, Netflix, Google and some others, then it is a done deal that ipv6 is indeed faster than ipv4.
>>
>> Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.
>
> Could work for CDN's... but what about the CFO of an ISP or MNO?
>
> Mark.

Shouldn’t the argument that we can reduce our CGN spend by 50-80% work there?
Especially when you couple it with the argument that IPv6 deployment will likely reduce CGN related support calls which are one of the
biggest expenses for most ISP/MNOs?

Owen

> On Nov 27, 2021, at 06:05 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/27/21 02:39, Jean St-Laurent via NANOG wrote:
>
>> But CFOs like monetization. Was that thread about IPv6 or CFO?
>
> In 2021, what's the difference?
>
> Mark.

Even in 2021, one improves network capabilities while the other counts beans.

Which is which is left as an exercise to the reader.

Owen

On Sat., Nov. 27, 2021, 13:34 Michael Thomas, <mike@mtcc.com> wrote:

>
> On 11/27/21 7:46 AM, Scott Morizot wrote:
>
> On Fri, Nov 26, 2021 at 6:51 PM Oliver O'Boyle <oliver.oboyle@gmail.com>
> wrote:
>
>> They're getting better at it, at least. They also recently added v6
>> support in their NLBs and you can get a /56 for every VPC for direct
>> access. I don't think they offer BYO v6 yet, as they do for v4, but it will
>> come.
>>
>
> Since we are deploying BYO IPv6 in AWS, I can assure you they do offer it
> now. That was a blocker for us.
>
> I thought it had to be some virtual private cloud setup? To get the long
> tail it needs to be a lot more simple. Like "here is the AAAA record" after
> autoconf.
>
Well, VPC is the only deployment model now. EC2 Classic is long gone
(though some long-time legacy customers may still have it as an option). If
you create an account, you get a default VPC. You can use it or create
another with a few clicks. Prefixes get assigned upon creation but you can
add more afterwards. It's actually pretty straightfoward. Setting up basic
DNS in Route53 is also pretty straightforward. There are no real barriers
up to this point.

> Mike
>

On Sat., Nov. 27, 2021, 12:59 Gary Buhrmaster, <gary.buhrmaster@gmail.com>
wrote:

> On Sat, Nov 27, 2021 at 5:05 PM Oliver O'Boyle <oliver.oboyle@gmail.com>
> wrote:
>
> > On Sat., Nov. 27, 2021, 10:46 Scott Morizot, <tmorizot@gmail.com> wrote:
> >> Since we are deploying BYO IPv6 in AWS, I can assure you they do offer
> it now. That was a blocker for us.
>
> > Wonderful! When did they start offering that?
>
> I believe it was announced back in the first half
> of 2020.
>
> As I recall it was limited to certain regions at the
> time of the original announcement (and being
> AWS it probably still has some region and/or
> resource specific availability limitations).
>

Likely. But if it was announced in 2020 then the rollout is either
complete, or mostly complete, by now.

>

On Fri, Nov 26, 2021 at 3:07 PM Michael Thomas <mike@mtcc.com> wrote:
>> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
>> Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.
>> Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

> This really hits my bs meter big time.

If I had to guess, this is an example of correlation is not causation.
Folks with IPv6 tend to be on savvier service providers who have
better performance for both IPv4 and IPv6. To find out for sure,
you'd have to do an experiment where same-user-same-server connections
are split between IPv4 and IPv6 and then measure the performance
difference. I don't know if anyone has done that but these particular
articles look like someone is just looking at the high-level metrics.
Those won't hold any statistical validity because they're not actually
random samples.

Regards,
Bill Herrin


--
William Herrin
bill@herrin.us
https://bill.herrin.us/

On 11/27/21 12:16 PM, William Herrin wrote:
> On Fri, Nov 26, 2021 at 3:07 PM Michael Thomas <mike@mtcc.com> wrote:
>>> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
>>> Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.
>>> Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4
>> This really hits my bs meter big time.
> If I had to guess, this is an example of correlation is not causation.
> Folks with IPv6 tend to be on savvier service providers who have
> better performance for both IPv4 and IPv6. To find out for sure,
> you'd have to do an experiment where same-user-same-server connections
> are split between IPv4 and IPv6 and then measure the performance
> difference. I don't know if anyone has done that but these particular
> articles look like someone is just looking at the high-level metrics.
> Those won't hold any statistical validity because they're not actually
> random samples.

I agree. It's pretty suspect that they didn't give the reason it was
happening. I mean, why the incuriosity?

Mike

On 26/11/2021 22:47, Jean St-Laurent via NANOG wrote:
> "And when IPv6 is in use, the median connection setup is 1.4 times
> faster than IPv4. This is primarily due to reduced NAT usage and
> improved routing."

Oh I believe IPv6 is faster but because of completely different reasons.
Modern faster connections more likely have IPv6 while old low-bandwidth
circuits may provide v4 only.

Some users may also use VPN which is almost always v4 only. Their VPN
may do funny routing, hair-pinning and similar behavior thus impacting
their performance.

--
Grzegorz Janoszka

Actually, I think it’s in the fine print here…

“Connection setup is 1.4 times faster”. I can believe that NAT adds almost 40% overhead to the connection setup (3-way handshake) and some
of the differences in packet handling in the fast path between v4 and v6 could contribute the small remaining difference.

I doubt it is due to different connections, since we’re talking about measurements against dual-stack sites reached from dual-stack end-users,
very likely traversing similar paths.

Owen


> On Nov 27, 2021, at 14:02 , Grzegorz Janoszka <grzegorz@janoszka.pl> wrote:
>
> On 26/11/2021 22:47, Jean St-Laurent via NANOG wrote:
>> "And when IPv6 is in use, the median connection setup is 1.4 times faster than IPv4. This is primarily due to reduced NAT usage and improved routing."
>
> Oh I believe IPv6 is faster but because of completely different reasons.
> Modern faster connections more likely have IPv6 while old low-bandwidth circuits may provide v4 only.
>
> Some users may also use VPN which is almost always v4 only. Their VPN may do funny routing, hair-pinning and similar behavior thus impacting their performance.
>
> --
> Grzegorz Janoszka

On 11/27/21 2:22 PM, Owen DeLong via NANOG wrote:
> Actually, I think it’s in the fine print here…
>
> “Connection setup is 1.4 times faster”. I can believe that NAT adds almost 40% overhead to the connection setup (3-way handshake) and some
> of the differences in packet handling in the fast path between v4 and v6 could contribute the small remaining difference.
>
> I doubt it is due to different connections, since we’re talking about measurements against dual-stack sites reached from dual-stack end-users,
> very likely traversing similar paths.
>
40% in isolation is pretty meaningless. If it's 40% of .1% overall it's
called a rounding error.

Mike

Well, 1.4x faster is a bit of an odd metric. I presume that means that connection set up times measured were on average
1/1.4 times as long for IPv6 as they were for IPv4, but there are other possible interpretations.

So really, that’s a convoluted way of saying it takes 29% less time to set up an IPv6 connection than an IPv4 connection on average.

I can believe that is likely in a scenario where one is dealing with IPv4 NAT overhead.

It’s still probably rounding error for any real world purpose, since we’re probably talking about something that normally takes
between 50 and 150 ms, so if it takes 1.4 times as long in IPv4, that’d be 70-210 ms, so still mostly under 1/5th of a second,
which is not below human perception, but likely below human notice.

Owen


> On Nov 27, 2021, at 14:30 , Michael Thomas <mike@mtcc.com> wrote:
>
>
> On 11/27/21 2:22 PM, Owen DeLong via NANOG wrote:
>> Actually, I think it’s in the fine print here…
>>
>> “Connection setup is 1.4 times faster”. I can believe that NAT adds almost 40% overhead to the connection setup (3-way handshake) and some
>> of the differences in packet handling in the fast path between v4 and v6 could contribute the small remaining difference.
>>
>> I doubt it is due to different connections, since we’re talking about measurements against dual-stack sites reached from dual-stack end-users,
>> very likely traversing similar paths.
>>
> 40% in isolation is pretty meaningless. If it's 40% of .1% overall it's called a rounding error.
>
> Mike

On Sat, Nov 27, 2021, 17:36 Owen DeLong via NANOG <nanog@nanog.org> wrote:

> Well, 1.4x faster is a bit of an odd metric. I presume that means that
> connection set up times measured were on average
> 1/1.4 times as long for IPv6 as they were for IPv4, but there are other
> possible interpretations.
>
> So really, that’s a convoluted way of saying it takes 29% less time to set
> up an IPv6 connection than an IPv4 connection on average.
>
> I can believe that is likely in a scenario where one is dealing with IPv4
> NAT overhead.
>


Why isn't this just inconsistent paths between V6 and V4/nat? (Divergent
topologies)

> On Nov 27, 2021, at 17:21 , Christopher Morrow <morrowc.lists@gmail.com> wrote:
>
>
>
> On Sat, Nov 27, 2021, 17:36 Owen DeLong via NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> wrote:
> Well, 1.4x faster is a bit of an odd metric. I presume that means that connection set up times measured were on average
> 1/1.4 times as long for IPv6 as they were for IPv4, but there are other possible interpretations.
>
> So really, that’s a convoluted way of saying it takes 29% less time to set up an IPv6 connection than an IPv4 connection on average.
>
> I can believe that is likely in a scenario where one is dealing with IPv4 NAT overhead.
>
>
> Why isn't this just inconsistent paths between V6 and V4/nat? (Divergent topologies)

At least in most of my real world experience, they don’t tend to diverge all that much.

Further, post-initiation performance seems to be largely on par v4<->v6 and without NAT, I see faster
v4 connection startup times. V6 appears to still have a slight advantage, but it’s more like 5-10% than
30%.

Owen

Mark Tinka wrote:

> On 11/27/21 17:07, Masataka Ohta wrote:
>
>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>
> I disagree

Try to type in raw IPv6 addresses.

Masataka Ohta

On Nov 27, 2021, at 7:39 PM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> ?Mark Tinka wrote:
>
>>> On 11/27/21 17:07, Masataka Ohta wrote:
>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>> I disagree
>
> Try to type in raw IPv6 addresses.

People are likely to use a technology originally developed because IPv4 had the same perception problem: DNS.

Fred Baker wrote:

>>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>>> I disagree
>>
>> Try to type in raw IPv6 addresses.
>
> People are likely to use a technology originally developed because > IPv4 had the same perception problem: DNS.

Here in nanog, we are talking about network operations, considerable
part of which can not rely on DNS.

Masataka Ohta

On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

On 11/28/21 06:43, Masataka Ohta wrote:

>
> Here in nanog, we are talking about network operations, considerable
> part of which can not rely on DNS.

And yet Facebook were unable to access their kit to fix their recent
outage because of it (or, lack of it).

There was a time when knowing the IP(v4) address of every interface of
every router in your network was cool. I have never had to care about
that in close to 15 years. Right up there with losing interest in making
software modems work in Linux, when it was a thing :-).

If you want to remain stuck in the past, we don't have to join you.

Mark.

Ipv6 can be shorter than ipv4.

Here is the proof:

ping6 ::1

is shorter than

ping 127.1

ipv6 addresses can be very small when done properly.

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mark Tinka
Sent: November 28, 2021 5:39 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

On 11/28/21 14:09, Jean St-Laurent wrote:

> Ipv6 can be shorter than ipv4.
>
> Here is the proof:
>
> ping6 ::1
>
> is shorter than
>
> ping 127.1
>
> ipv6 addresses can be very small when done properly.

The good news is the point of an IP address is not for its own sake.

Mark.

Mark Tinka wrote:

>> Here in nanog, we are talking about network operations, considerable
>> part of which can not rely on DNS.
>
> And yet Facebook were unable to access their kit to fix their recent
> outage because of it (or, lack of it).

Exactly.

That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.

> There was a time when knowing the IP(v4) address of every interface of
> every router in your network was cool.

I surely acknowledge your point that it is impossible to do so with
MAC address based IPv6 addresses, which is why IPv6 opex is so high.

But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.

> I have never had to care about
> that in close to 15 years. Right up there with losing interest in making
> software modems work in Linux, when it was a thing :-).

So, you are saying you haven't faced real operational problems
to loss DNS information for these 15 years.

Congratulations for your luck!

> If you want to remain stuck in the past, we don't have to join you.

Surely, the recent disaster of facebook happened in the recent
past.

So what?

Masataka Ohta

On 11/28/21 14:58, Masataka Ohta wrote:

> Exactly.
>
> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.

Yes, but that does not mean that DNS is not valuable, or cannot be
hardened.

Everything can break, even an IPv4 interface on a router port. Good
practice in network operations is what keeps these kinds of problems at
bay. I mean, why else do we have lists like these?

I am certain Facebook have hardened their DNS infrastructure, and that
particular failure scenario should not recur, given all the clever
people there, and around them.


>
>> There was a time when knowing the IP(v4) address of every interface
>> of every router in your network was cool.
>
> I surely acknowledge your point that it is impossible to do so with
> MAC address based IPv6 addresses, which is why IPv6 opex is so high.
>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.

That's just satisfying one's mental (or emotional) nits.

Routers (and customers) don't care about how anally we assign address
space. As long as it is compliant, does not conflict, and is correctly
routed.

That we cannot transpose our IPv4 mental/emotional habits on to IPv6
does not make IPv6 more complicated. It just makes us more stuck in our
ways.

After all, DHCPv6 still does not offer a default gateway.


> So, you are saying you haven't faced real operational problems
> to loss DNS information for these 15 years.
>
> Congratulations for your luck!

I am sure I have had a DNS issue of some sort or other in the past 15
years. The fact that I can't remember what it was is telling.


> Surely, the recent disaster of facebook happened in the recent
> past.
>
> So what?

And they have learned from it, and I dare say, fixed it.

Facebook will neither be disposing of DNS any time soon, nor will they
be dropping IPv6.

Mark.

søn. 28. nov. 2021 13.59 skrev Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp>:

>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.
>

99% if not 100% of our subnets have either only routers or only hosts + a
gateway. So that would be a strange rule to follow. Also very expensive if
we are talking public addressing.

I find that 10.x.y.z is not much if you want to have a system in your
subnet numbering. With ipv6 there is much more space to enable systematic
numbering schemes.

>

Mark Tinka wrote:

>> That facebook poorly managed their DNS to cause the recent disaster
>> is an important evidence to support my point that DNS, so often, may
>> not be helpful for network operations against disastrous failures,
>> including, but not limited to, DNS failures.
>
> Yes, but that does not mean that DNS is not valuable, or cannot be
> hardened.

As a person who proposed anycast DNS servers, against which facebook
operated their DNS, I'm so sure you are right.

> I am certain Facebook have hardened their DNS infrastructure, and that
> particular failure scenario should not recur, given all the clever
> people there, and around them.

All I can see is that there were a lot of stupid people in facebook.

I really hope less of them still remain there.

Masataka Ohta

On 11/28/21 15:33, Masataka Ohta wrote:

> As a person who proposed anycast DNS servers, against which facebook
> operated their DNS, I'm so sure you are right.

Facebook's mistake on this is an easily fixable one.

We've all been there. Nothing groundbreaking.


> All I can see is that there were a lot of stupid people in facebook.
>
> I really hope less of them still remain there.

Can't help you there...

Mark.

Baldur Norddahl wrote:

>> But, with manually configured IP addresses, it is trivially easy
>> to have a rule to assign lower part of IP addresses within a subnet
>> for hosts and upper part for routers, which is enough to troubleshoot
>> most network failures.

> 99% if not 100% of our subnets have either only routers or only hosts + a
> gateway.

It merely means you should not use MAC address based IP addresses for,
at least, routers, which is partly why opex of IPv4 is low.

> So that would be a strange rule to follow. Also very expensive if we
> are talking public addressing.
>
> I find that 10.x.y.z is not much if you want to have a system in
> your subnet numbering.

In most cases, it is enough that addresses are unique within
certain domain, which is why many ISPs are assigning addresses,
not guaranteed to be globally unique, to there internal routers.

> With ipv6 there is much more space to enable systematic numbering
> schemes.

More space, only to encourage stupid idea of MAC address based
addresses of IPv6/ND, is not required for systematic numbering.

Masataka Ohta

On 11/28/21 15:59, Masataka Ohta wrote:

> It merely means you should not use MAC address based IP addresses for,
> at least, routers, which is partly why opex of IPv4 is low.

I often wonder what Internet you use :-)...


> More space, only to encourage stupid idea of MAC address based
> addresses of IPv6/ND, is not required for systematic numbering.

I'm sure a router won't stop working because we did not assign it an IP
address "systematically". Nor will the spinning of the earth.

Mark.

Mark Tinka wrote:

>> As a person who proposed anycast DNS servers, against which facebook
>> operated their DNS, I'm so sure you are right.
>
> Facebook's mistake on this is an easily fixable one.

Certainly, but, merely because it is an easily avoided one.

> We've all been there.

People who really understand DNS, including but not limited to
anycast one, have never been there.

Masataka Ohta

I like to put some servers behind that scheme.



2601::443:xxxx for https servers

2601::25:xxxx for MTA servers.

2601::993:xxxx for IMAP



It gives a quick note of what is that ip even though it’s ipv6 and usually non-human readable.



Not sure what kind of scheme is use by medium/big ISP.



Do you go by zip code of the area covered or some kind of logical to help people know what is behind that ipv6 network?



Jean



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Baldur Norddahl
Sent: November 28, 2021 8:22 AM
To: NANOG <nanog@nanog.org>
Subject: Re: IPv6 and CDN's





søn. 28. nov. 2021 13.59 skrev Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp <mailto:mohta@necom830.hpcl.titech.ac.jp> >:


But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.



99% if not 100% of our subnets have either only routers or only hosts + a gateway. So that would be a strange rule to follow. Also very expensive if we are talking public addressing.



I find that 10.x.y.z is not much if you want to have a system in your subnet numbering. With ipv6 there is much more space to enable systematic numbering schemes.

On 11/28/21 16:13, Masataka Ohta wrote:

>
> Certainly, but, merely because it is an easily avoided one.

None of the us came out the womb knowing anything. We learned as we went
along. And we keep learning, right until our death.

To expect experience before it is experienced has always been unreasonable.


> People who really understand DNS, including but not limited to
> anycast one, have never been there.

Anycast was the result of things that broke. And even with Anycast, lots
of things break, still.

Continuity of service != never having been there.

Mark.

It certainly sounds like you’ve never operated a network at scale if you think knowing the IP address of something reduces Operational expense. The only way to truly reduce Opex at scale is automation.

Shane

> On Nov 28, 2021, at 9:13 AM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> ?Mark Tinka wrote:
>
>>> As a person who proposed anycast DNS servers, against which facebook
>>> operated their DNS, I'm so sure you are right.
>> Facebook's mistake on this is an easily fixable one.
>
> Certainly, but, merely because it is an easily avoided one.
>
>> We've all been there.
>
> People who really understand DNS, including but not limited to
> anycast one, have never been there.
>
> Masataka Ohta

On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:

> I like to put some servers behind that scheme.
>
> 2601::443:xxxx for https servers
>
> 2601::25:xxxx for MTA servers.
>
> 2601::993:xxxx for IMAP
>
> It gives a quick note of what is that ip even though it’s ipv6 and
> usually non-human readable.
>
> Not sure what kind of scheme is use by medium/big ISP.
>
> Do you go by zip code of the area covered or some kind of logical to
> help people know what is behind that ipv6 network?
>

We really aren't clever with IPv6 address design and assignment. The
most we do is assign:

    - 1x /48 to Loopbacks for all routers.
    - /48's per PoP for infrastructure links.
    - /48's per city for /56 assignments to customers.
    - /48's per city for assignments to customers.

We don't try to co-ordinate them in a "meaningful" way that would offer
visual identification. We feel that is just too much work, and that
getting IPv4/IPv6 parity for day-to-day operations is a challenge in and
of itself, without trying to be fancy.

Mark.

> On Nov 27, 2021, at 19:37 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Mark Tinka wrote:
>
>> On 11/27/21 17:07, Masataka Ohta wrote:
>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>> I disagree
>
> Try to type in raw IPv6 addresses.

Rarely necessary in the modern age, but really not significantly more difficult than IPv4 once
you become accustomed to it.

Owen

> On Nov 28, 2021, at 02:42 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/28/21 06:43, Masataka Ohta wrote:
>
>>
>> Here in nanog, we are talking about network operations, considerable
>> part of which can not rely on DNS.
>
> And yet Facebook were unable to access their kit to fix their recent outage because of it (or, lack of it).

I’d argue that failing to put the correct documentation in place for coping with a DNS outage was the bigger
issue than the DNS failure in the Facebook outage… So would a number of the engineers I know at Facebook.

> There was a time when knowing the IP(v4) address of every interface of every router in your network was cool. I have never had to care about that in close to 15 years. Right up there with losing interest in making software modems work in Linux, when it was a thing :-).

There was a time when every router in a moderately large network was less than 50. Those days are gone.

Today, it’s impossible to build large scale networks without depending on certain tools. That means that the
failure of those tools can be catastrophic if one is not properly prepared. This is simply the modern reality.
Proper preparation is harder than it used to be, but for any such network, there should be online and off-line
copies of sufficient documentation (which is adequately maintained) to cope restore service of any such
underlying facility quickly in the event of a failure.

Owen

> On Nov 28, 2021, at 04:58 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Mark Tinka wrote:
>
>>> Here in nanog, we are talking about network operations, considerable
>>> part of which can not rely on DNS.
>> And yet Facebook were unable to access their kit to fix their recent outage because of it (or, lack of it).
>
> Exactly.
>
> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.

I’d argue that the true failure was failure to document the system adequately to provide for prompt
resolution of the DNS problems in the absence of DNS and failure to properly distribute that knowledge
to those that would need it in such a circumstance.


>> There was a time when knowing the IP(v4) address of every interface of every router in your network was cool.
>
> I surely acknowledge your point that it is impossible to do so with
> MAC address based IPv6 addresses, which is why IPv6 opex is so high.

Hardly anyone uses MAC address based IPv6 addresses for anything, so your claim here is specious.

Statically or manually configured IPv6 addresses are no more difficult than the equivalent in IPv4, just
slightly longer.

Consider, for example, 192.159.10.7 vs. 2620:0:930::400:7

The 400 could have been omitted leaving 2620:0:930::7, but I chose to have additional flexibility that
is available from IPv6 in classifying addresses for particular purposes and decided to use the next
order quartet for that purpose.

> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.

It’s equally easy to do that in IPv6 as well (while I prefer to use the reverse strategy, using low order
addresses for routers and higher numbers for non-forwarding hosts).

>> I have never had to care about that in close to 15 years. Right up there with losing interest in making software modems work in Linux, when it was a thing :-).
>
> So, you are saying you haven't faced real operational problems
> to loss DNS information for these 15 years.
>
> Congratulations for your luck!

In reality, DNS properly implemented is extraordinarily reliable these days. It’s not completely failure-proof,
as Facebook recently demonstrated so thoroughly, but consider how extraordinary that failure was in order
to garner so much attention. Gone are the days when DNS failures were a part of daily operational life
that we simply accepted.

>> If you want to remain stuck in the past, we don't have to join you.
>
> Surely, the recent disaster of facebook happened in the recent
> past.

Yes, but if you really look at it, that was a failure of preparedness much more than a failure of DNS.

> So what?

So, the world has moved on and modern networks are built using tools you appear to prefer not to
depend on… I remember when an RS-232 port was the gold standard of being able to recover your
router. Today, we accept at least USB and in most cases even Ethernet as a viable alternative.

Things get better, but that comes with higher-level dependencies on underlying infrastructure. Fortunately,
the underlying infrastructure gets better as well, making that possible and even reasonable. The fact
that you prefer not to recognize this is a sign that you are failing to adapt to the present in favor of
living in the past as Mark stated.

Owen

> On Nov 28, 2021, at 08:55 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:
>
>> I like to put some servers behind that scheme.
>>
>> 2601::443:xxxx for https servers
>> 2601::25:xxxx for MTA servers.
>> 2601::993:xxxx for IMAP
>>
>> It gives a quick note of what is that ip even though it’s ipv6 and usually non-human readable.
>>
>> Not sure what kind of scheme is use by medium/big ISP.
>>
>> Do you go by zip code of the area covered or some kind of logical to help people know what is behind that ipv6 network?
>
> We really aren't clever with IPv6 address design and assignment. The most we do is assign:
>
> - 1x /48 to Loopbacks for all routers.
> - /48's per PoP for infrastructure links.
> - /48's per city for /56 assignments to customers.
> - /48's per city for assignments to customers.

Why are you so stingy with customer assignments?

Why not properly assign /48s to customers and /40s to cities?

Owen

On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp> wrote:

> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.
>

I don't want to wade into the middle of this argument, but has there been
more information about the recent facebook outage released that I missed?
All I've read seems to say that the loss of connectivity to their DNS
servers was a symptom, rather than the cause of the outage.

Dave

On Sun, Nov 28, 2021 at 1:28 PM Dave Bell <me@geordish.org> wrote:
> On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>> That facebook poorly managed their DNS to cause the recent disaster
> I don't want to wade into the middle of this argument, but has
> there been more information about the recent facebook outage
> released that I missed? All I've read seems to say that the loss of
> connectivity to their DNS servers was a symptom, rather than the
> cause of the outage.

Hi Dave,

You haven't missed anything. Facebook broke its backbone by mistake.
Before they could restore it, the DNS records went stale and the
now-stale servers withdrew themselves from the network as Facebook
designed them to do when the records go stale. Unfortunately for
Facebook, the internal servers did the same thing as the external
ones. This broke the authentication system which in turn broke
everything else, complicating their efforts to access the various
systems including the ones they could have copied and pasted IP
addresses from.

But, to hear Masataka tell it, copy and paste hasn't been invented yet
so we all type IP addresses by hand on our vt100 CRT terminals.

Regards,
Bill Herrin




--
William Herrin
bill@herrin.us
https://bill.herrin.us/

On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:
> Why not properly assign /48s to customers and /40s to cities?
> ----------------------------------------------------------------------------------

Side note: I recently tried to get /48 per customer with ARIN on
repeated emails and they refused.  We were already given an IPv6 block a
while back.  I told them I wanted to expand it so I could give out a /48
per customer and that we had more than 65535 customers, which is the
block we got; 65535 /48s.  I didn't even account for our needs.

Without arguing the reasons, we will have to hand out /56s, rather than
/48s because of this.  So, it's not all /48-unicorns, puppies and rainbows.

scott

On Sat, Nov 27, 2021 at 12:18 PM William Herrin <bill@herrin.us> wrote:
>
> On Fri, Nov 26, 2021 at 3:07 PM Michael Thomas <mike@mtcc.com> wrote:
> >> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
> >> Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.
> >> Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4
>
> > This really hits my bs meter big time.
>
> If I had to guess, this is an example of correlation is not causation.
> Folks with IPv6 tend to be on savvier service providers who have
> better performance for both IPv4 and IPv6. To find out for sure,
> you'd have to do an experiment where same-user-same-server connections
> are split between IPv4 and IPv6 and then measure the performance
> difference. I don't know if anyone has done that but these particular
> articles look like someone is just looking at the high-level metrics.
> Those won't hold any statistical validity because they're not actually
> random samples.

We wrote 110+ tests for flent.org to test services under load, you can
use -4 or -6, and all the plots against all different sorts of test
conditions, can be compared against each other easily.

Example of use:

flent --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -4 -t ipv4 rrul
flent --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -6-t ipv6 rrul
flent-gui *.flent.gz # and add-other-datafiles

There are also several tests in the flent suite (like rrul46) that try
ipv4 and 6 at exactly the same time. You typically run into (ISP)
bottleneck or (DC) host bandwidth limits first, unless you also
distribute the generated load across multiple machines (I use pdsh for
this, I'd like to know of tools in modern clouds to fire off a load
generator like this, or of other load generators). We have also been
using the irtt tool at a very high resolution (3ms) to map networks'
jitter and latency at "idle" with rather interesting results for
3/4/5g, wifi and starlink, but haven't been breaking down that data
between ipv4 and ipv6 as yet.

Other useful statistics to perhaps gather at scale would be TCP_INFO
rtt, loss, marking, retransmit stats from customer-facing apache or
other proxy servers, and break those down between ipv4 and ipv6 and by
AS.

> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> bill@herrin.us
> https://bill.herrin.us/



--
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

> On 29 Nov 2021, at 09:41, scott <surfer@mauigateway.com> wrote:
>
>
> On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:
>> Why not properly assign /48s to customers and /40s to cities?
>> ----------------------------------------------------------------------------------
>
> Side note: I recently tried to get /48 per customer with ARIN on repeated emails and they refused. We were already given an IPv6 block a while back. I told them I wanted to expand it so I could give out a /48 per customer and that we had more than 65535 customers, which is the block we got; 65535 /48s. I didn't even account for our needs.
>
> Without arguing the reasons, we will have to hand out /56s, rather than /48s because of this. So, it's not all /48-unicorns, puppies and rainbows.
>
> scott

Looks like a policy omission. You should be able to grow the per customer allocation up to /48 per customer.
One shouldn’t be stuck with /56 because one made a bad choice of prefix size initially.

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

sronan@ronan-online.com wrote:

> It certainly sounds like you’ve never operated a network at scale if
> you think knowing the IP address of something reduces Operational
> expense.

It's Mark, not me, who said:

: There was a time when knowing the IP(v4) address of every interface
: of every router in your network was cool.

> The only way to truly reduce Opex at scale is automation.

Automation by what? DNS?

Masataka Ohta

Dave Bell wrote:

>> That facebook poorly managed their DNS to cause the recent disaster
>> is an important evidence to support my point that DNS, so often, may
>> not be helpful for network operations against disastrous failures,
>> including, but not limited to, DNS failures.

> I don't want to wade into the middle of this argument, but has there been
> more information about the recent facebook outage released that I missed?

You should have missed:

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
The end result was that our DNS servers became unreachable even though
they were still operational. This made it impossible for the rest of the
internet to find our servers.

> All I've read seems to say that the loss of connectivity to their DNS
> servers was a symptom, rather than the cause of the outage.

See above.

Another part of the release should also be interesting:

and second, the total loss of DNS broke many of the internal
tools we'd normally use to investigate and resolve outages
like this.

That is an evidence for my statement above "that DNS, so often,
may not be helpful for network operations against disastrous
failures".

You can't rely on automatic tools over DNS, when DNS is failing.

Masataka Ohta

William Herrin wrote:

> But, to hear Masataka tell it, copy and paste hasn't been invented yet
> so we all type IP addresses by hand on our vt100 CRT terminals.

You should be using so advanced technologies to input ASCII
text with touch and swipe, which is very slow, even slower
than cut and paste.

But, you should remember that using ASCII keyboard (of vt100
or whatever) is the fastest way to input IPv4 addresses.

Or, maybe, you can't touch type.

Masataka Ohta

On 11/29/21 00:41, scott wrote:

> Side note: I recently tried to get /48 per customer with ARIN on
> repeated emails and they refused.  We were already given an IPv6 block
> a while back.  I told them I wanted to expand it so I could give out a
> /48 per customer and that we had more than 65535 customers, which is
> the block we got; 65535 /48s.  I didn't even account for our needs.
>
> Without arguing the reasons, we will have to hand out /56s, rather
> than /48s because of this.  So, it's not all /48-unicorns, puppies and
> rainbows.

We have two types of customers - that that get assigned a /48, and those
that get assigned a /56.

Mark.

On 11/29/21 03:11, Masataka Ohta wrote:

>
> It's Mark, not me, who said:
>
> : There was a time when knowing the IP(v4) address of every interface
> : of every router in your network was cool.

In case you missed the nuance, I haven't had to do this in over 20 years.

Mark.

On 11/29/21 03:33, Masataka Ohta wrote:

> The end result was that our DNS servers became unreachable even though
> they were still operational. This made it impossible for the rest of
> the internet to find our servers.

So your suggestion to map machine addresses to human-readable names
is... what?

Or should we all just get bigger brains and remember machine addresses
by heart :-)?


> That is an evidence for my statement above "that DNS, so often,
> may not be helpful for network operations against disastrous
> failures".

Don't be drawn into Facebook's size as being what all operators on the
Internet do.

If DNS, for all operators, died in the same way it did for Facebook, I'm
certain we'd all be too busy to answer each other on this thread.

Operations significantly smaller than Facebook have had well-architected
DNS deployment for yonks. Don't let Facebook's scale leave you with the
assumption that if they cocked it up, the rest of us don't have a chance
in hell.

Mark.

Mark Tinka wrote:

>> It's Mark, not me, who said:
>>
>> : There was a time when knowing the IP(v4) address of every interface
>> : of every router in your network was cool.
>
> In case you missed the nuance, I haven't had to do this in over 20 years.

Say it to Shane, not me. That you two can not communicate well
is not my problem.

Masataka Ohta

man. 29. nov. 2021 02.12 skrev Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp>:

>
>
> > The only way to truly reduce Opex at scale is automation.
>
> Automation by what? DNS?
>
> Masataka Ohta
>


Most of our customers are provisioned by Radius. The remaining are
configured by scripting using Netconf.

We use DNS to document the network. If our DNS was down and I need to
connect to a router in some city, do you really expect me to remember the
IP address? I would have to look it up and our chosen database for that
happens to be DNS. It has some obvious advantages.

Regards

Baldur

>

I remember when I was a junior in a major NOC, we had this management host with a local hosts file for all critical components.



Probably worth reviewing some old school techniques. ????



If you can automate your gazillion routers business, you probably can also automate a couple of hosts file.



Jean



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Baldur Norddahl
Sent: November 29, 2021 4:22 AM
To: NANOG <nanog@nanog.org>
Subject: Re: IPv6 and CDN's





man. 29. nov. 2021 02.12 skrev Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp <mailto:mohta@necom830.hpcl.titech.ac.jp> >:



> The only way to truly reduce Opex at scale is automation.

Automation by what? DNS?

Masataka Ohta





Most of our customers are provisioned by Radius. The remaining are configured by scripting using Netconf.



We use DNS to document the network. If our DNS was down and I need to connect to a router in some city, do you really expect me to remember the IP address? I would have to look it up and our chosen database for that happens to be DNS. It has some obvious advantages.



Regards



Baldur

> On Nov 28, 2021, at 15:51 , Mark Andrews <marka@isc.org> wrote:
>
>
>
>> On 29 Nov 2021, at 09:41, scott <surfer@mauigateway.com> wrote:
>>
>>
>> On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:
>>> Why not properly assign /48s to customers and /40s to cities?
>>> ----------------------------------------------------------------------------------
>>
>> Side note: I recently tried to get /48 per customer with ARIN on repeated emails and they refused. We were already given an IPv6 block a while back. I told them I wanted to expand it so I could give out a /48 per customer and that we had more than 65535 customers, which is the block we got; 65535 /48s. I didn't even account for our needs.
>>
>> Without arguing the reasons, we will have to hand out /56s, rather than /48s because of this. So, it's not all /48-unicorns, puppies and rainbows.
>>
>> scott
>
> Looks like a policy omission. You should be able to grow the per customer allocation up to /48 per customer.
> One shouldn’t be stuck with /56 because one made a bad choice of prefix size initially.

There is definitely something wrong here… Policy clearly states that you should be able to obtain an allocation large enough to provide /48s to all your customers if you so choose.

In fact, it is generally quite generous beyond that point.

Owen

> On Nov 28, 2021, at 23:19 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/29/21 00:41, scott wrote:
>
>> Side note: I recently tried to get /48 per customer with ARIN on repeated emails and they refused. We were already given an IPv6 block a while back. I told them I wanted to expand it so I could give out a /48 per customer and that we had more than 65535 customers, which is the block we got; 65535 /48s. I didn't even account for our needs.
>>
>> Without arguing the reasons, we will have to hand out /56s, rather than /48s because of this. So, it's not all /48-unicorns, puppies and rainbows.
>
> We have two types of customers - that that get assigned a /48, and those that get assigned a /56.
>
> Mark.

So why be stingy to the second class? Why not just assign everyone a /48?

Owen

> On Nov 28, 2021, at 23:25 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/29/21 03:33, Masataka Ohta wrote:
>
>> The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers.
>
> So your suggestion to map machine addresses to human-readable names is... what?

If you can limit the names to the characters a-f, i, o, s, and z then it’s possible to do so with IPv6 addresses natively.
(which you can’t do in IPv4).
(o=0, i=1, s=5, and z=2)

I’m not saying this is a good idea, but it is possible. There are a large number of english words that can be spelled with just those 9 characters.

Owen