Mailing List Archive

Fred Baker wrote:

>>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>>> I disagree
>>
>> Try to type in raw IPv6 addresses.
>
> People are likely to use a technology originally developed because > IPv4 had the same perception problem: DNS.

Here in nanog, we are talking about network operations, considerable
part of which can not rely on DNS.

Masataka Ohta

On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

On 11/28/21 06:43, Masataka Ohta wrote:

>
> Here in nanog, we are talking about network operations, considerable
> part of which can not rely on DNS.

And yet Facebook were unable to access their kit to fix their recent
outage because of it (or, lack of it).

There was a time when knowing the IP(v4) address of every interface of
every router in your network was cool. I have never had to care about
that in close to 15 years. Right up there with losing interest in making
software modems work in Linux, when it was a thing :-).

If you want to remain stuck in the past, we don't have to join you.

Mark.

Ipv6 can be shorter than ipv4.

Here is the proof:

ping6 ::1

is shorter than

ping 127.1

ipv6 addresses can be very small when done properly.

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mark Tinka
Sent: November 28, 2021 5:39 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

On 11/28/21 14:09, Jean St-Laurent wrote:

> Ipv6 can be shorter than ipv4.
>
> Here is the proof:
>
> ping6 ::1
>
> is shorter than
>
> ping 127.1
>
> ipv6 addresses can be very small when done properly.

The good news is the point of an IP address is not for its own sake.

Mark.

Mark Tinka wrote:

>> Here in nanog, we are talking about network operations, considerable
>> part of which can not rely on DNS.
>
> And yet Facebook were unable to access their kit to fix their recent
> outage because of it (or, lack of it).

Exactly.

That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.

> There was a time when knowing the IP(v4) address of every interface of
> every router in your network was cool.

I surely acknowledge your point that it is impossible to do so with
MAC address based IPv6 addresses, which is why IPv6 opex is so high.

But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.

> I have never had to care about
> that in close to 15 years. Right up there with losing interest in making
> software modems work in Linux, when it was a thing :-).

So, you are saying you haven't faced real operational problems
to loss DNS information for these 15 years.

Congratulations for your luck!

> If you want to remain stuck in the past, we don't have to join you.

Surely, the recent disaster of facebook happened in the recent
past.

So what?

Masataka Ohta

On 11/28/21 14:58, Masataka Ohta wrote:

> Exactly.
>
> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.

Yes, but that does not mean that DNS is not valuable, or cannot be
hardened.

Everything can break, even an IPv4 interface on a router port. Good
practice in network operations is what keeps these kinds of problems at
bay. I mean, why else do we have lists like these?

I am certain Facebook have hardened their DNS infrastructure, and that
particular failure scenario should not recur, given all the clever
people there, and around them.


>
>> There was a time when knowing the IP(v4) address of every interface
>> of every router in your network was cool.
>
> I surely acknowledge your point that it is impossible to do so with
> MAC address based IPv6 addresses, which is why IPv6 opex is so high.
>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.

That's just satisfying one's mental (or emotional) nits.

Routers (and customers) don't care about how anally we assign address
space. As long as it is compliant, does not conflict, and is correctly
routed.

That we cannot transpose our IPv4 mental/emotional habits on to IPv6
does not make IPv6 more complicated. It just makes us more stuck in our
ways.

After all, DHCPv6 still does not offer a default gateway.


> So, you are saying you haven't faced real operational problems
> to loss DNS information for these 15 years.
>
> Congratulations for your luck!

I am sure I have had a DNS issue of some sort or other in the past 15
years. The fact that I can't remember what it was is telling.


> Surely, the recent disaster of facebook happened in the recent
> past.
>
> So what?

And they have learned from it, and I dare say, fixed it.

Facebook will neither be disposing of DNS any time soon, nor will they
be dropping IPv6.

Mark.

søn. 28. nov. 2021 13.59 skrev Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp>:

>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.
>

99% if not 100% of our subnets have either only routers or only hosts + a
gateway. So that would be a strange rule to follow. Also very expensive if
we are talking public addressing.

I find that 10.x.y.z is not much if you want to have a system in your
subnet numbering. With ipv6 there is much more space to enable systematic
numbering schemes.

>

Mark Tinka wrote:

>> That facebook poorly managed their DNS to cause the recent disaster
>> is an important evidence to support my point that DNS, so often, may
>> not be helpful for network operations against disastrous failures,
>> including, but not limited to, DNS failures.
>
> Yes, but that does not mean that DNS is not valuable, or cannot be
> hardened.

As a person who proposed anycast DNS servers, against which facebook
operated their DNS, I'm so sure you are right.

> I am certain Facebook have hardened their DNS infrastructure, and that
> particular failure scenario should not recur, given all the clever
> people there, and around them.

All I can see is that there were a lot of stupid people in facebook.

I really hope less of them still remain there.

Masataka Ohta

On 11/28/21 15:33, Masataka Ohta wrote:

> As a person who proposed anycast DNS servers, against which facebook
> operated their DNS, I'm so sure you are right.

Facebook's mistake on this is an easily fixable one.

We've all been there. Nothing groundbreaking.


> All I can see is that there were a lot of stupid people in facebook.
>
> I really hope less of them still remain there.

Can't help you there...

Mark.

Baldur Norddahl wrote:

>> But, with manually configured IP addresses, it is trivially easy
>> to have a rule to assign lower part of IP addresses within a subnet
>> for hosts and upper part for routers, which is enough to troubleshoot
>> most network failures.

> 99% if not 100% of our subnets have either only routers or only hosts + a
> gateway.

It merely means you should not use MAC address based IP addresses for,
at least, routers, which is partly why opex of IPv4 is low.

> So that would be a strange rule to follow. Also very expensive if we
> are talking public addressing.
>
> I find that 10.x.y.z is not much if you want to have a system in
> your subnet numbering.

In most cases, it is enough that addresses are unique within
certain domain, which is why many ISPs are assigning addresses,
not guaranteed to be globally unique, to there internal routers.

> With ipv6 there is much more space to enable systematic numbering
> schemes.

More space, only to encourage stupid idea of MAC address based
addresses of IPv6/ND, is not required for systematic numbering.

Masataka Ohta

On 11/28/21 15:59, Masataka Ohta wrote:

> It merely means you should not use MAC address based IP addresses for,
> at least, routers, which is partly why opex of IPv4 is low.

I often wonder what Internet you use :-)...


> More space, only to encourage stupid idea of MAC address based
> addresses of IPv6/ND, is not required for systematic numbering.

I'm sure a router won't stop working because we did not assign it an IP
address "systematically". Nor will the spinning of the earth.

Mark.

Mark Tinka wrote:

>> As a person who proposed anycast DNS servers, against which facebook
>> operated their DNS, I'm so sure you are right.
>
> Facebook's mistake on this is an easily fixable one.

Certainly, but, merely because it is an easily avoided one.

> We've all been there.

People who really understand DNS, including but not limited to
anycast one, have never been there.

Masataka Ohta

I like to put some servers behind that scheme.



2601::443:xxxx for https servers

2601::25:xxxx for MTA servers.

2601::993:xxxx for IMAP



It gives a quick note of what is that ip even though it’s ipv6 and usually non-human readable.



Not sure what kind of scheme is use by medium/big ISP.



Do you go by zip code of the area covered or some kind of logical to help people know what is behind that ipv6 network?



Jean



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Baldur Norddahl
Sent: November 28, 2021 8:22 AM
To: NANOG <nanog@nanog.org>
Subject: Re: IPv6 and CDN's





søn. 28. nov. 2021 13.59 skrev Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp <mailto:mohta@necom830.hpcl.titech.ac.jp> >:


But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.



99% if not 100% of our subnets have either only routers or only hosts + a gateway. So that would be a strange rule to follow. Also very expensive if we are talking public addressing.



I find that 10.x.y.z is not much if you want to have a system in your subnet numbering. With ipv6 there is much more space to enable systematic numbering schemes.

On 11/28/21 16:13, Masataka Ohta wrote:

>
> Certainly, but, merely because it is an easily avoided one.

None of the us came out the womb knowing anything. We learned as we went
along. And we keep learning, right until our death.

To expect experience before it is experienced has always been unreasonable.


> People who really understand DNS, including but not limited to
> anycast one, have never been there.

Anycast was the result of things that broke. And even with Anycast, lots
of things break, still.

Continuity of service != never having been there.

Mark.

It certainly sounds like you’ve never operated a network at scale if you think knowing the IP address of something reduces Operational expense. The only way to truly reduce Opex at scale is automation.

Shane

> On Nov 28, 2021, at 9:13 AM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> ?Mark Tinka wrote:
>
>>> As a person who proposed anycast DNS servers, against which facebook
>>> operated their DNS, I'm so sure you are right.
>> Facebook's mistake on this is an easily fixable one.
>
> Certainly, but, merely because it is an easily avoided one.
>
>> We've all been there.
>
> People who really understand DNS, including but not limited to
> anycast one, have never been there.
>
> Masataka Ohta

On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:

> I like to put some servers behind that scheme.
>
> 2601::443:xxxx for https servers
>
> 2601::25:xxxx for MTA servers.
>
> 2601::993:xxxx for IMAP
>
> It gives a quick note of what is that ip even though it’s ipv6 and
> usually non-human readable.
>
> Not sure what kind of scheme is use by medium/big ISP.
>
> Do you go by zip code of the area covered or some kind of logical to
> help people know what is behind that ipv6 network?
>

We really aren't clever with IPv6 address design and assignment. The
most we do is assign:

    - 1x /48 to Loopbacks for all routers.
    - /48's per PoP for infrastructure links.
    - /48's per city for /56 assignments to customers.
    - /48's per city for assignments to customers.

We don't try to co-ordinate them in a "meaningful" way that would offer
visual identification. We feel that is just too much work, and that
getting IPv4/IPv6 parity for day-to-day operations is a challenge in and
of itself, without trying to be fancy.

Mark.

> On Nov 27, 2021, at 19:37 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Mark Tinka wrote:
>
>> On 11/27/21 17:07, Masataka Ohta wrote:
>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>> I disagree
>
> Try to type in raw IPv6 addresses.

Rarely necessary in the modern age, but really not significantly more difficult than IPv4 once
you become accustomed to it.

Owen

> On Nov 28, 2021, at 02:42 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/28/21 06:43, Masataka Ohta wrote:
>
>>
>> Here in nanog, we are talking about network operations, considerable
>> part of which can not rely on DNS.
>
> And yet Facebook were unable to access their kit to fix their recent outage because of it (or, lack of it).

I’d argue that failing to put the correct documentation in place for coping with a DNS outage was the bigger
issue than the DNS failure in the Facebook outage… So would a number of the engineers I know at Facebook.

> There was a time when knowing the IP(v4) address of every interface of every router in your network was cool. I have never had to care about that in close to 15 years. Right up there with losing interest in making software modems work in Linux, when it was a thing :-).

There was a time when every router in a moderately large network was less than 50. Those days are gone.

Today, it’s impossible to build large scale networks without depending on certain tools. That means that the
failure of those tools can be catastrophic if one is not properly prepared. This is simply the modern reality.
Proper preparation is harder than it used to be, but for any such network, there should be online and off-line
copies of sufficient documentation (which is adequately maintained) to cope restore service of any such
underlying facility quickly in the event of a failure.

Owen

> On Nov 28, 2021, at 04:58 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Mark Tinka wrote:
>
>>> Here in nanog, we are talking about network operations, considerable
>>> part of which can not rely on DNS.
>> And yet Facebook were unable to access their kit to fix their recent outage because of it (or, lack of it).
>
> Exactly.
>
> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.

I’d argue that the true failure was failure to document the system adequately to provide for prompt
resolution of the DNS problems in the absence of DNS and failure to properly distribute that knowledge
to those that would need it in such a circumstance.


>> There was a time when knowing the IP(v4) address of every interface of every router in your network was cool.
>
> I surely acknowledge your point that it is impossible to do so with
> MAC address based IPv6 addresses, which is why IPv6 opex is so high.

Hardly anyone uses MAC address based IPv6 addresses for anything, so your claim here is specious.

Statically or manually configured IPv6 addresses are no more difficult than the equivalent in IPv4, just
slightly longer.

Consider, for example, 192.159.10.7 vs. 2620:0:930::400:7

The 400 could have been omitted leaving 2620:0:930::7, but I chose to have additional flexibility that
is available from IPv6 in classifying addresses for particular purposes and decided to use the next
order quartet for that purpose.

> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.

It’s equally easy to do that in IPv6 as well (while I prefer to use the reverse strategy, using low order
addresses for routers and higher numbers for non-forwarding hosts).

>> I have never had to care about that in close to 15 years. Right up there with losing interest in making software modems work in Linux, when it was a thing :-).
>
> So, you are saying you haven't faced real operational problems
> to loss DNS information for these 15 years.
>
> Congratulations for your luck!

In reality, DNS properly implemented is extraordinarily reliable these days. It’s not completely failure-proof,
as Facebook recently demonstrated so thoroughly, but consider how extraordinary that failure was in order
to garner so much attention. Gone are the days when DNS failures were a part of daily operational life
that we simply accepted.

>> If you want to remain stuck in the past, we don't have to join you.
>
> Surely, the recent disaster of facebook happened in the recent
> past.

Yes, but if you really look at it, that was a failure of preparedness much more than a failure of DNS.

> So what?

So, the world has moved on and modern networks are built using tools you appear to prefer not to
depend on… I remember when an RS-232 port was the gold standard of being able to recover your
router. Today, we accept at least USB and in most cases even Ethernet as a viable alternative.

Things get better, but that comes with higher-level dependencies on underlying infrastructure. Fortunately,
the underlying infrastructure gets better as well, making that possible and even reasonable. The fact
that you prefer not to recognize this is a sign that you are failing to adapt to the present in favor of
living in the past as Mark stated.

Owen

> On Nov 28, 2021, at 08:55 , Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:
>
>> I like to put some servers behind that scheme.
>>
>> 2601::443:xxxx for https servers
>> 2601::25:xxxx for MTA servers.
>> 2601::993:xxxx for IMAP
>>
>> It gives a quick note of what is that ip even though it’s ipv6 and usually non-human readable.
>>
>> Not sure what kind of scheme is use by medium/big ISP.
>>
>> Do you go by zip code of the area covered or some kind of logical to help people know what is behind that ipv6 network?
>
> We really aren't clever with IPv6 address design and assignment. The most we do is assign:
>
> - 1x /48 to Loopbacks for all routers.
> - /48's per PoP for infrastructure links.
> - /48's per city for /56 assignments to customers.
> - /48's per city for assignments to customers.

Why are you so stingy with customer assignments?

Why not properly assign /48s to customers and /40s to cities?

Owen

On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp> wrote:

> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.
>

I don't want to wade into the middle of this argument, but has there been
more information about the recent facebook outage released that I missed?
All I've read seems to say that the loss of connectivity to their DNS
servers was a symptom, rather than the cause of the outage.

Dave

On Sun, Nov 28, 2021 at 1:28 PM Dave Bell <me@geordish.org> wrote:
> On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>> That facebook poorly managed their DNS to cause the recent disaster
> I don't want to wade into the middle of this argument, but has
> there been more information about the recent facebook outage
> released that I missed? All I've read seems to say that the loss of
> connectivity to their DNS servers was a symptom, rather than the
> cause of the outage.

Hi Dave,

You haven't missed anything. Facebook broke its backbone by mistake.
Before they could restore it, the DNS records went stale and the
now-stale servers withdrew themselves from the network as Facebook
designed them to do when the records go stale. Unfortunately for
Facebook, the internal servers did the same thing as the external
ones. This broke the authentication system which in turn broke
everything else, complicating their efforts to access the various
systems including the ones they could have copied and pasted IP
addresses from.

But, to hear Masataka tell it, copy and paste hasn't been invented yet
so we all type IP addresses by hand on our vt100 CRT terminals.

Regards,
Bill Herrin




--
William Herrin
bill@herrin.us
https://bill.herrin.us/

On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:
> Why not properly assign /48s to customers and /40s to cities?
> ----------------------------------------------------------------------------------

Side note: I recently tried to get /48 per customer with ARIN on
repeated emails and they refused.  We were already given an IPv6 block a
while back.  I told them I wanted to expand it so I could give out a /48
per customer and that we had more than 65535 customers, which is the
block we got; 65535 /48s.  I didn't even account for our needs.

Without arguing the reasons, we will have to hand out /56s, rather than
/48s because of this.  So, it's not all /48-unicorns, puppies and rainbows.

scott

On Sat, Nov 27, 2021 at 12:18 PM William Herrin <bill@herrin.us> wrote:
>
> On Fri, Nov 26, 2021 at 3:07 PM Michael Thomas <mike@mtcc.com> wrote:
> >> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
> >> Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.
> >> Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4
>
> > This really hits my bs meter big time.
>
> If I had to guess, this is an example of correlation is not causation.
> Folks with IPv6 tend to be on savvier service providers who have
> better performance for both IPv4 and IPv6. To find out for sure,
> you'd have to do an experiment where same-user-same-server connections
> are split between IPv4 and IPv6 and then measure the performance
> difference. I don't know if anyone has done that but these particular
> articles look like someone is just looking at the high-level metrics.
> Those won't hold any statistical validity because they're not actually
> random samples.

We wrote 110+ tests for flent.org to test services under load, you can
use -4 or -6, and all the plots against all different sorts of test
conditions, can be compared against each other easily.

Example of use:

flent --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -4 -t ipv4 rrul
flent --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -6-t ipv6 rrul
flent-gui *.flent.gz # and add-other-datafiles

There are also several tests in the flent suite (like rrul46) that try
ipv4 and 6 at exactly the same time. You typically run into (ISP)
bottleneck or (DC) host bandwidth limits first, unless you also
distribute the generated load across multiple machines (I use pdsh for
this, I'd like to know of tools in modern clouds to fire off a load
generator like this, or of other load generators). We have also been
using the irtt tool at a very high resolution (3ms) to map networks'
jitter and latency at "idle" with rather interesting results for
3/4/5g, wifi and starlink, but haven't been breaking down that data
between ipv4 and ipv6 as yet.

Other useful statistics to perhaps gather at scale would be TCP_INFO
rtt, loss, marking, retransmit stats from customer-facing apache or
other proxy servers, and break those down between ipv4 and ipv6 and by
AS.

> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> bill@herrin.us
> https://bill.herrin.us/



--
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC