Mailing List Archive

>> OK, then Disney+ or Hulu or whoever. Peering wars never end well. Don't even need postcards, just stick the flyer in with the bill.
>
> Is that really cheaper and easier than deploying IPv6? Really?

The cost of putting flyers in the bills rounds to zero, so yes, really.
I expect these companies all have plans to support v6 eventually, someday,
once they're retired and replaced all of the old junk that handles v6
poorly or not at all, but you know about accountants and depreciation.

> Remember, I didn’t suggest that Comcast turn off IPv4… I suggested that
> Comcast start passing along some of the added expense of maintaining
> IPv4 connectivity to the customers that want it.

Ah, so they should start adding a gratuitous charge for a feature they
have always provided as part of the basic service. How many milliseconds
do you think it'll take for the Congress to haul their CEO in front of a
committee?

> That day may not be today, but it is coming and I don’t think it’s as far off as you imagine.

Nothing personal, but people have been saying exactly that for about 25
years now. Please forgive me if I continue not to hold my breath.

R's,
John

> On Sep 17, 2021, at 21:03 , John R. Levine <johnl@iecc.com> wrote:
>
>>> OK, then Disney+ or Hulu or whoever. Peering wars never end well. Don't even need postcards, just stick the flyer in with the bill.
>>
>> Is that really cheaper and easier than deploying IPv6? Really?
>
> The cost of putting flyers in the bills rounds to zero, so yes, really. I expect these companies all have plans to support v6 eventually, someday, once they're retired and replaced all of the old junk that handles v6 poorly or not at all, but you know about accountants and depreciation.

Unless their infrastructure runs significantly on hardware and software pre-2004 (unlikely), so does the cost of adding IPv6 to their content servers. Especially if they’re using a CDN such as Akamai.
>
>> Remember, I didn’t suggest that Comcast turn off IPv4… I suggested that Comcast start passing along some of the added expense of maintaining IPv4 connectivity to the customers that want it.
>
> Ah, so they should start adding a gratuitous charge for a feature they have always provided as part of the basic service. How many milliseconds do you think it'll take for the Congress to haul their CEO in front of a committee?

I doubt that their CEO would get hauled in at all. Even if he did, I would think this would be one of the easiest hearings ever for them as literally, they could easily show the increased costs of continuing to provide IPv4 services and note that they didn’t want to jack up everyone’s bills to support the customers that need IPv4, so they’ve made IPv4 an opt-in value added service instead of punishing customers that don’t patronize laggards. (I’m sure their lawyers would say it better, I’m blunt).

>> That day may not be today, but it is coming and I don’t think it’s as far off as you imagine.
>
> Nothing personal, but people have been saying exactly that for about 25 years now. Please forgive me if I continue not to hold my breath.

Nope… People have been saying that IPv4 would stop being the lingua franca of the internet for 25+ years. There’s still hope for that, but I agree it’s been disappointingly and tragically slow.

OTOH, the idea of doing cost-recovery on the additional nuisance that is IPv4 is relatively novel and hasn’t been floated that I’m aware of more than 3 or 4 years ago.

Bottom line is that IPv4 continues to increase costs for eyeball providers. They’ll need to recover that cost. At some point, the cost will be enough of a differentiator that customers willing to accept v6-only service will get a break.

If they don’t do it as an IPv4 surcharge, they could do it as an IPv6-only discount… That might even be better. Either way, the net effect is the same… Suddenly, customers have a monetary advantage to push their content providers toward providing v6.

Owen

It appears that Owen DeLong via NANOG <owen@delong.com> said:
>> The cost of putting flyers in the bills rounds to zero, so yes, really. I expect these companies all have plans
>to support v6 eventually, someday, once they're retired and replaced all of the old junk that handles v6 poorly or
>not at all, but you know about accountants and depreciation.
>
>Unless their infrastructure runs significantly on hardware and software pre-2004 (unlikely), so does the cost of
>adding IPv6 to their content servers. Especially if they’re using a CDN such as Akamai.

I wasn't talking about switches and routers. I was talking about every single piece of software and equipment that
they use for support and marketing and customer service and all the other stuff that big companies do.

As I may have said once or twice, eventuallly it'll all be replaced so it works on IPv6 but we're not holding our breath.

R's,
John

On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl@iecc.com> wrote:

> It appears that Owen DeLong via NANOG <owen@delong.com> said:
> >> The cost of putting flyers in the bills rounds to zero, so yes, really.
> I expect these companies all have plans
> >to support v6 eventually, someday, once they're retired and replaced all
> of the old junk that handles v6 poorly or
> >not at all, but you know about accountants and depreciation.
> >
> >Unless their infrastructure runs significantly on hardware and software
> pre-2004 (unlikely), so does the cost of
> >adding IPv6 to their content servers. Especially if they’re using a CDN
> such as Akamai.
>
> I wasn't talking about switches and routers. I was talking about every
> single piece of software and equipment that
> they use for support and marketing and customer service and all the other
> stuff that big companies do.
>
> As I may have said once or twice, eventuallly it'll all be replaced so it
> works on IPv6 but we're not holding our breath.
>

Glad you noted this. Thinking this was/is purely a hardware cycle problem
related to normal/forced upgrade strategies. On that point, most hardware
I know of from 2004 in larger networks is long fully depreciated and
sweating assets 15+ years can happen, but I don't personally think this is
the biggest issue.

As you noted John, its the plethora of software, support systems, tooling,
and most important in many environments - legacy customer management and
provisioning systems that can be the limiting factor. I recall looking
back when leading IPv6 turn-up, those were the biger problems to solve
for. Often such systems are extremely expensive to touch and working on
them required prioritization against direct revenue generating projects
(opportunity cost) . Replacing routers was just a money problem.

I am by far not saying I agree with choices made by hold-outs, but I also
understand this is for many, not just an engineering problem to solve.

regards,

Victor K


>
> R's,
> John
>

> As you noted John, its the plethora of software, support systems, tooling,
> and most important in many environments - legacy customer management and
> provisioning systems that can be the limiting factor. ...

Just looking around my office, I have a Cisco SPA112 two-port ATA. It's
been discontinued but Cisco says they will support it through 2025 and
they shipped a firmware update in 2019. It works fine on IPv4 behind a
NAT. It has no v6 support at all and never will. Multiply that by a
zillion and you can see that IPv4 isn't going away any time soon.

Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On Fri, 17 Sep 2021 21:15:00 -0700
Owen DeLong via NANOG <nanog@nanog.org> wrote:

> Unless their infrastructure runs significantly on hardware and
> software pre-2004 (unlikely), so does the cost of adding IPv6 to
> their content servers. Especially if they’re using a CDN such as
> Akamai.

Owen, I have nothing but respect for you, but this is a
fantasy... I provide FTTx services to business and residential. I had
to fight, and grind, and test for over a year to get a mix of hardware
and software that would provide anything resembling IPv6 equivalent
services to most of our customers. The only devices in my network that
worked with few problems are my Adtran gpon/xgs-pon cards (try to find
DHCPv6 option-18 support on anything else)... EVERYTHING else I used
from my Juniper routers to customer CPE had to go through more rounds
of testing and bug fixes than I could name - for years.

I've provided static v6 services to business customers for a
long time (with no takers), but dynamic, scalable residential services
was very hard. There are still holes in our infrastructure because most
vendors I am dealing with are doing very little to no v6 testing and
still think I am a weirdo for asking for it. Every ACS vendor is
either just now working on it, or thinks they have it until I point out
to them that they don't. There have been some vendors that were good
to work with: Juniper fixed the bugs I reported once I was able to
prove to them it really was on there end (DHCPv6 relay server). ZyXel
has been good to work with; they care about and fix bugs that are
reported.

There are also big vendors I won't work with any more because
they do not have full v6 support for features I need, and they have no
plans to have it. I'm not a big enough customer for them to care about
what I want. I have devices with 2 year old software and zero v6
support and none is coming ever; these are not no-name vendors; they
are big.

People who think modern equipment is ready to provide native
dual-stack services at scale to their customers are either using stuff
very similar to mine, or are simply not doing it yet or have a lot of
compromises.

--TimH

I concur that the problem is not a routing hardware problem. It's a
perception problem with the various ISPs. I have fiber service with AT&T.

My little server farm endpoints all have IPv6 addresses, including the
edge router. I also have a plan to allocate IPv6 addresses to my LAN
devices, and to protect the LAN devices from outside interference by
rules in the NFTABLES firewall that include connection tracking on
outbound requests. (IPv4 will still use NAT to keep nefarious people
from probing my internals.)

Specifically, when I was doing my mail server refresh (moving from Red
Hat to Canonical) I decided it was time to offer IPv6 connectivity in
the mail server to "future proof" my setup. That included adding AAAA
records in my DNS zone files. Failure! The issues:

1. I learned that there are no "static addresses" in IPv6, as far as
AT&T was concerned. By all appearances, though, the IPv6 /64 is
relatively static, for now, similar to the way that early cable modem
deployments kept the same IPv4 addresses. (Until the cable people
started forcing changes on DHCP lease renewal, that is.)

2. My request for PTR records was denied, which means I can't satisfy
Best Practices for a mail server in the IPv6 space. No PTR records, no
redirection of ip6.apra space, nothing. I include AT&T's refusal below.

3. I don't know how to get an IPv6 allocation from ARIN, how to request
AT&T to route it, or how to deal with the DNS server issues. Oh, I know
how to configure BIND9; I would prefer using a 24/7/365 provider. For
example, my master zone files are with Register.com, so if my circuit
goes down the name resolution still happens. Register.com appears not
to provide reverse-DNS PTR zone support (in6.arpa). A Google search
turned up NOTHING for in6.arpa hosting.

That tells me that IPv6 is not "Internet Ready" for small users. Given
the level of FU responses I get trying to work with it, I will stop
banging my head against the wall.

So I stick with IPv4, because that will be the "standard" until the day
I die, as far as I can tell.

(I removed the AAAA record, so as not to confuse mail server that DO
operate IPv6.)

> Subject: RE: Need IPv6 PTR record for my IPv6 mail server
> Date: Mon, 19 Jul 2021 12:52:53 +0000
> From: Prov-DNS <prov-dns@att.com>
> To: Prov-DNS <prov-dns@att.com>, att@satchell.net <att@satchell.net>
>
>
> Hello
> We don't process DNS request on IPv6 addresses. We only process DNS
> request on IPv4 static assigned addresses. If you would like us to
> process a DNS request for you on a IPv4 address please provide the
> following information.
>
> IPv4 address you would like the record created for Host name you would > like that IP address pointed to
>
> Thanks
> Michael AT&T Prov-DNS
>
> -----Original Message-----
> From: Stephen Satchell <att@satchell.net>
> Sent: Friday, July 16, 2021 5:42 PM
> To: DNSUpdates cB <g12988@att.com>
> Subject: Need IPv6 PTR record for my IPv6 mail server
>
> Here is the record I need inserted into your ip6.arpa DNS zone:
>
>>
2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. 0
IN PTR smtp.satchell.net.
>
> This is the result from the question section of a dig(1) request for the PTR record for my IPv6 address 2600:1700:79b0:ddc0::32, and the fully-qualified domain name of the server.
>
> You can verify the information using dig smtp.satchell.net AAAA and
> checking the reverse.
>
> This is the only server in my collection that needs the IPv6 pointer.

I mostly agree with this. Even new hardware like eero doesn't do v6 by default. It's just off. So many things are like this. It's nice that LTE and other deployments have v6 by default. Last time I knew providers like t mobile are great but their MVNOs like Ultra Mobile do not do v6.

All this and we will get there. I'm expecting another decade or two though.

Sent from my TI-99/4a

> On Sep 18, 2021, at 4:29 PM, John R. Levine <johnl@iecc.com> wrote:
>
> IPv4 isn't going away any time soon.

Also, I realise I'm kinda taking your comment out of context and
jumping on it to harp on my favorite pet peeve, so, yeah, sorry about
that.

--TimH

On Sat, 18 Sep 2021 13:28:02 -0700
Tim Howe <tim.h@bendtel.com> wrote:

> On Fri, 17 Sep 2021 21:15:00 -0700
> Owen DeLong via NANOG <nanog@nanog.org> wrote:
>
> > Unless their infrastructure runs significantly on hardware and
> > software pre-2004 (unlikely), so does the cost of adding IPv6 to
> > their content servers. Especially if they’re using a CDN such as
> > Akamai.
>
> Owen, I have nothing but respect for you, but this is a
> fantasy... I provide FTTx services to business and residential. I had
> to fight, and grind, and test for over a year to get a mix of hardware
> and software that would provide anything resembling IPv6 equivalent
> services to most of our customers. The only devices in my network that
> worked with few problems are my Adtran gpon/xgs-pon cards (try to find
> DHCPv6 option-18 support on anything else)... EVERYTHING else I used
> from my Juniper routers to customer CPE had to go through more rounds
> of testing and bug fixes than I could name - for years.
>
> I've provided static v6 services to business customers for a
> long time (with no takers), but dynamic, scalable residential services
> was very hard. There are still holes in our infrastructure because most
> vendors I am dealing with are doing very little to no v6 testing and
> still think I am a weirdo for asking for it. Every ACS vendor is
> either just now working on it, or thinks they have it until I point out
> to them that they don't. There have been some vendors that were good
> to work with: Juniper fixed the bugs I reported once I was able to
> prove to them it really was on there end (DHCPv6 relay server). ZyXel
> has been good to work with; they care about and fix bugs that are
> reported.
>
> There are also big vendors I won't work with any more because
> they do not have full v6 support for features I need, and they have no
> plans to have it. I'm not a big enough customer for them to care about
> what I want. I have devices with 2 year old software and zero v6
> support and none is coming ever; these are not no-name vendors; they
> are big.
>
> People who think modern equipment is ready to provide native
> dual-stack services at scale to their customers are either using stuff
> very similar to mine, or are simply not doing it yet or have a lot of
> compromises.
>
> --TimH

It tells you that AT&T don’t treat IPv6 on equal footing to IPv4 and nothing more.

There is nothing at the protocol level stopping AT&T offering a similar level of service. Don’t equate poor implementation with the protocol being broken.
--
Mark Andrews

> On 19 Sep 2021, at 07:19, Stephen Satchell <list@satchell.net> wrote:
>
> ?I concur that the problem is not a routing hardware problem. It's a perception problem with the various ISPs. I have fiber service with AT&T.
>
> My little server farm endpoints all have IPv6 addresses, including the edge router. I also have a plan to allocate IPv6 addresses to my LAN devices, and to protect the LAN devices from outside interference by rules in the NFTABLES firewall that include connection tracking on outbound requests. (IPv4 will still use NAT to keep nefarious people from probing my internals.)
>
> Specifically, when I was doing my mail server refresh (moving from Red Hat to Canonical) I decided it was time to offer IPv6 connectivity in the mail server to "future proof" my setup. That included adding AAAA records in my DNS zone files. Failure! The issues:
>
> 1. I learned that there are no "static addresses" in IPv6, as far as AT&T was concerned. By all appearances, though, the IPv6 /64 is relatively static, for now, similar to the way that early cable modem deployments kept the same IPv4 addresses. (Until the cable people started forcing changes on DHCP lease renewal, that is.)
>
> 2. My request for PTR records was denied, which means I can't satisfy Best Practices for a mail server in the IPv6 space. No PTR records, no redirection of ip6.apra space, nothing. I include AT&T's refusal below.
>
> 3. I don't know how to get an IPv6 allocation from ARIN, how to request AT&T to route it, or how to deal with the DNS server issues. Oh, I know how to configure BIND9; I would prefer using a 24/7/365 provider. For example, my master zone files are with Register.com, so if my circuit goes down the name resolution still happens. Register.com appears not to provide reverse-DNS PTR zone support (in6.arpa). A Google search turned up NOTHING for in6.arpa hosting.
>
> That tells me that IPv6 is not "Internet Ready" for small users. Given the level of FU responses I get trying to work with it, I will stop banging my head against the wall.
>
> So I stick with IPv4, because that will be the "standard" until the day I die, as far as I can tell.
>
> (I removed the AAAA record, so as not to confuse mail server that DO operate IPv6.)
>
>> Subject: RE: Need IPv6 PTR record for my IPv6 mail server
>> Date: Mon, 19 Jul 2021 12:52:53 +0000
>> From: Prov-DNS <prov-dns@att.com>
>> To: Prov-DNS <prov-dns@att.com>, att@satchell.net <att@satchell.net>
>> Hello We don't process DNS request on IPv6 addresses. We only process DNS
>> request on IPv4 static assigned addresses. If you would like us to
>> process a DNS request for you on a IPv4 address please provide the
>> following information.
>> IPv4 address you would like the record created for Host name you would > like that IP address pointed to
> >
>> Thanks
>> Michael AT&T Prov-DNS
>> -----Original Message-----
>> From: Stephen Satchell <att@satchell.net>
>> Sent: Friday, July 16, 2021 5:42 PM
>> To: DNSUpdates cB <g12988@att.com>
>> Subject: Need IPv6 PTR record for my IPv6 mail server
>> Here is the record I need inserted into your ip6.arpa DNS zone:
> 2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. 0 IN PTR smtp.satchell.net.
>> This is the result from the question section of a dig(1) request for the PTR record for my IPv6 address 2600:1700:79b0:ddc0::32, and the fully-qualified domain name of the server.
>> You can verify the information using dig smtp.satchell.net AAAA and checking the reverse.
>> This is the only server in my collection that needs the IPv6 pointer.
>

According to Mark Andrews <marka@isc.org>:
>It tells you that AT&T don’t treat IPv6 on equal footing to IPv4 and nothing more.

Indeed but since AT&T is about 1/4 of the US broadband market, and our screwed up telco
politics means there is often no practical competitor available, that's a big problem.

R's,
John

PS: that's separate from what he said about equipment which nomninally has v6 support but not
in a way that you can actually use.
--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

> On Sep 18, 2021, at 11:37 , John Levine <johnl@iecc.com> wrote:
>
> It appears that Owen DeLong via NANOG <owen@delong.com> said:
>>> The cost of putting flyers in the bills rounds to zero, so yes, really. I expect these companies all have plans
>> to support v6 eventually, someday, once they're retired and replaced all of the old junk that handles v6 poorly or
>> not at all, but you know about accountants and depreciation.
>>
>> Unless their infrastructure runs significantly on hardware and software pre-2004 (unlikely), so does the cost of
>> adding IPv6 to their content servers. Especially if they’re using a CDN such as Akamai.
>
> I wasn't talking about switches and routers. I was talking about every single piece of software and equipment that
> they use for support and marketing and customer service and all the other stuff that big companies do.

That doesn’t all have to change in order to make their services available on IPv6 also.

IPv6 is not an all-or-nothing thing.

If your backend is all IPv4 all the time and you want to keep it that way, more power to you. I encourage my competitors to try that.

However, if your customer-facing services are IPv4-only, that’s not hard to fix in most cases and it’s really obnoxious not to do so.

> As I may have said once or twice, eventuallly it'll all be replaced so it works on IPv6 but we're not holding our breath.

I’m not holding my breath, but I’m also trying to argue reasonable approaches and realistic solutions here.

You seem to be looking for excuses to claim the problem that needs to be solved is harder than it is to justify not solving it.

Owen

> On Sep 18, 2021, at 12:34 , Victor Kuarsingh <victor@jvknet.com> wrote:
>
>
>
> On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl@iecc.com <mailto:johnl@iecc.com>> wrote:
> It appears that Owen DeLong via NANOG <owen@delong.com <mailto:owen@delong.com>> said:
> >> The cost of putting flyers in the bills rounds to zero, so yes, really. I expect these companies all have plans
> >to support v6 eventually, someday, once they're retired and replaced all of the old junk that handles v6 poorly or
> >not at all, but you know about accountants and depreciation.
> >
> >Unless their infrastructure runs significantly on hardware and software pre-2004 (unlikely), so does the cost of
> >adding IPv6 to their content servers. Especially if they’re using a CDN such as Akamai.
>
> I wasn't talking about switches and routers. I was talking about every single piece of software and equipment that
> they use for support and marketing and customer service and all the other stuff that big companies do.
>
> As I may have said once or twice, eventuallly it'll all be replaced so it works on IPv6 but we're not holding our breath.
>
> Glad you noted this. Thinking this was/is purely a hardware cycle problem related to normal/forced upgrade strategies. On that point, most hardware I know of from 2004 in larger networks is long fully depreciated and sweating assets 15+ years can happen, but I don't personally think this is the biggest issue.

It’s certainly not purely hardware, but it also doesn’t require solving the entire problem across the entire backend.

What is urgently needed is to fix the customer-facing front end so that it speaks both protocols (or at least speaks IPv6).

> As you noted John, its the plethora of software, support systems, tooling, and most important in many environments - legacy customer management and provisioning systems that can be the limiting factor. I recall looking back when leading IPv6 turn-up, those were the biger problems to solve for. Often such systems are extremely expensive to touch and working on them required prioritization against direct revenue generating projects (opportunity cost) . Replacing routers was just a money problem.

Why do those systems have to be updated in order to deliver the service to the customer in both protocols?

I mean sure, you’ll probably need to fix some logging databases that think that a customers address can be logged as a uint32_t, but that’s a relatively small number of systems that need to be updated in that case and it’s not like expanding the field to uint128_t in the database is incompatible with the old software during the upgrade process.

> I am by far not saying I agree with choices made by hold-outs, but I also understand this is for many, not just an engineering problem to solve.

I agree there are some systems that may make this more complex in some environments, but I don’t agree that it’s
impossible (or even particularly difficult) for a content provider to deliver their content on both protocols today even
if they don’t upgrade their back-end systems.

Owen

> On Sep 18, 2021, at 13:25 , John R. Levine <johnl@iecc.com> wrote:
>
>> As you noted John, its the plethora of software, support systems, tooling,
>> and most important in many environments - legacy customer management and
>> provisioning systems that can be the limiting factor. ...
>
> Just looking around my office, I have a Cisco SPA112 two-port ATA. It's been discontinued but Cisco says they will support it through 2025 and they shipped a firmware update in 2019. It works fine on IPv4 behind a NAT. It has no v6 support at all and never will. Multiply that by a zillion and you can see that IPv4 isn't going away any time soon.

This isn’t about removing IPv4 from every last corner of the internet. It’s about adding IPv6 to the content providers that are blocking the ability to build new eyeball networks v6 only.

Owen

I haven’t tried the PTR thing yet, but I do have a small business client that has AT&T business internet and they were able to get a static /56 (For some reason, AT&T refused to do a /48, but we did push them on it.)

If it turns out they won’t do PTR or more likely NS for our ip6.arpa zone, then we’ll probably start looking for an alternative provider or get an HE /48 over a tunnel which will do PTR or NS records appropriately.

Owen


> On Sep 18, 2021, at 14:19 , Stephen Satchell <list@satchell.net> wrote:
>
> I concur that the problem is not a routing hardware problem. It's a perception problem with the various ISPs. I have fiber service with AT&T.
>
> My little server farm endpoints all have IPv6 addresses, including the edge router. I also have a plan to allocate IPv6 addresses to my LAN devices, and to protect the LAN devices from outside interference by rules in the NFTABLES firewall that include connection tracking on outbound requests. (IPv4 will still use NAT to keep nefarious people from probing my internals.)
>
> Specifically, when I was doing my mail server refresh (moving from Red Hat to Canonical) I decided it was time to offer IPv6 connectivity in the mail server to "future proof" my setup. That included adding AAAA records in my DNS zone files. Failure! The issues:
>
> 1. I learned that there are no "static addresses" in IPv6, as far as AT&T was concerned. By all appearances, though, the IPv6 /64 is relatively static, for now, similar to the way that early cable modem deployments kept the same IPv4 addresses. (Until the cable people started forcing changes on DHCP lease renewal, that is.)
>
> 2. My request for PTR records was denied, which means I can't satisfy Best Practices for a mail server in the IPv6 space. No PTR records, no redirection of ip6.apra space, nothing. I include AT&T's refusal below.
>
> 3. I don't know how to get an IPv6 allocation from ARIN, how to request AT&T to route it, or how to deal with the DNS server issues. Oh, I know how to configure BIND9; I would prefer using a 24/7/365 provider. For example, my master zone files are with Register.com, so if my circuit goes down the name resolution still happens. Register.com appears not to provide reverse-DNS PTR zone support (in6.arpa). A Google search turned up NOTHING for in6.arpa hosting.
>
> That tells me that IPv6 is not "Internet Ready" for small users. Given the level of FU responses I get trying to work with it, I will stop banging my head against the wall.
>
> So I stick with IPv4, because that will be the "standard" until the day I die, as far as I can tell.
>
> (I removed the AAAA record, so as not to confuse mail server that DO operate IPv6.)
>
>> Subject: RE: Need IPv6 PTR record for my IPv6 mail server
>> Date: Mon, 19 Jul 2021 12:52:53 +0000
>> From: Prov-DNS <prov-dns@att.com>
>> To: Prov-DNS <prov-dns@att.com>, att@satchell.net <att@satchell.net>
>> Hello We don't process DNS request on IPv6 addresses. We only process DNS
>> request on IPv4 static assigned addresses. If you would like us to
>> process a DNS request for you on a IPv4 address please provide the
>> following information.
>> IPv4 address you would like the record created for Host name you would > like that IP address pointed to
> >
>> Thanks
>> Michael AT&T Prov-DNS
>> -----Original Message-----
>> From: Stephen Satchell <att@satchell.net>
>> Sent: Friday, July 16, 2021 5:42 PM
>> To: DNSUpdates cB <g12988@att.com>
>> Subject: Need IPv6 PTR record for my IPv6 mail server
>> Here is the record I need inserted into your ip6.arpa DNS zone:
> 2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. 0 IN PTR smtp.satchell.net.
>> This is the result from the question section of a dig(1) request for the PTR record for my IPv6 address 2600:1700:79b0:ddc0::32, and the fully-qualified domain name of the server.
>> You can verify the information using dig smtp.satchell.net AAAA and checking the reverse.
>> This is the only server in my collection that needs the IPv6 pointer.

On 9/18/21 8:58 PM, Owen DeLong wrote:
> I haven’t tried the PTR thing yet, but I do have a small business client that has AT&T business internet and they were able to get a static /56 (For some reason, AT&T refused to do a /48, but we did push them on it.)

When I checked, there were NO options for ANY static IPv6. Perhaps the
devil is in the details of my particular "business Internet" package.
If "package" is the right term; I use them only for upstream
connectivity and rental of IP (and IPv6) addresses.

> If it turns out they won’t do PTR or more likely NS for our ip6.arpa zone, then we’ll probably start looking for an alternative provider

That's the problem with a facilities-based ISP, there are no alternative
providers. Oh, sure, I could get Spectrum here. Indeed, I had a
circuit: when I had their business service I had even more problems with
them than I do with this one.

> or get an HE /48 over a tunnel which will do PTR or NS records appropriately.

Hurricane Electric? Seriously? I had them when I was working at a web
host company quite a while ago. Have they improved their service desk?
The downside is that I would have a serial pair of points of failure for
my connectivity.

IPv6 was supposed to SOLVE the problems, not create more problems.

I look back longingly to that product from the 80s: Internet-in-a-box.

I also remember the birth of Interop, when I visited Telebit at a
session to work out the interoperability snags in PPP implementations
among a handful of vendors.

John Levine wrote:

>> Unless their infrastructure runs significantly on hardware and
>> software pre-2004 (unlikely), so does the cost of adding IPv6 to
>> their content servers. Especially if they$B!G(Bre using a CDN such as
>> Akamai.
>
> I wasn't talking about switches and routers.

But, on routers, IPv6 costs four times more than IPv4 to
look up routing table with TCAM or Patricia tree.

It is not a problem yet, merely because full routing table of
IPv6 is a lot smaller than that of IPv4, which means most
small ISPs and multihomed sites do not support IPv6.


Mark Andrews wrote:

> There is nothing at the protocol level stopping AT&T offering a
> similar level of service.

Setting up reverse DNS lookup for 16B address is annoying,
which may stop AT&T offering it.

> Don$B!G(Bt equate poor implementation with the protocol being broken.

IPv6 is broken in several ways. One of the worst thing is its
address length.

Masataka Ohta

People who keep thinking this is a technical problem that can be
engineered away are confused. People who think the relative cost of
doing lookup for IPV4/IPV6 is visible to TCO are confused. Just
because you can observe technical differences doesn't mean they are
important, it may mean you're being affected by availability
heuristics bias, you think that the things you understand must contain
the solution to the problem.

IPv6 problem is, very few companies in developed markets need it to do
business, as customers are just bouncing between established players,
no new organic growth. Those companies choosing to do it increase
their cost for no utility, so it is an objectively bad decision for
many to do IPv6.
People who have sentimental attachment to versions of IP protocols are
a minority, most just want that customers continue paying their
invoices and keep accepting the product.

It is almost guaranteed we are married to IPv4 past our life cycles,
because there will be a lot of drivers to keep it. Even though
dual-stack increases cost for our vendors and us, each of us can
transfer the cost to our customers with margins, fixing it would mean
less revenue. Like infosec, it'll want things to remain relatively
broken, as everyone in position to change it are capitalising on
keeping it.



--
++ytti

On 9/18/21 11:20 PM, Masataka Ohta wrote:
> Mark Andrews wrote:
>
> > There is nothing at the protocol level stopping AT&T offering a
> > similar level of service.
>
> Setting up reverse DNS lookup for 16B address is annoying,
> which may stop AT&T offering it.

How many mail servers are on the Internet today? I don't know. How
many business customers (large and small) does AT&T have? I don't know,
and don't expect ever to know. I assume by "16B" you mean "16 billion";
if so, why did you select that value?

Based on the routing tables on my edge router for my existing
connection-based allocation, I have a IPv6 address block with a /60
prefix. The AT&T SBCIS allocation for IPv6 is 2600:1700::/28
(4.294.967.296 /60 prefixes), and the parent range is /12. (In a prior
message, someone mentioned that their customer was able to obtain a
static /56 IPv6 address block. Don't know how they did that, unless
they are tunneling to a different provider like HE.)

AT&T does offer PTR records for IPv4 static addresses -- I have a set of
static IPv4 addresses (which I pay for monthly) and one associated
IN-ADDR.ARPA PTR record. (I used to have *two* sets of static IP IPv4
addresses -- both paid for monthly -- and two associated IN-ADDR.ARPA
PTR records, but I released one of those sets when I discontinued my
long-time DSL service with AT&T.)

From the AT&T community forum, from two years ago, a moderator says
this: "IPv6 Its [sic] are assigned to your connection; IPv4 static IP
blocks are assigned to you. This is why they still don't offer reverse
PTR delegation."

What's missing? A static prefix of IPv6, and one IP6.ARPA PTR record.
I'm willing to pay for IPv6 static addresses, as long as I can get the
one IP6.ARPA PTR record for my mail server. Connection-based prefix
would be fine, but I still need the PTR record to satisfy the Best
Practices requirements for mail servers.

(Why do I run my own mail server? When I started out with DSL many
years ago, I used Pacific Bell's mail servers. The IP reputation was to
the point that mail from my systems was blocked by so many mail servers
around the 'Net. So, Postfix locally. Never looked back.)

Saku Ytti wrote:

> It is almost guaranteed we are married to IPv4 past our life cycles,
> because there will be a lot of drivers to keep it.

So, the war was between "IPv4 with NAT" and "IPv4 dual
stacked with IPv6". If IPv6 were simple, quickly standardized
and easily deployable, which are technical points, it could
have won.

Masataka Ohta

> On Sep 18, 2021, at 23:20 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> John Levine wrote:
>
>>> Unless their infrastructure runs significantly on hardware and
>>> software pre-2004 (unlikely), so does the cost of adding IPv6 to
>>> their content servers. Especially if they’re using a CDN such as
>>> Akamai.
>> I wasn't talking about switches and routers.
>
> But, on routers, IPv6 costs four times more than IPv4 to
> look up routing table with TCAM or Patricia tree.
>
> It is not a problem yet, merely because full routing table of
> IPv6 is a lot smaller than that of IPv4, which means most
> small ISPs and multihomed sites do not support IPv6.

Well, it’s a combination. Even with full v6 adoption, the routing table in v6 should be substantially smaller.

Compare AS6939 v4 vs. v6:

+ 6939 is probably the most v6 fully implemented ASN on the planet.
+ 6939 announces 4 of their own prefixes in v6 (They do originate 19 prefixes on behalf of customers)
+ 6939 announces at least 41 of their own prefixes in v4 and originates myriad customer prefixes in v4.

That’s more than 10x the prefixes in v4 for the same network fully dual-stacked vs. what is announced in v6.

v4 is so thoroughly fragmented and v6 is a lot less likely to become so.

>
>
> Mark Andrews wrote:
>
> > There is nothing at the protocol level stopping AT&T offering a
> > similar level of service.
>
> Setting up reverse DNS lookup for 16B address is annoying,
> which may stop AT&T offering it.

Why would one need to do that? For customers with a static prefix that want reverse DNS capabilities,
offer to point the NS records for their prefix wherever they want and let them run their own DNS servers.

>
> > Don’t equate poor implementation with the protocol being broken.
>
> IPv6 is broken in several ways. One of the worst thing is its
> address length.

Why do you think 128 bits isn’t enough?

Owen

> On Sep 10, 2021, at 00:21 , Bjørn Mork <bjorn@mork.no> wrote:
>
> Owen DeLong via NANOG <nanog@nanog.org> writes:
>
>> The addresses aren’t the major cost of providing IPv4 services.
>>
>> CGN boxes, support calls, increasing size of routing table = buying new routers, etc.
>
> You're counting dual-stack costs as if IPv4 was the optional protocol.
> That's a fantasy world. Time to get out of la-la land now.

No, I’m counting them as if they are the increasing cost of continuing to support IPv4.

> Your edge routers can do CGN for all connected users just fine. Yes,
> there is still a cost both in resources and management, but you'll have
> to weigh that against the cost of doing dual-stack on the same box. I'm
> not convinced dual-stack wins.

It does. At least in my environments.

> Don't know what you're thinking of wrt support calls, but dual-stack has
> some failure modes which are difficult to understand for both end users
> and support. NAT is pretty well understood in comparison.

Single layer NAT, sure. But double-layer NAT has some oddities that you
might not have encountered yet…

1. Products which are built on really strange assumptions about everyone
having the same NAT environment.

For example, Philips Hue makes an assumption that if you are in the
same household, your Hue Gateway and your phones and laptops will
all have the same public IP address.

This has two unexpected ramifications:

1. You cannot actually complete their registration process for their
cloud services if you don’t NAT everything to the same address
or proxy it all through a common proxy address.

2. If you are behind CGN, you and your neighbors are going to be
considered a single household (at least everyone behind the
same CGN address). Of course, this assumes that you get a
consistent single public CGN address for everything in your
house. If you don’t, then you get a combination of this problem
with problem 1 above and life gets very interesting.

2. NAT Traversal technologies that don’t cope well with an added layer.

3. Added and inconsistent latency through CGN boxes degrading
several online experiences, including voice, interactive video,
and most of all several types of gaming.

There are many more and each of them generates additional support calls
to the ISP about “The internet is broken”.

> Your routing tables won't grow with IPv4 or CGN. They grow when you add
> IPv6.

Um, please review the IPv4 routing table report over the past few years
and tell me that again.

For your convenience: https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step


>
>> Increased cost of developers having to work around NAT and NAT
>> becoming ever more complex with multiple layers, etc.
>
> And this can be avoided by reconfiguring the local network somehow? Or
> are we talking about an Internet without IPv4? This is even more
> fantastic than the idea that IPv4 is optional in the local network.

We are talking about internet where IPv4 prevalence continues to drop. Whether
it can be avoided or not, however, it is a factor in the ever increasing cost of IPv4.

>
>> All of these are the things driving the ever increasing cost of IPv4
>> services, not just the cost of the addresses.
>
> Yes, the cost of addresses is not prohibitive, and there is no
> indication it will be.

Agreed… But the other costs are also continuing to increase. None of these
costs exist in IPv6 save a one-time deployment cost.

> The consolidation of hosting services have reduced the need for globally
> routable addresses. You don't host your own mail server and web server
> anymore, even if you're a large organisation.

Lots do, actually.

> Most ISPs haven't yet
> taken advantage of this. They are still giving globally routable IPv4
> addresses to customers which have no need for that. These addresses can
> be re-allocated for CGN if there is a need. This is obviously still not
> free, but it does limit the price of fresh IPv4 addresses.

Lots of things you don’t expect break when you stop giving at least one IPv4 GUA
to your customers.

> The other costs you list will not affect an IPv4 only shop at all.

This simply isn’t true.

Owen

Owen,


On Sat, Sep 18, 2021 at 23:51 Owen DeLong <owen@delong.com> wrote:

> On Sep 18, 2021, at 12:34 , Victor Kuarsingh <victor@jvknet.com> wrote:
>
> On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl@iecc.com> wrote:
>
>> It appears that Owen DeLong via NANOG <owen@delong.com> said:
>>
>>
> Glad you noted this. Thinking this was/is purely a hardware cycle problem
> related to normal/forced upgrade strategies. On that point, most hardware
> I know of from 2004 in larger networks is long fully depreciated and
> sweating assets 15+ years can happen, but I don't personally think this is
> the biggest issue.
>
>
> It’s certainly not purely hardware, but it also doesn’t require solving
> the entire problem across the entire backend.
>
> What is urgently needed is to fix the customer-facing front end so that it
> speaks both protocols (or at least speaks IPv6).
>

This is a great question. So when we approached this (going back a while
now) this is what I thought too. But I was responsible to solve this end
to end. So just updating front end (CPEs, network routers, DHCP,
provisioning, IP address planning, security, peering/transit, staff
training, test harnesses/plans, etc) was not sufficient to turn on IPv6.

The high cost and time challenge issues were not upgrading backend servers
to IPv6, but backend provisioning systems, tools, customer support tools,
their training, and related items. These systems were far more complex and
costly to address (especially when considering opportunity cost). Without
all of this in place, we could not actually deploy IPv6 for production use
(it’s not just Turing it on, its our ability to operate it down to the
person answer the phone, the technician that installs and/or fixes/replaces
items in home).

Also at that time vendor CPE hardware was not as far along on IPv6
readiness (approx mid-decade 2000s). Getting reliable code at that time
was hard with a lot of brokenness (which also could not go into production
until it was reliable). Perhaps this a non-issue today, but it certainly
was not a something that was “ready to go” even 10-15 years. This (IPv6
readiness in CPE code) was also competing with other priorities often
blowing up timelines which meant it had to wait as not releasing code into
production on a strict timeline was not an option for business reasons.

All said and done, we got it completed, but it far most costly than we
first thought, and IPv4 costs did not go away after we were completed.
Sure it’s possible to reduce those costs with an aggressive program, but it
is not as simple as many think.


> As you noted John, its the plethora of software, support systems, tooling,
> and most important in many environments - legacy customer management and
> provisioning systems that can be the limiting factor. I recall looking
> back when leading IPv6 turn-up, those were the biger problems to solve
> for. Often such systems are extremely expensive to touch and working on
> them required prioritization against direct revenue generating projects
> (opportunity cost) . Replacing routers was just a money problem.
>
>
> Why do those systems have to be updated in order to deliver the service to
> the customer in both protocols?
>

Addressed in above comment.

When it comes to turning on IPv6 and reducing your dependency on IPv4, your
mileage may vary (in terms of real costs, complexities and deployment).

Regards,

Victor K


> I mean sure, you’ll probably need to fix some logging databases that think
> that a customers address can be logged as a uint32_t, but that’s a
> relatively small number of systems that need to be updated in that case and
> it’s not like expanding the field to uint128_t in the database is
> incompatible with the old software during the upgrade process.
>
> I am by far not saying I agree with choices made by hold-outs, but I also
> understand this is for many, not just an engineering problem to solve.
>
>
> I agree there are some systems that may make this more complex in some
> environments, but I don’t agree that it’s
> impossible (or even particularly difficult) for a content provider to
> deliver their content on both protocols today even
> if they don’t upgrade their back-end systems.
>
> Owen
>
>

It appears that Stephen Satchell <list@satchell.net> said:
>> or get an HE /48 over a tunnel which will do PTR or NS records appropriately.
>
>Hurricane Electric? Seriously?

I've been using HE's free ipv6 tunnels for ten years. They work great.
I don't ever recall any downtime. They assign you a /64 by default,
/48 on request, and delegate the rDNS wherever you want. One points at my server which
is in a rack somewhere, one points at the router on my home fiber connection.

Since I set it up they filter port 25 by default for obvious reasons but will unblock
if you ask nicely and sound like you know what you're doing. Geolocation doesn't work,
and now and then someone (Wikipedia) decides it's an evil VPN and blocks or filters it
but I haven't found that to be much of a problem in practice.

R's,
John

Owen DeLong wrote:

>> But, on routers, IPv6 costs four times more than IPv4 to look up
>> routing table with TCAM or Patricia tree.
>>
>> It is not a problem yet, merely because full routing table of IPv6
>> is a lot smaller than that of IPv4, which means most small ISPs and
>> multihomed sites do not support IPv6.
>
> Well, it$B!G(Bs a combination. Even with full v6 adoption, the routing
> table in v6 should be substantially smaller.

Not at all.

> Compare AS6939 v4 vs. v6:

That is not a meaningful comparison.

As mergers of ASes increases the number of announcements
and IPv4 addresses were allocated a lot earlier than those
of IPv6, comparing the current numbers of announcements is
not meaningful.

As a result, size of global routing table will keep
increasing unless there are other factors to limit it.

An important factor is that, for IPv4 with globally
routable /24, the absolute upper limit is merely 16M,
to be looked up by a single memory access of conventional
SRAM without needing TCAM. OTOH, IPv6 is hopeless.

Another favorite factor for IPv4 is that, though most of
routing table entries are generated from small entities
having a /24 assuming NAT, if such entities are merged,
renumbering is not so much a pain and they are motivated
to rely on a single /24 and sell others. OTOH, there is
no such motivation for IPv6.

> v4 is so thoroughly fragmented and v6 is a lot less likely to become
> so.

It is true that fragmentation is a problem. However, it merely
means that IPv6 address space will also be fragmented and that
IPv4 can but IPv6 can't be deployed at full scale,

Masataka Ohta