Mailing List Archive

Hi,

> > v4 is so thoroughly fragmented and v6 is a lot less likely to become
> > so.
>
> It is true that fragmentation is a problem. However, it merely means that IPv6
> address space will also be fragmented and that
> IPv4 can but IPv6 can't be deployed at full scale,

Just this week We had our first customer asking if we can setup BGP to route the /48 they received from the headquarters in the states.
They are asking us to provide a few v4 addresses , but to use their own v6 block.
Yes they are a large conglomerate with their own AS and a large v6 allocation, so not a common customer, but they have hundreds of offices everywhere in the world where they are doing this...
I can just see the Cxx presenting their solution at some event and it becoming the new thing to do
What you are still using your providers addresses? You must be crazy... We assign our own it's much better...
A couple hundred corporations like this and the v6 table would surpass v4...

Brian

> On Sep 19, 2021, at 16:17 , Victor Kuarsingh <victor@jvknet.com> wrote:
>
> Owen,
>
>
> On Sat, Sep 18, 2021 at 23:51 Owen DeLong <owen@delong.com <mailto:owen@delong.com>> wrote:
>> On Sep 18, 2021, at 12:34 , Victor Kuarsingh <victor@jvknet.com <mailto:victor@jvknet.com>> wrote:
>>
>> On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl@iecc.com <mailto:johnl@iecc.com>> wrote:
>> It appears that Owen DeLong via NANOG <owen@delong.com <mailto:owen@delong.com>> said:
>>
>>
>> Glad you noted this. Thinking this was/is purely a hardware cycle problem related to normal/forced upgrade strategies. On that point, most hardware I know of from 2004 in larger networks is long fully depreciated and sweating assets 15+ years can happen, but I don't personally think this is the biggest issue.
>
> It’s certainly not purely hardware, but it also doesn’t require solving the entire problem across the entire backend.
>
> What is urgently needed is to fix the customer-facing front end so that it speaks both protocols (or at least speaks IPv6).
>
> This is a great question. So when we approached this (going back a while now) this is what I thought too. But I was responsible to solve this end to end. So just updating front end (CPEs, network routers, DHCP, provisioning, IP address planning, security, peering/transit, staff training, test harnesses/plans, etc) was not sufficient to turn on IPv6.
>
> The high cost and time challenge issues were not upgrading backend servers to IPv6, but backend provisioning systems, tools, customer support tools, their training, and related items. These systems were far more complex and costly to address (especially when considering opportunity cost). Without all of this in place, we could not actually deploy IPv6 for production use (it’s not just Turing it on, its our ability to operate it down to the person answer the phone, the technician that installs and/or fixes/replaces items in home).

This sounds like an eyeball environment… Note that my question was in the context of the laggard content providers.

Eyeball providers are going to have to do something eventually whether they like it or not and I’m a whole lot less worried about a forcing function for them.

The major eyeball providers have basically completed this. Hopefully that means that the vendors you’re using have been forced to upgrade their code and such to accommodate by now.

> Also at that time vendor CPE hardware was not as far along on IPv6 readiness (approx mid-decade 2000s). Getting reliable code at that time was hard with a lot of brokenness (which also could not go into production until it was reliable). Perhaps this a non-issue today, but it certainly was not a something that was “ready to go” even 10-15 years. This (IPv6 readiness in CPE code) was also competing with other priorities often blowing up timelines which meant it had to wait as not releasing code into production on a strict timeline was not an option for business reasons.

Sure, but a lot of that has gotten better in the recent years, largely driven by some of the major eyeball ISPs.

> All said and done, we got it completed, but it far most costly than we first thought, and IPv4 costs did not go away after we were completed. Sure it’s possible to reduce those costs with an aggressive program, but it is not as simple as many think.

Agreed… IPv4 costs for you, as an eyeball provider, can’t really go away until you can start doing one or more of the following:
1. Raising your overall rates and providing a discount to IPv6-only customers
2. Adding a surcharge to your billing for customers still requiring IPv4 services
3. Turning off or reducing availability of IPv4 native services and moving more towards IPv4AAS
4. Turning off IPv4 services outright

All of those basically depend on moving a few key laggard content providers forward.

>> As you noted John, its the plethora of software, support systems, tooling, and most important in many environments - legacy customer management and provisioning systems that can be the limiting factor. I recall looking back when leading IPv6 turn-up, those were the biger problems to solve for. Often such systems are extremely expensive to touch and working on them required prioritization against direct revenue generating projects (opportunity cost) . Replacing routers was just a money problem.
>
> Why do those systems have to be updated in order to deliver the service to the customer in both protocols?
>
> Addressed in above comment.

Eyeball… Now think about it in the context of a content provider… Especially one that is behind a CDN where it’s literally simply flipping the switch on the CDN
that says front end our stuff with both protocols.

> When it comes to turning on IPv6 and reducing your dependency on IPv4, your mileage may vary (in terms of real costs, complexities and deployment).

Agreed… Much more variable for eyeballs, but still some variability for content providers, too. Nonetheless, I think it is generally quite simple for content providers today while there are still some issues that remain unsolved on the eyeball side of things.

The good news is that once a few laggard content providers add IPv6 capabilities, eyeball providers can start treating IPv4 as (mostly) optional in a lot of areas, so that those which have deployed IPv6 to their customers in a meaningful way can start to shrink their IPv4 costs.

Owen

> On Sep 20, 2021, at 05:15 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Owen DeLong wrote:
>
>>> But, on routers, IPv6 costs four times more than IPv4 to look up
>>> routing table with TCAM or Patricia tree.
>>> It is not a problem yet, merely because full routing table of IPv6
>>> is a lot smaller than that of IPv4, which means most small ISPs and
>>> multihomed sites do not support IPv6.
>> Well, it’s a combination. Even with full v6 adoption, the routing
>> table in v6 should be substantially smaller.
>
> Not at all.
>
>> Compare AS6939 v4 vs. v6:
>
> That is not a meaningful comparison.
>
> As mergers of ASes increases the number of announcements
> and IPv4 addresses were allocated a lot earlier than those
> of IPv6, comparing the current numbers of announcements is
> not meaningful.

Mergers of ASes does not increase announcements in IPv4 nearly as much as slow-start
and repeated expanding requests for additional IPv4 space have.

> As a result, size of global routing table will keep
> increasing unless there are other factors to limit it.

Sure, but it’s very clear that the rate of increase for IPv6 appears to be roughly 1/8th that of
IPv4, so even if IPv6 was 4x as expensive, the growth rate in cost would still be 1/2 that of
IPv4.

> An important factor is that, for IPv4 with globally
> routable /24, the absolute upper limit is merely 16M,
> to be looked up by a single memory access of conventional
> SRAM without needing TCAM. OTOH, IPv6 is hopeless.

The reality is that IPv4 will never be completely disaggregated into /24s and IPv6 will never
be completely disaggregated into /48s, so this is actually meaningless and not predictive
in any way.

> Another favorite factor for IPv4 is that, though most of
> routing table entries are generated from small entities
> having a /24 assuming NAT, if such entities are merged,
> renumbering is not so much a pain and they are motivated
> to rely on a single /24 and sell others. OTOH, there is
> no such motivation for IPv6.

There is no need for such motivation in IPv6 and better yet, since the two organizations
have fully globally unique addresses deployed throughout their network, there’s no risk
of collisions in RFC-1918 space necessitating large renumbering projects to merge the
networks. Indeed, all you need to do is turn on forwarding between the two networks
and you’re basically done.

As such, no, that’s an advantage that clearly goes to IPv6, not IPv4.

>
>> v4 is so thoroughly fragmented and v6 is a lot less likely to become
>> so.
>
> It is true that fragmentation is a problem. However, it merely
> means that IPv6 address space will also be fragmented and that
> IPv4 can but IPv6 can't be deployed at full scale,

Why would IPv6 become fragmented? When a provider the size of HE can
easily get a /24 from ARIN, what need is there for fragmentation. Sure, they
started with a /32 and they’re keeping that, but the equivalent growth in IPv4
would have resulted in a /20 + /19 + /18 + /17 + /16 + /15 + /14 + /13 and likely
an additional /12 eventually. So you get 7-8 prefixes for the same factor of growth
as 2 prefixes in IPv4.

It simply doesn’t make sense to claim that fragmentation in v6 will be nearly as bad
as it is in v4 because we don’t have the problems of slow start and we’re no longer
trying to balance scarcity against aggregation.

Owen

> On Sep 20, 2021, at 06:48 , Brian Turnbow via NANOG <nanog@nanog.org> wrote:
>
> Hi,
>
>>> v4 is so thoroughly fragmented and v6 is a lot less likely to become
>>> so.
>>
>> It is true that fragmentation is a problem. However, it merely means that IPv6
>> address space will also be fragmented and that
>> IPv4 can but IPv6 can't be deployed at full scale,
>
> Just this week We had our first customer asking if we can setup BGP to route the /48 they received from the headquarters in the states.
> They are asking us to provide a few v4 addresses , but to use their own v6 block.
> Yes they are a large conglomerate with their own AS and a large v6 allocation, so not a common customer, but they have hundreds of offices everywhere in the world where they are doing this...
> I can just see the Cxx presenting their solution at some event and it becoming the new thing to do
> What you are still using your providers addresses? You must be crazy... We assign our own it's much better...
> A couple hundred corporations like this and the v6 table would surpass v4...

In such a case, I’m not convinced that’s a bad thing.

Owen

Owen DeLong wrote:

>> As mergers of ASes increases the number of announcements and IPv4
>> addresses were allocated a lot earlier than those of IPv6,
>> comparing the current numbers of announcements is not meaningful.
>
> Mergers of ASes does not increase announcements in IPv4 nearly as
> much as slow-start and repeated expanding requests for additional
> IPv4 space have.

That *was* a factor, when increased number of subscribers
meant more free addresses.

Today, as /24 can afford hundreds of thousands of subscribers
by NAT, only very large retail ISPs need more than one
announcement for IPv4.

>> As a result, size of global routing table will keep increasing
>> unless there are other factors to limit it.
>
> Sure, but it$B!G(Bs very clear that the rate of increase for IPv6 appears
> to be roughly 1/8th that of IPv4,

It merely means IPv6 is not deployed at all by small ISPs
and multihomed sites.

> The reality is that IPv4 will never be completely disaggregated into
> /24s

You are so optimistic.

> and IPv6 will never be completely disaggregated into /48s, so
> this is actually meaningless and not predictive in any way.

That IPv6 will be disaggregated into /40 or even /32 is disastrous.

> There is no need for such motivation in IPv6 and better yet,

Then, in a long run, IPv6 will be disaggregated into /32 or /40.

> since
> the two organizations have fully globally unique addresses deployed
> throughout their network, there's no risk of collisions in RFC-1918
> space necessitating large renumbering projects to merge the networks.

You fully misunderstand why NAT is so popular today defeating IPv6.

Even if two organizations are merged, sites of the organizations
are, in general, not merged.

As private address space behind NAT is used by each site
independently, there is no renumbering occur for the private
addresses.

Masataka Ohta

On 2021-09-19 09:20, Masataka Ohta wrote:
> John Levine wrote:
>
>>> Unless their infrastructure runs significantly on hardware and
>>> software pre-2004 (unlikely), so does the cost of adding IPv6 to
>>> their content servers. Especially if they’re using a CDN such as
>>> Akamai.
>>
>> I wasn't talking about switches and routers.
>
> But, on routers, IPv6 costs four times more than IPv4 to
> look up routing table with TCAM or Patricia tree.
>
> It is not a problem yet, merely because full routing table of
> IPv6 is a lot smaller than that of IPv4, which means most
> small ISPs and multihomed sites do not support IPv6.
>
>
> Mark Andrews wrote:
>
>> There is nothing at the protocol level stopping AT&T offering a
>> similar level of service.
>
> Setting up reverse DNS lookup for 16B address is annoying,
> which may stop AT&T offering it.
>
>> Don’t equate poor implementation with the protocol being broken.
>
> IPv6 is broken in several ways. One of the worst thing is its
> address length.
>
> Masataka Ohta
+1
Different scope problem: on inexpensive software BRAS solutions
(PPPoE/IPoE). Enabling ipv6 just jacked up neighbour table usage and
lookups cost in benchmark profiling, because now it have to keep for all
users IPv6 /64 + MAC entries.
Another drop is neighbor discovery on device with 10k IPOE termination
vlans and privacy extensions.
Also, i wonder how this changed?
https://blog.bimajority.org/2014/09/05/the-network-nightmare-that-ate-my-week/
Another problem is privacy extension and IoT, they are not supported in
lwip stack shipped with most of IoT SoC. As far as i see in git it is
not added yet too.
And SLAAC vs DHCPv6, again, first lacking some critical features, and
second is often not implemented properly.

As many say - this is tiny, a drops of mess and complexities, but the
ocean is made up of tiny drops. All these little things lead to the fact
that very few want to mess with v6.

On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <
mohta@necom830.hpcl.titech.ac.jp> wrote:

> Today, as /24 can afford hundreds of thousands of subscribers
> by NAT, only very large retail ISPs need more than one
> announcement for IPv4.
>

You can only have about 200-300 subscribers per IPv4 address on a CGN. If
you try to go further than that, for example by using symmetric NAT, you
will increase the number of customers that want to get a public IPv4 of
their own. That will actually decrease the combined efficiency and cause
you to need more, not less, IPv4 addresses.

Without checking our numbers, I believe we have at least 10% of the
customers that are paying for a public IPv4 to escape our CGN. This means a
/24 will only be enough for about 2500 customers maximum. The "nat
escapers" drown out the efficiency of the NAT pool.

The optimization you need to do is to make the CGN as customer friendly as
possible instead of trying to squeeze the maximum customers per CGN IPv4
address.

Perhaps IPv6 can lower the number of people that need to escape IPv4 nat.
If it helps just a little bit, that alone will make implementing IPv6 worth
it for smaller emerging operators. Buying IPv4 has become very expensive.
Yes you can profit from selling a public IPv4 address to the customer, but
there is also the risk that the customer just goes to the incumbent, which
has old large pools of IPv4 and provides it for free.

Regards,

Baldur

Where does this "You can only have about 200-300 subscribers per IPv4
address on a CGN." limit come from? I have seen several apartment
complexes run on a single static IPv4 address using a Mikrotik with
NAT.

On Wed, Sep 22, 2021 at 2:49 PM Baldur Norddahl
<baldur.norddahl@gmail.com> wrote:
>
>
>
> On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>>
>> Today, as /24 can afford hundreds of thousands of subscribers
>> by NAT, only very large retail ISPs need more than one
>> announcement for IPv4.
>
>
> You can only have about 200-300 subscribers per IPv4 address on a CGN. If you try to go further than that, for example by using symmetric NAT, you will increase the number of customers that want to get a public IPv4 of their own. That will actually decrease the combined efficiency and cause you to need more, not less, IPv4 addresses.
>
> Without checking our numbers, I believe we have at least 10% of the customers that are paying for a public IPv4 to escape our CGN. This means a /24 will only be enough for about 2500 customers maximum. The "nat escapers" drown out the efficiency of the NAT pool.
>
> The optimization you need to do is to make the CGN as customer friendly as possible instead of trying to squeeze the maximum customers per CGN IPv4 address.
>
> Perhaps IPv6 can lower the number of people that need to escape IPv4 nat. If it helps just a little bit, that alone will make implementing IPv6 worth it for smaller emerging operators. Buying IPv4 has become very expensive. Yes you can profit from selling a public IPv4 address to the customer, but there is also the risk that the customer just goes to the incumbent, which has old large pools of IPv4 and provides it for free.
>
> Regards,
>
> Baldur
>

And how many apartments where covered by that single IP address? Was this
where there is a restriction on other providers so the occupants had no
choice of wireline ISP?

> On 23 Sep 2021, at 09:38, Colton Conor <colton.conor@gmail.com> wrote:
>
> Where does this "You can only have about 200-300 subscribers per IPv4
> address on a CGN." limit come from? I have seen several apartment
> complexes run on a single static IPv4 address using a Mikrotik with
> NAT.
>
> On Wed, Sep 22, 2021 at 2:49 PM Baldur Norddahl
> <baldur.norddahl@gmail.com> wrote:
>>
>>
>>
>> On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>>>
>>> Today, as /24 can afford hundreds of thousands of subscribers
>>> by NAT, only very large retail ISPs need more than one
>>> announcement for IPv4.
>>
>>
>> You can only have about 200-300 subscribers per IPv4 address on a CGN. If you try to go further than that, for example by using symmetric NAT, you will increase the number of customers that want to get a public IPv4 of their own. That will actually decrease the combined efficiency and cause you to need more, not less, IPv4 addresses.
>>
>> Without checking our numbers, I believe we have at least 10% of the customers that are paying for a public IPv4 to escape our CGN. This means a /24 will only be enough for about 2500 customers maximum. The "nat escapers" drown out the efficiency of the NAT pool.
>>
>> The optimization you need to do is to make the CGN as customer friendly as possible instead of trying to squeeze the maximum customers per CGN IPv4 address.
>>
>> Perhaps IPv6 can lower the number of people that need to escape IPv4 nat. If it helps just a little bit, that alone will make implementing IPv6 worth it for smaller emerging operators. Buying IPv4 has become very expensive. Yes you can profit from selling a public IPv4 address to the customer, but there is also the risk that the customer just goes to the incumbent, which has old large pools of IPv4 and provides it for free.
>>
>> Regards,
>>
>> Baldur
>>

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

> On Sep 22, 2021, at 07:47 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>
> Owen DeLong wrote:
>
>>> As mergers of ASes increases the number of announcements and IPv4
>>> addresses were allocated a lot earlier than those of IPv6,
>>> comparing the current numbers of announcements is not meaningful.
>> Mergers of ASes does not increase announcements in IPv4 nearly as
>> much as slow-start and repeated expanding requests for additional
>> IPv4 space have.
>
> That *was* a factor, when increased number of subscribers
> meant more free addresses.

It’s still a factor as many providers are purchasing addresses rather than deploy CGN
because they don’t want the expensive phone calls CGN causes.

> Today, as /24 can afford hundreds of thousands of subscribers
> by NAT, only very large retail ISPs need more than one
> announcement for IPv4.

I fail to grasp this desire to move the majority of users from second class citizens of the
network to third class all in the interests of forestalling the inevitable.

>
>>> As a result, size of global routing table will keep increasing
>>> unless there are other factors to limit it.
>> Sure, but it’s very clear that the rate of increase for IPv6 appears
>> to be roughly 1/8th that of IPv4,
>
> It merely means IPv6 is not deployed at all by small ISPs
> and multihomed sites.

Not true. Judging by the number of /48s in the table, IPv6 is relatively
widely deployed by multi homed end sites and judging by the number
of /32 to /40 prefixes, also widely deployed by small-iso ISPs.

>
> > The reality is that IPv4 will never be completely disaggregated into
> > /24s
>
> You are so optimistic.

Yes, I’ll be surprised if (e.g. Apple, HP) part out their /8s in to /24s.

I’ll be surprised if a bunch of large organizations fully part out
their /16s and such.

I doubt any major eyeball ISPs will be significantly disbursing or
disaggregating any of their large blocks any time soon.

I suppose you can call that optimism. I call it realism.

Frankly, the faster IPv4 fully fragments, the better because that only
serves to make continuing to carry it all the more expensive, further
making the case for IPv6.

>
> > and IPv6 will never be completely disaggregated into /48s, so
> > this is actually meaningless and not predictive in any way.
>
> That IPv6 will be disaggregated into /40 or even /32 is disastrous.

Which it won’t.

It’s unlikely we will fully deploy 2000::/3 in the lifetime of anyone
on this list today.

>> There is no need for such motivation in IPv6 and better yet,
>
> Then, in a long run, IPv6 will be disaggregated into /32 or /40.

Not likely… Too many providers and large organizations getting
/20s and /24s for that to happen.

>> since
>> the two organizations have fully globally unique addresses deployed
>> throughout their network, there's no risk of collisions in RFC-1918
>> space necessitating large renumbering projects to merge the networks.
>
> You fully misunderstand why NAT is so popular today defeating IPv6.

Maybe… I certainly don’t understand why it is popular. It’s simply awful.

> Even if two organizations are merged, sites of the organizations
> are, in general, not merged.

Seems rather pointless and counterproductive.

> As private address space behind NAT is used by each site
> independently, there is no renumbering occur for the private
> addresses.

Well, as GUA would be globally unique to each site, there would be
a full ability to merge the sites _AND_ no renumbering cost.

Can you explain any way in which NAT is somehow better?

Owen

tor. 23. sep. 2021 01.39 skrev Colton Conor <colton.conor@gmail.com>:

> Where does this "You can only have about 200-300 subscribers per IPv4
> address on a CGN." limit come from? I have seen several apartment
> complexes run on a single static IPv4 address using a Mikrotik with
> NAT.
>

It is our observation as the limit before you regularly run out of ports
using Linux as a CGN server.

It will still work if you have more users on an IP. The users will just
experience session failures at peak. Low levels of that might show up as
pictures that fail to load on a web page and be ok when the user reloads.
This will increase the number of support calls and the number of customers
that asks to escape the CGN. Or people will live with it and just think
that the Internet connection is low quality.

Regards

Baldur

Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> writes:

> That IPv6 will be disaggregated into /40 or even /32 is disastrous.

It won't.

No ISPs will deaggregate anything.

Some multi-site enterprises might assign a /48 per remote site from
their single prefix, and want those /48s routed via some transit peers.
But this does not imply that their prefix is split into the maximum
number of /48s. The number of routes is limited to the number of
separate network sites. There is no need to worry about this number
ever exceeding the number of IPv4 prefixes.

And those /48s will most likely be filtered out of the global table
forever. Enterprises wanting such a configuration will depend on
special transit agreements to carry and aggregate them for the rest of
the world. This is not a problem.

There are real issues with dual-stack, as this thread started out with.
I don't think there is a need neither to invent IPv6 problems, nor to
promote IPv6 advantages. What we need is a way out of dual-stack-hell.





Bjørn

300 apartments Mark. No, it's bulk internet and wifi so a single provider.

On Wed, Sep 22, 2021 at 8:01 PM Mark Andrews <marka@isc.org> wrote:
>
> And how many apartments where covered by that single IP address? Was this
> where there is a restriction on other providers so the occupants had no
> choice of wireline ISP?
>
> > On 23 Sep 2021, at 09:38, Colton Conor <colton.conor@gmail.com> wrote:
> >
> > Where does this "You can only have about 200-300 subscribers per IPv4
> > address on a CGN." limit come from? I have seen several apartment
> > complexes run on a single static IPv4 address using a Mikrotik with
> > NAT.
> >
> > On Wed, Sep 22, 2021 at 2:49 PM Baldur Norddahl
> > <baldur.norddahl@gmail.com> wrote:
> >>
> >>
> >>
> >> On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
> >>>
> >>> Today, as /24 can afford hundreds of thousands of subscribers
> >>> by NAT, only very large retail ISPs need more than one
> >>> announcement for IPv4.
> >>
> >>
> >> You can only have about 200-300 subscribers per IPv4 address on a CGN. If you try to go further than that, for example by using symmetric NAT, you will increase the number of customers that want to get a public IPv4 of their own. That will actually decrease the combined efficiency and cause you to need more, not less, IPv4 addresses.
> >>
> >> Without checking our numbers, I believe we have at least 10% of the customers that are paying for a public IPv4 to escape our CGN. This means a /24 will only be enough for about 2500 customers maximum. The "nat escapers" drown out the efficiency of the NAT pool.
> >>
> >> The optimization you need to do is to make the CGN as customer friendly as possible instead of trying to squeeze the maximum customers per CGN IPv4 address.
> >>
> >> Perhaps IPv6 can lower the number of people that need to escape IPv4 nat. If it helps just a little bit, that alone will make implementing IPv6 worth it for smaller emerging operators. Buying IPv4 has become very expensive. Yes you can profit from selling a public IPv4 address to the customer, but there is also the risk that the customer just goes to the incumbent, which has old large pools of IPv4 and provides it for free.
> >>
> >> Regards,
> >>
> >> Baldur
> >>
>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
>

> There are real issues with dual-stack, as this thread started out with.
> I don't think there is a need neither to invent IPv6 problems, nor to
> promote IPv6 advantages. What we need is a way out of dual-stack-hell.

I don’t disagree, but a reversion to IPv4-only certainly won’t do it.

I think the only way out is through. Unfortunately, the IPv6 resistant forces
are making that hard for everyone else.

Owen

On Thu, Sep 23, 2021 at 3:42 AM Baldur Norddahl <baldur.norddahl@gmail.com>
wrote:

>
>
> tor. 23. sep. 2021 01.39 skrev Colton Conor <colton.conor@gmail.com>:
>
>> Where does this "You can only have about 200-300 subscribers per IPv4
>> address on a CGN." limit come from? I have seen several apartment
>> complexes run on a single static IPv4 address using a Mikrotik with
>> NAT.
>>
>
> It is our observation as the limit before you regularly run out of ports
> using Linux as a CGN server.
>
> It will still work if you have more users on an IP. The users will just
> experience session failures at peak. Low levels of that might show up as
> pictures that fail to load on a web page and be ok when the user reloads.
> This will increase the number of support calls and the number of customers
> that asks to escape the CGN. Or people will live with it and just think
> that the Internet connection is low quality.
>
>
This sounds like very naive nat state management behavior.
Ideally, you'd be able to maintain state of:
original-src/dst/ports/proto -> in-interface/external ip/port/proto

unless some internal/original src is double using port/proto ... you should
really
have the ability to nat quite a large number of things to a single ipv4
address.

Of course as layers of nat get deeper you may lose some useful state :(
you may be able to use tcp seq numbers or other items in the state though
to overcome.

Side question on this thread…

Is it everyones current expectation that if a provider were to switch to IPv6 and drop IPv4 that the customers would all be just fine with that? I believe that there are several applications used by some of the the loudest customers (typically gamers and network gurus), not to mention some business applications that would break or be sub-optimal at best. I see CGN as the band aid to this issue, not the cure to the problem.

Discuss…?

- Brian

> On Sep 23, 2021, at 10:46 AM, Owen DeLong via NANOG <nanog@nanog.org> wrote:
>
>> There are real issues with dual-stack, as this thread started out with.
>> I don't think there is a need neither to invent IPv6 problems, nor to
>> promote IPv6 advantages. What we need is a way out of dual-stack-hell.
>
> I don’t disagree, but a reversion to IPv4-only certainly won’t do it.
>
> I think the only way out is through. Unfortunately, the IPv6 resistant forces
> are making that hard for everyone else.
>
> Owen
>

On Thu, 23 Sept 2021 at 21:48, Christopher Morrow <morrowc.lists@gmail.com>
wrote:

> This sounds like very naive nat state management behavior.
> Ideally, you'd be able to maintain state of:
> original-src/dst/ports/proto -> in-interface/external ip/port/proto
>

What you describe is called symmetric NAT and is the kind which has the
highest impact on customers. Many NAT traversal techniques fail to work
with this type of NAT. STUN does not work with symmetric NAT.

You purposely want a lesser NAT type which makes NAT traversal easy. This
enables more applications to function despite the NAT. Almost every CPE
avoids symmetric NAT for one of the lesser types for this reason.


>
> unless some internal/original src is double using port/proto ... you
> should really
> have the ability to nat quite a large number of things to a single ipv4
> address.
>

There is no point in optimizing your customer per public CGN IP address
ratio and multiple reasons for not doing so. At some point you will
experience attacks on the CGN address or blacklisting in online games and
so on. Having less customers affected each time that happens is better.

In our network approximately 10% of new customers signed up within the last
year have asked to escape the CGN and get a public IP. This means I need
100 IPv4 addresses per 1000 new customers. Plus 4 IPv4 addresses for the
CGN server. What difference would it make if we could optimize those 4
addresses to be just 2 or maybe 1? What if it caused just a couple more
customers to request a public IP address because they got annoyed by a more
restrictive NAT or by failed sessions?

Regards,

Baldur

The DMCA notices for that single ipv4 /32 must be interesting.


On Thu, Sep 23, 2021 at 11:35 AM Colton Conor <colton.conor@gmail.com>
wrote:

> 300 apartments Mark. No, it's bulk internet and wifi so a single provider.
>
> On Wed, Sep 22, 2021 at 8:01 PM Mark Andrews <marka@isc.org> wrote:
> >
> > And how many apartments where covered by that single IP address? Was this
> > where there is a restriction on other providers so the occupants had no
> > choice of wireline ISP?
> >
> > > On 23 Sep 2021, at 09:38, Colton Conor <colton.conor@gmail.com> wrote:
> > >
> > > Where does this "You can only have about 200-300 subscribers per IPv4
> > > address on a CGN." limit come from? I have seen several apartment
> > > complexes run on a single static IPv4 address using a Mikrotik with
> > > NAT.
> > >
> > > On Wed, Sep 22, 2021 at 2:49 PM Baldur Norddahl
> > > <baldur.norddahl@gmail.com> wrote:
> > >>
> > >>
> > >>
> > >> On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <
> mohta@necom830.hpcl.titech.ac.jp> wrote:
> > >>>
> > >>> Today, as /24 can afford hundreds of thousands of subscribers
> > >>> by NAT, only very large retail ISPs need more than one
> > >>> announcement for IPv4.
> > >>
> > >>
> > >> You can only have about 200-300 subscribers per IPv4 address on a
> CGN. If you try to go further than that, for example by using symmetric
> NAT, you will increase the number of customers that want to get a public
> IPv4 of their own. That will actually decrease the combined efficiency and
> cause you to need more, not less, IPv4 addresses.
> > >>
> > >> Without checking our numbers, I believe we have at least 10% of the
> customers that are paying for a public IPv4 to escape our CGN. This means a
> /24 will only be enough for about 2500 customers maximum. The "nat
> escapers" drown out the efficiency of the NAT pool.
> > >>
> > >> The optimization you need to do is to make the CGN as customer
> friendly as possible instead of trying to squeeze the maximum customers per
> CGN IPv4 address.
> > >>
> > >> Perhaps IPv6 can lower the number of people that need to escape IPv4
> nat. If it helps just a little bit, that alone will make implementing IPv6
> worth it for smaller emerging operators. Buying IPv4 has become very
> expensive. Yes you can profit from selling a public IPv4 address to the
> customer, but there is also the risk that the customer just goes to the
> incumbent, which has old large pools of IPv4 and provides it for free.
> > >>
> > >> Regards,
> > >>
> > >> Baldur
> > >>
> >
> > --
> > Mark Andrews, ISC
> > 1 Seymour St., Dundas Valley, NSW 2117, Australia
> > PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
> >
>

Owen DeLong via NANOG wrote:
>> There are real issues with dual-stack, as this thread started out with.
>> I don't think there is a need neither to invent IPv6 problems, nor to
>> promote IPv6 advantages. What we need is a way out of dual-stack-hell.
> I don’t disagree, but a reversion to IPv4-only certainly won’t do it.

For everyone who does have enough IPv4 addresses, it does. This is the
problem in a nutshell. If that starts trending, IPv6 is done.

> I think the only way out is through.

I hope not, both for IPv6 sake and for the network users. We dont know
how much longer the goal will take, there is materializing a real
possibility we will never quite reach it, and the potholes on the way
are pretty rough.

And as the trip winds on, the landscape is changing, not necessarily for
the better.

One more "any decade now" and another IPv4 replacement/extension might
just happen on the scene and catch on, rendering IPv6 the most wasteful
global technical debacle to date.


> Unfortunately, the IPv6 resistant forces
> are making that hard for everyone else.
>
> Owen

You say that as if it was a surprise, when it should not have been, and
you say that as if something can be done about it, which we should know
by now cannot be the primary focus, since it cannot be done in any
timely fashion. If at all.

Its time to throw mud on the wall and see what sticks. Dual stack and
wait is an ongoing failure slouching to disaster.

Joe

So a single level of NAT and a similar level of customers to that which was
stated could be supported by a single IP. This is not quite a apples to
apples comparison to the double NAT scenario being described below but
close enough for the number of sessions.

Mark

> On 24 Sep 2021, at 01:34, Colton Conor <colton.conor@gmail.com> wrote:
>
> 300 apartments Mark. No, it's bulk internet and wifi so a single provider.
>
> On Wed, Sep 22, 2021 at 8:01 PM Mark Andrews <marka@isc.org> wrote:
>>
>> And how many apartments where covered by that single IP address? Was this
>> where there is a restriction on other providers so the occupants had no
>> choice of wireline ISP?
>>
>>> On 23 Sep 2021, at 09:38, Colton Conor <colton.conor@gmail.com> wrote:
>>>
>>> Where does this "You can only have about 200-300 subscribers per IPv4
>>> address on a CGN." limit come from? I have seen several apartment
>>> complexes run on a single static IPv4 address using a Mikrotik with
>>> NAT.
>>>
>>> On Wed, Sep 22, 2021 at 2:49 PM Baldur Norddahl
>>> <baldur.norddahl@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
>>>>>
>>>>> Today, as /24 can afford hundreds of thousands of subscribers
>>>>> by NAT, only very large retail ISPs need more than one
>>>>> announcement for IPv4.
>>>>
>>>>
>>>> You can only have about 200-300 subscribers per IPv4 address on a CGN. If you try to go further than that, for example by using symmetric NAT, you will increase the number of customers that want to get a public IPv4 of their own. That will actually decrease the combined efficiency and cause you to need more, not less, IPv4 addresses.
>>>>
>>>> Without checking our numbers, I believe we have at least 10% of the customers that are paying for a public IPv4 to escape our CGN. This means a /24 will only be enough for about 2500 customers maximum. The "nat escapers" drown out the efficiency of the NAT pool.
>>>>
>>>> The optimization you need to do is to make the CGN as customer friendly as possible instead of trying to squeeze the maximum customers per CGN IPv4 address.
>>>>
>>>> Perhaps IPv6 can lower the number of people that need to escape IPv4 nat. If it helps just a little bit, that alone will make implementing IPv6 worth it for smaller emerging operators. Buying IPv4 has become very expensive. Yes you can profit from selling a public IPv4 address to the customer, but there is also the risk that the customer just goes to the incumbent, which has old large pools of IPv4 and provides it for free.
>>>>
>>>> Regards,
>>>>
>>>> Baldur
>>>>
>>
>> --
>> Mark Andrews, ISC
>> 1 Seymour St., Dundas Valley, NSW 2117, Australia
>> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
>>

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

> On Sep 23, 2021, at 12:50 , Brian Johnson <brian.johnson@netgeek.us> wrote:
>
> Side question on this thread…
>
> Is it everyones current expectation that if a provider were to switch to IPv6 and drop IPv4 that the customers would all be just fine with that? I believe that there are several applications used by some of the the loudest customers (typically gamers and network gurus), not to mention some business applications that would break or be sub-optimal at best. I see CGN as the band aid to this issue, not the cure to the problem.

Today? no.

At some point when a relatively small number of remaining laggards among major content providers move forward? Yes.

Do you really think that those applications/vendors wouldn’t move quickly if a couple of major eyeball providers announced “Effective X date”, we’re going to start offering a $X/month discount to any customer(s) who are willing to stop using IPv4.

You an only cover an arterial bleed with a band-aid for so long before it becomes silly, septic even. If you’re wondering how quick that point is coming up, I suggest you check your mirrors.

Owen

>
> Discuss…?
>
> - Brian
>
>> On Sep 23, 2021, at 10:46 AM, Owen DeLong via NANOG <nanog@nanog.org> wrote:
>>
>>> There are real issues with dual-stack, as this thread started out with.
>>> I don't think there is a need neither to invent IPv6 problems, nor to
>>> promote IPv6 advantages. What we need is a way out of dual-stack-hell.
>>
>> I don’t disagree, but a reversion to IPv4-only certainly won’t do it.
>>
>> I think the only way out is through. Unfortunately, the IPv6 resistant forces
>> are making that hard for everyone else.
>>
>> Owen
>>
>

> On Sep 23, 2021, at 13:26 , Joe Maimon <jmaimon@jmaimon.com> wrote:
>
>
>
> Owen DeLong via NANOG wrote:
>>> There are real issues with dual-stack, as this thread started out with.
>>> I don't think there is a need neither to invent IPv6 problems, nor to
>>> promote IPv6 advantages. What we need is a way out of dual-stack-hell.
>> I don’t disagree, but a reversion to IPv4-only certainly won’t do it.
>
> For everyone who does have enough IPv4 addresses, it does. This is the problem in a nutshell. If that starts trending, IPv6 is done

This is virtually no-one with any growth. That’s why Azure, AWS, and Google have been snapping up every large prefix that comes on
the market as fast as they can. I don’t see how this can possibly trend long enough to matter before it hits that brick wall.
.
>> I think the only way out is through.
>
> I hope not, both for IPv6 sake and for the network users. We dont know how much longer the goal will take, there is materializing a real possibility we will never quite reach it, and the potholes on the way are pretty rough.

By “the only way out is through” I meant that the only way we can get back to anything resembling mono-stack is, in fact, to complete the transition to IPv6.

> And as the trip winds on, the landscape is changing, not necessarily for the better.

The IPv4 landscape will continue to get worse and worse. It cannot possibly get better, there just aren’t enough addresses for that.

> One more "any decade now" and another IPv4 replacement/extension might just happen on the scene and catch on, rendering IPv6 the most wasteful global technical debacle to date.

If that’s what it takes to move forward with a protocol that has enough addresses, then so be it. I’m not attached to IPv6 particularly, but I recognize that IPv4 can’t keep up. As such, IPv6 is just the best current candidate. If someone offers a better choice, I’m all for it.

>> Unfortunately, the IPv6 resistant forces
>> are making that hard for everyone else.
>>
>> Owen
>
> You say that as if it was a surprise, when it should not have been, and you say that as if something can be done about it, which we should know by now cannot be the primary focus, since it cannot be done in any timely fashion. If at all.

It’s not a surprise, but it is a tragedy.

There are things that can be done about it, but nobody currently wants to do them.

> Its time to throw mud on the wall and see what sticks. Dual stack and wait is an ongoing failure slouching to disaster.

IPv4 is an ongoing failure slouching to disaster, but the IPv6-resistant among us remain in denial about that.

At some point, we are going to have to make a choice about how much longer we want to keep letting them hold us back. It will not be an easy choice, it will not be convenient, and it will not be simple.

The question is how much more pain an dhow much longer will it take before the choice becomes less difficult than the wait?

Owen

> On Sep 23, 2021, at 6:49 PM, Owen DeLong <owen@delong.com> wrote:
>
>
>
>> On Sep 23, 2021, at 12:50 , Brian Johnson <brian.johnson@netgeek.us> wrote:
>>
>> Side question on this thread…
>>
>> Is it everyones current expectation that if a provider were to switch to IPv6 and drop IPv4 that the customers would all be just fine with that? I believe that there are several applications used by some of the the loudest customers (typically gamers and network gurus), not to mention some business applications that would break or be sub-optimal at best. I see CGN as the band aid to this issue, not the cure to the problem.
>
> Today? no.
>
> At some point when a relatively small number of remaining laggards among major content providers move forward? Yes.

So do we just bleed out in the mean time?

>
> Do you really think that those applications/vendors wouldn’t move quickly if a couple of major eyeball providers announced “Effective X date”, we’re going to start offering a $X/month discount to any customer(s) who are willing to stop using IPv4.

I’d be happy to suggest this to my clients, but it’s not a real thing yet. Plus, the average human (even the average CSR at a small regional provider network) has no idea what this means.

>
> You an only cover an arterial bleed with a band-aid for so long before it becomes silly, septic even. If you’re wondering how quick that point is coming up, I suggest you check your mirrors.

Triage suggests that you assist in succession of the current bleeding before being concerned about the next time you are cut. I have BGN deployments that have been in place for 4+ years with little customer knowledge. It has allowed clients to avoid the IPv4 market issues, and in some instances, become a source for others to help compensate for the initial CGN expense.

I totally agree with you in spirit, but I am working on the problem of now, not the problem of some point in the future. The cost of CGN is becoming less expensive than IPv4 space acquisition. I wish this weren’t true.

>
> Owen
>
>>
>> Discuss…?
>>
>> - Brian
>>
>>> On Sep 23, 2021, at 10:46 AM, Owen DeLong via NANOG <nanog@nanog.org> wrote:
>>>
>>>> There are real issues with dual-stack, as this thread started out with.
>>>> I don't think there is a need neither to invent IPv6 problems, nor to
>>>> promote IPv6 advantages. What we need is a way out of dual-stack-hell.
>>>
>>> I don’t disagree, but a reversion to IPv4-only certainly won’t do it.
>>>
>>> I think the only way out is through. Unfortunately, the IPv6 resistant forces
>>> are making that hard for everyone else.
>>>
>>> Owen
>>>
>>
>

On Thu, Sep 23, 2021 at 4:13 PM Baldur Norddahl <baldur.norddahl@gmail.com>
wrote:

>
>
> On Thu, 23 Sept 2021 at 21:48, Christopher Morrow <morrowc.lists@gmail.com>
> wrote:
>
>> This sounds like very naive nat state management behavior.
>> Ideally, you'd be able to maintain state of:
>> original-src/dst/ports/proto -> in-interface/external ip/port/proto
>>
>
> What you describe is called symmetric NAT and is the kind which has the
> highest impact on customers. Many NAT traversal techniques fail to work
> with this type of NAT. STUN does not work with symmetric NAT.
>
>
why. (does stun/etc not work, I mean)
Is the answer: "Because to do it right you need an ALG and that's more
costly/problematic."
or is the answer some other thing?


> You purposely want a lesser NAT type which makes NAT traversal easy. This
> enables more applications to function despite the NAT. Almost every CPE
> avoids symmetric NAT for one of the lesser types for this reason.
>
>

>> unless some internal/original src is double using port/proto ... you
>> should really
>> have the ability to nat quite a large number of things to a single ipv4
>> address.
>>
>
> There is no point in optimizing your customer per public CGN IP address
> ratio and multiple reasons for not doing so. At some point you will
> experience attacks on the CGN address or blacklisting in online games and
> so on. Having less customers affected each time that happens is better.
>
>
Sorry, my point wasn't: "Oh you should just 1 address!"
For many reasons that's crazy sauce.

I was just pointing out that you'd be ABLE TO, if you wanted, sure more ips
is better though.
mostly the '100 ports per backend address' isn't really correct, if you are
smarter in the nat state management.

In our network approximately 10% of new customers signed up within the last
> year have asked to escape the CGN and get a public IP. This means I need
> 100 IPv4 addresses per 1000 new customers. Plus 4 IPv4 addresses for the
> CGN server. What difference would it make if we could optimize those 4
> addresses to be just 2 or maybe 1? What if it caused just a couple more
> customers to request a public IP address because they got annoyed by a more
> restrictive NAT or by failed sessions?
>
> Regards,
>
> Baldur
>
>

It appears that Brian Johnson <brian.johnson@netgeek.us> said:
>Side question on this thread…
>
>Is it everyones current expectation that if a provider were to switch to IPv6 and drop IPv4 that the customers would all be
>just fine with that?

Try sending e-mail to AOL/Yahoo or Hotmail/Outlook over IPv6.

R's,
John