Mailing List Archive

ipv4/25s and above
I am kind of curious as to the distribution of connections to smaller
companies and other entities that need more than one ipv4 address, but
don't run BGP. So, for as an ISP or infrastructure provider, what is
the typical percentage nowadays of /32s /31s /30s... /25s of stuff
that gets run "elsewhere"?

Is there any correlation between the number of IPs a customer gets and
the amount of bandwidth they buy?

Obviously "retail", home use is /32s and there's an increasing amount
of CGNAT, but I can't help but imagine there are thousands of folk
running /27s and /29s for every /24 or /22 out there.

I've been paying 15/month for a /29 for forever, but barely use it.

--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
Re: ipv4/25s and above [ In reply to ]
Dave,

I work for a smaller ISP in the Midwest with clients coast-to-coast. I deliver internet to about 2/3 rds of them over private ethernet circuits, the rest I deliver private addressed SIP service. Aside a handful of them who advertise their own /24 to me over BGP, the rest are exclusively smaller than that. /29's and /30's are the most common, with a peppering of /28's and /27's. My total IP space is about a /19.

Cheers!


?On 11/16/22, 8:41 AM, "NANOG on behalf of Dave Taht" <nanog-bounces+sam=coeosolutions.com@nanog.org on behalf of dave.taht@gmail.com> wrote:

I am kind of curious as to the distribution of connections to smaller
companies and other entities that need more than one ipv4 address, but
don't run BGP. So, for as an ISP or infrastructure provider, what is
the typical percentage nowadays of /32s /31s /30s... /25s of stuff
that gets run "elsewhere"?

Is there any correlation between the number of IPs a customer gets and
the amount of bandwidth they buy?

Obviously "retail", home use is /32s and there's an increasing amount
of CGNAT, but I can't help but imagine there are thousands of folk
running /27s and /29s for every /24 or /22 out there.

I've been paying 15/month for a /29 for forever, but barely use it.

--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
Re: ipv4/25s and above [ In reply to ]
On 11/16/22 16:39, Dave Taht wrote:
> I am kind of curious as to the distribution of connections to smaller
> companies and other entities that need more than one ipv4 address, but
> don't run BGP. So, for as an ISP or infrastructure provider, what is
> the typical percentage nowadays of /32s /31s /30s... /25s of stuff
> that gets run "elsewhere"?
>
> Is there any correlation between the number of IPs a customer gets and
> the amount of bandwidth they buy?
>
> Obviously "retail", home use is /32s and there's an increasing amount
> of CGNAT, but I can't help but imagine there are thousands of folk
> running /27s and /29s for every /24 or /22 out there.
>
> I've been paying 15/month for a /29 for forever, but barely use it.

For our DIA/Enterprise business, we offer customers a /30 for p2p link,
and a /29 as initial standard for onward assignment within their LAN.

Interestingly, most customers will NAT on the p2p address space, and
barely use the onward assignment. In such cases, link migrations cause
issues, because customers bake their internal configurations against
those p2p IP addresses, which are pegged to specific edge routers on our
side that can't move due to our need to minimize the iBGP footprint in
the network. It's not a major issue in absolute terms, but a headache
nonetheless.

We can offer customers up to a /24 maximum (i.e., we will let the /29
expand into a /24), and no more. If they need more than a /24, time to
speak to AFRINIC.

We don't charge for address space. Our Sales people want us to, but we
don't feel it makes sense. It's not a big-enough deterrent for us to
keep IPv4 stock. And when we do run out of IPv4 space, it will be like
having billions of $$ on a deserted island.

Mark.
Re: ipv4/25s and above [ In reply to ]
Mark Tinka wrote:
>
> For our DIA/Enterprise business, we offer customers a /30 for p2p
> link, and a /29 as initial standard for onward assignment within their
> LAN.

You could instead use a /31. Or private/enterprise-private or unnumbered
and route them the single /32 to use for their NAT on say a loopback
interface. And the /29 ? I would reserve it but not assign it without a
formal request.
>
> Interestingly, most customers will NAT on the p2p address space, and
> barely use the onward assignment. In such cases, link migrations cause
> issues, because customers bake their internal configurations against
> those p2p IP addresses, which are pegged to specific edge routers on
> our side that can't move due to our need to minimize the iBGP
> footprint in the network. It's not a major issue in absolute terms,
> but a headache nonetheless.
>
See above.

> We can offer customers up to a /24 maximum (i.e., we will let the /29
> expand into a /24), and no more.

Either you have lots of fallow ground or very few customers. I dont see
how this strategy would work elsewhere.

> If they need more than a /24, time to speak to AFRINIC.
>
> We don't charge for address space. Our Sales people want us to, but we
> don't feel it makes sense. It's not a big-enough deterrent for us to
> keep IPv4 stock. And when we do run out of IPv4 space, it will be like
> having billions of $$ on a deserted island.
>
> Mark.
>
Your sales people are right. Since you can deliver quite usable service
that enables them to operate just as they have before with a single /32,
and with technical advantages to yourself, all the extra wasted integers
should be bringing in value.

And since its quite nice to assign a loopback to every CPE anyways, two
birds, one stone. Carry that in your iBGP.

Best,

Joe
Re: ipv4/25s and above [ In reply to ]
On 11/17/22 19:55, Joe Maimon wrote:

>
> You could instead use a /31.

We could, but many of our DIA customers have all manner of CPE's that
may or may not support this. Having unique designs per customer does not
scale well.


> Or private/enterprise-private

Yeah, don't like that, although I understand why some operators use it.


> or unnumbered and route them the single /32 to use for their NAT on
> say a loopback interface.

Same as the CPE issue I described above.


> And the /29 ? I would reserve it but not assign it without a formal
> request.


We have some products that can be considered sub-DIA that do not come
with the /29 as standard. Those tend to be the majority of the market.


> Either you have lots of fallow ground or very few customers.

A bit of both.

Our main business is wholesale IP Transit. The Enterprise piece is
growing, but we are not trying to save the world. It's a specific focus,
and we don't do consumer.


> I dont see how this strategy would work elsewhere.

To be honest, we'll keep using IPv4 for as long as we have it, and for
as long as we can get it from AFRINIC. But it's not where we are betting
the farm - that is for IPv6.


> Your sales people are right. Since you can deliver quite usable
> service that enables them to operate just as they have before with a
> single /32, and with technical advantages to yourself, all the extra
> wasted integers should be bringing in value.

At the risk of using IP addresses to prop uo sales numbers, and then you
run out sooner because one customer decided to pay lots of $$ for a /22,
even when they don't need it. Not the kind of potato I want to taste,
because we have seen that proposed far too many times to know it will
become a reality when the commissions are due.

Mark.
Re: ipv4/25s and above [ In reply to ]
Mark Tinka wrote:
>
>
> On 11/17/22 19:55, Joe Maimon wrote:
>
>>
>> You could instead use a /31.
>
> We could, but many of our DIA customers have all manner of CPE's that
> may or may not support this. Having unique designs per customer does
> not scale well.

its almost 2023. /31 support is easily mandatory. You should make it
mandatory.

How many customer addressing designs does that total, 2? So that would
be 1 more than you have already? Dont buy it.

Its 2023, your folk should be able to handle addressing more advanced
than from the 90s. And your betting the future on IPv6?

>
>
>> Or private/enterprise-private
>
> Yeah, don't like that, although I understand why some operators use it.

The only issue with it is traceroute although that isnt necessarily a
problem.

And the CPE sourced traffic should either be all internal or sourced
from the loopback.

OTOH CoPP protection becomes a lot easier when fewer of the CP
addressing is globally routed.

>
>
>> or unnumbered and route them the single /32 to use for their NAT on
>> say a loopback interface.
>
> Same as the CPE issue I described above.

Truth is the real issue isnt CPE support, its user support. Most
users(even their "IT" folk) really cant wrap their brain around more
than the bare basic concepts, if that much.

And you can simply say, IPv4 is limited, this is the base model
addressing included with the service, if your inabilities are preventing
you from using networking techniques that have been standardized and
usable for decades, then feel free to pony up for either for your
comfort level or for support of your inabilities.

>
> To be honest, we'll keep using IPv4 for as long as we have it, and for
> as long as we can get it from AFRINIC. But it's not where we are
> betting the farm - that is for IPv6.

This is the crux. So long as you can obtain more, justifiable
consumption is rewarded with additional resources, dis-incentivizing any
addressing efficiency progress from the ancient /30 p2p + /29(or larger)
routed block.

You may wish to lay groundwork to nibble backwards when runout occurs
for you.

Its on Afrinic to try and preserve their pool if they wish to by doing
things such as getting it across that progress in addressing efficiency
is an important consideration in fulfilling requests for additional
resources.

>
>
>> Your sales people are right. Since you can deliver quite usable
>> service that enables them to operate just as they have before with a
>> single /32, and with technical advantages to yourself, all the extra
>> wasted integers should be bringing in value.
>
> At the risk of using IP addresses to prop uo sales numbers, and then
> you run out sooner because one customer decided to pay lots of $$ for
> a /22, even when they don't need it.

If they need more, they pay more and they get more. Most of the rest of
the world is operating or moving to operate in that fashion.

You would still require them to submit a formal request documenting
their need. And paying more is likely to mean that its a more honest
request.

But see the crux above. If your RiR isnt frowning on such behavior then
its poor strategy to implement it.

Although, if you can get away with it, allocating the /30 + /29 and
implementing it in an easy-to-clawback approach might be a good strategy.

> Not the kind of potato I want to taste, because we have seen that
> proposed far too many times to know it will become a reality when the
> commissions are due.
>
> Mark.
>
>
Thats a question of internal discipline without motivating external
factors. Odds are those factors will arrive and I would advise
preparedness for them.

Joe
Re: ipv4/25s and above [ In reply to ]
>
>> Either you have lots of fallow ground or very few customers.
>
> A bit of both.

Regarding the former, perhaps you should return some of that to AFRINIC as required in your RSA before throwing stones at other providers in the region.

Owen
Re: ipv4/25s and above [ In reply to ]
> On Nov 18, 2022, at 03:44, Joe Maimon <jmaimon@jmaimon.com> wrote:
>
>
>
> Mark Tinka wrote:
>>
>>
>> On 11/17/22 19:55, Joe Maimon wrote:
>>
>>>
>>> You could instead use a /31.
>>
>> We could, but many of our DIA customers have all manner of CPE's that may or may not support this. Having unique designs per customer does not scale well.
>
> its almost 2023. /31 support is easily mandatory. You should make it mandatory.

Much of Africa in 2023 runs on what the US put into the resale market in the late 1990s, tragically.

> Its 2023, your folk should be able to handle addressing more advanced than from the 90s. And your betting the future on IPv6?

They don’t really have a lot of alternatives.

>> To be honest, we'll keep using IPv4 for as long as we have it, and for as long as we can get it from AFRINIC. But it's not where we are betting the farm - that is for IPv6.

And yet you wonder why I consider AFRINIC’s artificial extension of the free pool through draconian austerity measures to be a global problem?

> Its on Afrinic to try and preserve their pool if they wish to by doing things such as getting it across that progress in addressing efficiency is an important consideration in fulfilling requests for additional resources.

Instead of this, they’re mostly ignoring policy, implementing draconian restrictions on people getting space from the free pool, and buying into various forms of reality avoidance.

> But see the crux above. If your RiR isnt frowning on such behavior then its poor strategy to implement it.

So far, AFRINIC has given a complete pass to Tinka’s organization and their documented excessive unused address space despite policy that prohibits them from doing so. However, AFRINIC management and board seem to have extreme difficulty with reading their governing documents in anything resembling a logical interpretation.

Owen
Re: ipv4/25s and above [ In reply to ]
On 11/18/22 13:44, Joe Maimon wrote:

>
> its almost 2023. /31 support is easily mandatory. You should make it
> mandatory.

I don't make the gear.


> How many customer addressing designs does that total, 2? So that would
> be 1 more than you have already? Dont buy it.

That's fine.


>
> Its 2023, your folk should be able to handle addressing more advanced
> than from the 90s.

Customers will do what they do.


> And your betting the future on IPv6?

Actually, yeah.


> The only issue with it is traceroute although that isnt necessarily a
> problem.

We and some of our customers find that useful.


>
> And the CPE sourced traffic should either be all internal or sourced
> from the loopback.

It's not my CPE, I don't get to make the rules.


>
> OTOH CoPP protection becomes a lot easier when fewer of the CP
> addressing is globally routed.

As does not connecting any cables to the device.

Seriously, all good points. We just have more pressing issues to deal with.


> Truth is the real issue isnt CPE support, its user support. Most
> users(even their "IT" folk) really cant wrap their brain around more
> than the bare basic concepts, if that much.
>
> And you can simply say, IPv4 is limited, this is the base model
> addressing included with the service, if your inabilities are
> preventing you from using networking techniques that have been
> standardized and usable for decades, then feel free to pony up for
> either for your comfort level or for support of your inabilities.

Okay.


> This is the crux. So long as you can obtain more, justifiable
> consumption is rewarded with additional resources, dis-incentivizing
> any addressing efficiency progress from the ancient /30 p2p + /29(or
> larger) routed block.
>
> You may wish to lay groundwork to nibble backwards when runout occurs
> for you.
>
> Its on Afrinic to try and preserve their pool if they wish to by doing
> things such as getting it across that progress in addressing
> efficiency is an important consideration in fulfilling requests for
> additional resources.

We aren't really going to expend too many resources on trying to delay
the use of IPv6. I understand that I am speaking from a place of
priviledge, having as much IPv4 as we do, but even then, we will connect
as many as we connect using either "efficient" or "inefficient"
techniques, and neither will prevent run-out, eventually.

We want to be able to keep servicing customers, long after we are gone.
That is what we care about.


>
> If they need more, they pay more and they get more. Most of the rest
> of the world is operating or moving to operate in that fashion.

It's fine for most of the world to do what it wants. I won't begrudge
them that. Over here, we will do what we feel works for us.


>
> You would still require them to submit a formal request documenting
> their need. And paying more is likely to mean that its a more honest
> request.

We do this already, only without asking them for cash.


>
> But see the crux above. If your RiR isnt frowning on such behavior
> then its poor strategy to implement it.

I might have missed the part where RIR's tell me how to operate.


>
> Although, if you can get away with it, allocating the /30 + /29 and
> implementing it in an easy-to-clawback approach might be a good strategy.

There is reasonable customer movement between competitors that address
space comes and goes.


> Thats a question of internal discipline without motivating external
> factors. Odds are those factors will arrive and I would advise
> preparedness for them.

Have you ever been in sales :-)?

Mark.
ipv4/25s and above [ In reply to ]
Dear NANOG-ers,
Hope this email finds you in good health!
Please see my comments below, inline...
Thanks.

Le samedi 19 novembre 2022, Owen DeLong via NANOG <nanog@nanog.org> a
écrit :

> >
> >> Either you have lots of fallow ground or very few customers.
> >
> > A bit of both.
>
> Regarding the former, perhaps you should return some of that to AFRINIC as
> required in your RSA before throwing stones at other providers in the
> region.


>
Hi Owen,
Thanks for your email, brother!

Please, could you elaborate on the above?
Remark! you may need to start a separate thread :-/
...yes! i have already read your next email.

Shalom,
--sb.



> Owen
>
>

--

Best Regards !
__
baya.sylvain[AT cmNOG DOT cm]|<https://cmnog.cm/dokuwiki/Structure>
Subscribe to Mailing List: <https://lists.cmnog.cm/mailman/listinfo/cmnog/>
__
#?LASAINTEBIBLE?|#?Romains15?:33«Que LE ?#?DIEU? de ?#?Paix? soit avec vous
tous! ?#?Amen?!»
?#?MaPrière? est que tu naisses de nouveau. #Chrétiennement?
«Comme une biche soupire après des courants d’eau, ainsi mon âme soupire
après TOI, ô DIEU!»(#Psaumes42:2)
Re: ipv4/25s and above [ In reply to ]
On 11/18/22 6:44 AM, Joe Maimon wrote:
>> We could, but many of our DIA customers have all manner of CPE's that
>> may or may not support this. Having unique designs per customer does
>> not scale well.
> its almost 2023. /31 support is easily mandatory. You should make it
> mandatory.

Mikrotik still doesn't support /31 addressing. I had a customer who was
configuring their "router" the other day and we found this out. Has to move
to a /30 on the link.

He's in Africa, and I'm 100% certain Mikrotik is a go to customer router
there. There's people peering on Mikrotik even!
--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net
Re: ipv4/25s and above [ In reply to ]
Le Sat, Nov 19, 2022 at 01:39:59PM -0500, Bryan Fields a ?crit :
> On 11/18/22 6:44 AM, Joe Maimon wrote:
> >> We could, but many of our DIA customers have all manner of CPE's that
> >> may or may not support this. Having unique designs per customer does
> >> not scale well.
> > its almost 2023. /31 support is easily mandatory. You should make it
> > mandatory.
>
> Mikrotik still doesn't support /31 addressing. I had a customer who was
> configuring their "router" the other day and we found this out. Has to move
> to a /30 on the link.
>

You cannot configure a /31 on a Mikrotik yet you can play with /32 to overcome
this limit.
Re: ipv4/25s and above [ In reply to ]
I do not like mikrotik, but I need to say that RouterOS does support /31.

All that you need to do, beyond set /31 at address for netmask, is check if
the other address is defined at the network address.



Em sáb., 19 de nov. de 2022 15:58, Denis Fondras <xxnog@ledeuns.net>
escreveu:

> Le Sat, Nov 19, 2022 at 01:39:59PM -0500, Bryan Fields a écrit :
> > On 11/18/22 6:44 AM, Joe Maimon wrote:
> > >> We could, but many of our DIA customers have all manner of CPE's that
> > >> may or may not support this. Having unique designs per customer does
> > >> not scale well.
> > > its almost 2023. /31 support is easily mandatory. You should make it
> > > mandatory.
> >
> > Mikrotik still doesn't support /31 addressing. I had a customer who was
> > configuring their "router" the other day and we found this out. Has to
> move
> > to a /30 on the link.
> >
>
> You cannot configure a /31 on a Mikrotik yet you can play with /32 to
> overcome
> this limit.
>
Re: ipv4/25s and above [ In reply to ]
On 11/19/22 3:12 PM, Douglas Fischer wrote:
> I do not like mikrotik, but I need to say that RouterOS does support /31.
>
> All that you need to do, beyond set /31 at address for netmask, is check if
> the other address is defined at the network address.

Can you show some docs on this? I gave a subnet config to my customer of
255.255.255.254 and it wouldn't take it. I reconfigured it to a /30 (.252)
and it worked for them. This was a on a RB2011 about 3 months ago.

I search at the time showed it was a know issue. We laughed about it and
moved on.

--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net
Re: ipv4/25s and above [ In reply to ]
On Sat, Nov 19, 2022 at 5:13 PM Douglas Fischer
<fischerdouglas@gmail.com> wrote:
>
> I do not like mikrotik, but I need to say that RouterOS does support /31.
>
> All that you need to do, beyond set /31 at address for netmask, is check if the other address is defined at the network address.


Is /31 supported only in bleeding-edge versions such as v7 or in the
more conventional RouterOS versions ?


Rubens
Re: ipv4/25s and above [ In reply to ]
>> But see the crux above. If your RiR isnt frowning on such behavior then its poor strategy to implement it.
>
> I might have missed the part where RIR's tell me how to operate.

Allow me to help:

https://afrinic.net/ast/pdf/services/afrinic-rsa-en-201801.pdf?
afrinic-rsa-en-201801
PDF Document · 128 KB

Specifically, I refer you to sections 2(b), 2(c), 2(d), 4, 6, and 7.

Owen
Re: ipv4/25s and above [ In reply to ]
Before this conversation forked off in a direction I didn't want it to go,
I'd like to thank everyone, privately and publicly, that gave me a hint as
to the distribution of /25s and greater in their networks.

I was at the time, trying to get "libreqos.io" to crack the 32k customer
barrier, which we did soon afterwards, and to get a decent histogram for
our trie emulations of what real customer traffic might look like once we
started scaling past 40Gbit. (the XDP trie lookup is pretty expensive for
us currently, but a lot cheaper than the /32s)

Some idea of failover mechanisms other than BGP (igps are all over the map)
came from this conversation also. I would certainly like to learn more
about how well that works.

A new question I have... is there any decent rules for CGNAT vs bandwidth
vs customer allocations of ipv4 space? I've had a few horror stories shared
with me privately about that, of note was 9000 *fiber* users behind a
single /32. with no way to haul ipv6 or more ips there....