Mailing List Archive

Perhaps the provider only had a single person maintaining the tooling they
used to interact with the IRR records, that person left/was laid off, and
it broke. Perhaps they don't have anyone else that can make it work again,
and they don't want to hire someone else, so they fell back to paper.

Perhaps they have a legal reason to require a paper trail and not rely on
IRR records.

Plenty of possibilities, all plausible.

On Mon, Feb 26, 2024 at 1:58?PM Seth Mattinen via NANOG <nanog@nanog.org>
wrote:

> Why do companies still insist on, or deploy new systems that rely on
> paper LOA for IP and ASN resources? How can this be considered more
> trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right
> now where a VPS provider I'm using will no longer use IRR and only
> accepts new paper LOAs. In the year 2024. I don't understand how anyone
> can go backwards like that.
>
> ~Seth
>

On Mon, 26 Feb 2024 10:57:05 -0800
Seth Mattinen via NANOG <nanog@nanog.org> wrote:

> Why do companies still insist on, or deploy new systems that rely on
> paper LOA for IP and ASN resources? How can this be considered more
> trustworthy than RIR based IRR records?

For routing, some have been proposing that the RPKI. There was some
discussion here a few months ago:

<https://mailman.nanog.org/pipermail/nanog/2023-November/224035.html>

Shortly thereafter this blog post appeared:

<https://mailman.nanog.org/pipermail/nanog/2023-November/224035.html>

> And I'm not even talking about old companies, I have a situation
> right now where a VPS provider I'm using will no longer use IRR and
> only accepts new paper LOAs. In the year 2024. I don't understand how
> anyone can go backwards like that.

Did you ask them why or can you name the provider?

John

A paper LOA is a legally binding document, an IRR record is an IRR record.

Falsifying an LOA that is transmitted digitally is wire fraud and can
basically be handed right over to a DA for injunction and prosecution.

Falsifying IRR records on the other hand leaves more work for the ISP's
lawyers to walk a judge (and jury) through the entire purpose and use of
that system, as opposed to "here's a super important sheet of paper that
they lied on case closed".

-Matt

On Mon, Feb 26, 2024 at 11:57?AM Seth Mattinen via NANOG <nanog@nanog.org>
wrote:

> Why do companies still insist on, or deploy new systems that rely on
> paper LOA for IP and ASN resources? How can this be considered more
> trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right
> now where a VPS provider I'm using will no longer use IRR and only
> accepts new paper LOAs. In the year 2024. I don't understand how anyone
> can go backwards like that.
>
> ~Seth
>


--
Matt Erculiani

One thing that I recently read on this mailing list, is that at least in the US, a transmitting a fraudulent LOA is a federal crime - wire fraud. [0]

Being able to hopefully charge and convict someone performing fraud is a useful deterrent.

-joe

[0] - https://pc.nanog.org/static/published/meetings/NANOG77/2108/20191028_Elverson_Your_As_Is_v1.pdf, page 13.


On 2/26/2024 at 12:58 PM, "Seth Mattinen via NANOG" <nanog@nanog.org> wrote:
>
>Why do companies still insist on, or deploy new systems that rely
>on
>paper LOA for IP and ASN resources? How can this be considered
>more
>trustworthy than RIR based IRR records?
>
>And I'm not even talking about old companies, I have a situation
>right
>now where a VPS provider I'm using will no longer use IRR and only
>accepts new paper LOAs. In the year 2024. I don't understand how
>anyone
>can go backwards like that.
>
>~Seth

Highly anecdotal, but we’ve always refused to provide them, and they’ve always set it up without an LOA.

YMMV since we negotiate larger contracts, but we’ve only ever been asked maybe twice? Both times they admitted they had no idea why they asked for it, so it just seems like some process they forgot to get rid of.

-Dan

> On Feb 26, 2024, at 13:59, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
>
> ?Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
>
> ~Seth

I can’t speak for all providers but when it comes to some downstream
networks we will usually request an LOA as additional proof that the
customer is authorized to announce the prefixes, in addition to the IRR
objects and (where possible) RPKI ROAs. Mainly only a thing where RPKI is
not possible and the only route object available is in a non-auth database
such as RADB. Overall it helps keep a paper trail (as Tom said) in case
someone comes knocking.

Kind regards,
Peter


On Mon, Feb 26, 2024 at 14:13 Tom Beecher <beecher@beecher.cc> wrote:

> Perhaps the provider only had a single person maintaining the tooling they
> used to interact with the IRR records, that person left/was laid off, and
> it broke. Perhaps they don't have anyone else that can make it work again,
> and they don't want to hire someone else, so they fell back to paper.
>
> Perhaps they have a legal reason to require a paper trail and not rely on
> IRR records.
>
> Plenty of possibilities, all plausible.
>
> On Mon, Feb 26, 2024 at 1:58?PM Seth Mattinen via NANOG <nanog@nanog.org>
> wrote:
>
>> Why do companies still insist on, or deploy new systems that rely on
>> paper LOA for IP and ASN resources? How can this be considered more
>> trustworthy than RIR based IRR records?
>>
>> And I'm not even talking about old companies, I have a situation right
>> now where a VPS provider I'm using will no longer use IRR and only
>> accepts new paper LOAs. In the year 2024. I don't understand how anyone
>> can go backwards like that.
>>
>> ~Seth
>>
>

Authentication by letterhead?

Paper LOAs are unauthenticated documents, not worth the paper they are
written on. Usually FAXed, which is even less authenticatable (is that a
word?).

Prosecutors are capable of using digital documents. Do it all the time
with echecks, credit cards, ecommerce orders and ACH payments. But LOAs
are typically civil disputes, not criminal, when someone mistypes an IP
address.

They should verifiy the information in the paper LOA with a registry
anyway. Since LOAs have no intrinsic value, wouldn't be worth the
prosecutors time.

Usually a salesperson or order entry clerk thinks its required because
they've always required it. But no one in the legal department actually
knows what to do with a LOA or how to authenticate them.

Because carriers never authenticate LOAs.


On Mon, 26 Feb 2024, Matt Erculiani wrote:
> A paper LOA is a legally binding document, an IRR record is an IRR record.
> Falsifying an LOA that is transmitted digitally is wire fraud and can
> basically be handed right over to a DA for injunction and prosecution.
>
> Falsifying IRR records on the other hand leaves more work for the ISP's
> lawyers to walk a judge (and jury) through the entire purpose and use of
> that system, as opposed to "here's a super important sheet of paper that
> they lied on case closed". 
>
> -Matt
>
> On Mon, Feb 26, 2024 at 11:57?AM Seth Mattinen via NANOG <nanog@nanog.org>
> wrote:
> Why do companies still insist on, or deploy new systems that
> rely on
> paper LOA for IP and ASN resources? How can this be considered
> more
> trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation
> right
> now where a VPS provider I'm using will no longer use IRR and
> only
> accepts new paper LOAs. In the year 2024. I don't understand how
> anyone
> can go backwards like that.
>
> ~Seth
>
>
>
> --
> Matt Erculiani
>
>

Most important parts on the LOA are the explicit ASN, the name to be found
in the cross-connect order portal and local contact data. Contractors need
that.

Global networks rarely have a contact appropriate for provisioning in a
public facing database.

On Mon, Feb 26, 2024 at 14:50 Sean Donelan <sean@donelan.com> wrote:

> Authentication by letterhead?
>
> Paper LOAs are unauthenticated documents, not worth the paper they are
> written on. Usually FAXed, which is even less authenticatable (is that a
> word?).
>
> Prosecutors are capable of using digital documents. Do it all the time
> with echecks, credit cards, ecommerce orders and ACH payments. But LOAs
> are typically civil disputes, not criminal, when someone mistypes an IP
> address.
>
> They should verifiy the information in the paper LOA with a registry
> anyway. Since LOAs have no intrinsic value, wouldn't be worth the
> prosecutors time.
>
> Usually a salesperson or order entry clerk thinks its required because
> they've always required it. But no one in the legal department actually
> knows what to do with a LOA or how to authenticate them.
>
> Because carriers never authenticate LOAs.
>
>
> On Mon, 26 Feb 2024, Matt Erculiani wrote:
> > A paper LOA is a legally binding document, an IRR record is an IRR
> record.
> > Falsifying an LOA that is transmitted digitally is wire fraud and can
> > basically be handed right over to a DA for injunction and prosecution.
> >
> > Falsifying IRR records on the other hand leaves more work for the ISP's
> > lawyers to walk a judge (and jury) through the entire purpose and use of
> > that system, as opposed to "here's a super important sheet of paper that
> > they lied on case closed".
> >
> > -Matt
> >
> > On Mon, Feb 26, 2024 at 11:57?AM Seth Mattinen via NANOG <
> nanog@nanog.org>
> > wrote:
> > Why do companies still insist on, or deploy new systems that
> > rely on
> > paper LOA for IP and ASN resources? How can this be considered
> > more
> > trustworthy than RIR based IRR records?
> >
> > And I'm not even talking about old companies, I have a situation
> > right
> > now where a VPS provider I'm using will no longer use IRR and
> > only
> > accepts new paper LOAs. In the year 2024. I don't understand how
> > anyone
> > can go backwards like that.
> >
> > ~Seth
> >
> >
> >
> > --
> > Matt Erculiani
> >
> >
>

There is one purpose: to facilitate IP fraud, and maintain currently fraudulently routed IPs.

Anyone can dummy up a LOA. And there is still quite a lot of unrouted IP space. VPS providers know this, and know their customers are submitting fake LOAs. But it is sort of the business VPS providers are in.

Is it some sort of serious crime in the US though? Well, just submit the LOA from outside the US. Plus, the entity being defrauded is the IP holder, not the VPS provider or their customer. If you are an IP holder, good luck getting the VPS provider to give you a copy of the fake LOA. It is not in their interest to throw their customers under the bus. You would have to give them a court order. So if you look for unrouted IP space, registered to a non-US organization (ex. Canada), and submit a fake LOA from another country (London, UK for instance), you are unlikely to get tracked down for wire fraud.

And you might ask, well, why would a VPS provider accept an LOA from the UK for an IP block registered to a Canadian organization? Well, clearly it isn’t in the VPS provider’s interest to look into the LOAs too much. As long as the IP space is unrouted, they will approve it. The LOA is basically just a liability shield for the VPS provider. It is not a crime to be deceived, though the due diligence beggars belief.

So I had this happen. There was a /24 being hijacked by a VPS provider. I told them this was fraud, and they asked me if I wanted to “rescind the LOA”. I told them I never gave them a LOA. They dropped the /24 immediately. They refused to provide a copy of the LOA. So pretty hard to pursue any sort of wire fraud charges.

So a VPS provider asking for a paper LOA is basically asking you to lie to them, to protect them from liability. They will just drop the IP prefix if there is any contact from the actual IP holder.



Tom



> On Feb 26, 2024, at 10:57 AM, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
>
> Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
>
> ~Seth

I don't have any examples of anyone still using paper LOAs except for
Cogent.

Aaron


On 2/26/2024 12:57 PM, Seth Mattinen via NANOG wrote:
> Why do companies still insist on, or deploy new systems that rely on
> paper LOA for IP and ASN resources? How can this be considered more
> trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right
> now where a VPS provider I'm using will no longer use IRR and only
> accepts new paper LOAs. In the year 2024. I don't understand how
> anyone can go backwards like that.
>
> ~Seth

Also known as an cross-connect order form.

Why FAX a piece of paper?

Nobody cross-checks it, until after it goes wrong.

On Mon, 26 Feb 2024, Ren Provo wrote:
> Most important parts on the LOA are the explicit ASN, the name to be found
> in the cross-connect order portal and local contact data.  Contractors need
> that.
>
> Global networks rarely have a contact appropriate for provisioning in a
> public facing database.
>
> On Mon, Feb 26, 2024 at 14:50 Sean Donelan <sean@donelan.com> wrote:
> Authentication by letterhead?
>
> Paper LOAs are unauthenticated documents, not worth the paper
> they are
> written on. Usually FAXed, which is even less authenticatable
> (is that a
> word?).
>
> Prosecutors are capable of using digital documents. Do it all
> the time
> with echecks, credit cards, ecommerce orders and ACH payments. 
> But LOAs
> are typically civil disputes, not criminal, when someone
> mistypes an IP
> address.
>
> They should verifiy the information in the paper LOA with a
> registry
> anyway.  Since LOAs have no intrinsic value, wouldn't be worth
> the
> prosecutors time.
>
> Usually a salesperson or order entry clerk thinks its required
> because
> they've always required it.  But no one in the legal department
> actually
> knows what to do with a LOA or how to authenticate them.
>
> Because carriers never authenticate LOAs.
>
>
> On Mon, 26 Feb 2024, Matt Erculiani wrote:
> > A paper LOA is a legally binding document, an IRR record is an
> IRR record.
> > Falsifying an LOA that is transmitted digitally is wire fraud
> and can
> > basically be handed right over to a DA for injunction and
> prosecution.
> >
> > Falsifying IRR records on the other hand leaves more work for
> the ISP's
> > lawyers to walk a judge (and jury) through the entire purpose
> and use of
> > that system, as opposed to "here's a super important sheet of
> paper that
> > they lied on case closed". 
> >
> > -Matt
> >
> > On Mon, Feb 26, 2024 at 11:57?AM Seth Mattinen via NANOG
> <nanog@nanog.org>
> > wrote:
> >       Why do companies still insist on, or deploy new systems
> that
> >       rely on
> >       paper LOA for IP and ASN resources? How can this be
> considered
> >       more
> >       trustworthy than RIR based IRR records?
> >
> >       And I'm not even talking about old companies, I have a
> situation
> >       right
> >       now where a VPS provider I'm using will no longer use
> IRR and
> >       only
> >       accepts new paper LOAs. In the year 2024. I don't
> understand how
> >       anyone
> >       can go backwards like that.
> >
> >       ~Seth
> >
> >
> >
> > --
> > Matt Erculiani
> >
> >
>
>
>

On 2/26/24 10:57, Seth Mattinen via NANOG wrote:
> Why do companies still insist on, or deploy new systems that rely on
> paper LOA for IP and ASN resources? How can this be considered more
> trustworthy than RIR based IRR records?

* They're an authoritative signed document with legal penalties for
forgery.

* The same LOA is often required by datacenter operators and other third
parties for cross-connect authority, etc.

--
Jay Hennigan - jay@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

We just switched over to IRR routing with Cogent, it is available.  It's
just not on by default.

Best Regards,

Jason

On 2/26/24 3:14 PM, Aaron Wendel wrote:
> I don't have any examples of anyone still using paper LOAs except for
> Cogent.
>
> Aaron
>
>
> On 2/26/2024 12:57 PM, Seth Mattinen via NANOG wrote:
>> Why do companies still insist on, or deploy new systems that rely on
>> paper LOA for IP and ASN resources? How can this be considered more
>> trustworthy than RIR based IRR records?
>>
>> And I'm not even talking about old companies, I have a situation
>> right now where a VPS provider I'm using will no longer use IRR and
>> only accepts new paper LOAs. In the year 2024. I don't understand how
>> anyone can go backwards like that.
>>
>> ~Seth
>

Hi Seth,

LOAs can't be considered more trustworthy than IRR objects. The RIRs operate IRRdb services as part of the services they offer which network operators should be using instead of the free and paid non-authoritative IRRdb operators.

If you don’t mind, could you please reach out to me off-list with who the VPS hosting provider is that is only accepting LOAs? I’d like to reach out to them to discuss their decision.

I’m doing a talk at APRICOT 2024 on using ROAs to replace LOAs. In my view there's no reason why network operators cannot use ROAs instead to validate the routes received from their peers, be they upstream or downstream.

Regards,
Christopher Hawker


Sent from my iPhone

On 27 Feb 2024, at 1:57?am, Seth Mattinen via NANOG <nanog@nanog.org> wrote:

?Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?

And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.

~Seth

Hi All,

There is this blogpost from the FIRST netsec-sig group, about this topic,
available at
https://www.first.org/blog/20231222-Is-the-LoA-DoA-for-Routing


I totally agree with Christopher. The above blogpost ends with (for those
who don't like to follow links):

"With the current level of RPKI adoption, now is time to adopt it as the
best current practice, to discontinue the usage of LOAs for authorization
of routing, and to instead rely on ROV, ROAs, and the cryptographic trust
we all can obtain from them!"


Best Regards,
Carlos


On Tue, 27 Feb 2024, Christopher Hawker wrote:

> Hi Seth,
>
> LOAs can't be considered more trustworthy than IRR objects. The RIRs operate IRRdb services as part of the services they offer which
> network operators should be using instead of the free and paid non-authoritative IRRdb operators.
>
> If you don?t mind, could you please reach out to me off-list with who the VPS hosting provider is that is only accepting LOAs? I?d like to
> reach out to them to discuss their decision.
>
> I?m doing a talk at APRICOT 2024 on using ROAs to replace LOAs. In my view there's no reason why network operators cannot use ROAs instead
> to validate the routes received from their peers, be they upstream or downstream.
>
> Regards,
> Christopher Hawker
>
>
> Sent from my iPhone
>
> On 27 Feb 2024, at 1:57?am, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
>
> ?Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be
> considered more trustworthy than RIR based IRR records?
>
> And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use
> IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
>
> ~Seth
>
>
>

Hi,
(please see inline)


On Mon, 26 Feb 2024, Tom Samplonius wrote:

>
> There is one purpose: to facilitate IP fraud, and maintain currently fraudulently routed IPs.

Yes!


> Anyone can dummy up a LOA. And there is still quite a lot of unrouted
> IP space.

Yes. But the endgame is not always the same, when miscreants push fake
LOAs (for routing).

I was recently made aware about https://loa.tools

This is how easy it gets......



> VPS providers know this, and know their customers are submitting fake
> LOAs.

Then it's a good idea to require cryptographic evidence of
ownership/authorization, by resorting to RPKI/ROV.



> But it is sort of the business VPS providers are in.

That can by true for some. I hope it isn't true for the majority of them.



> Is it some sort of serious crime in the US though? Well, just submit
> the LOA from outside the US. Plus, the entity being defrauded is the IP
> holder, not the VPS provider or their customer. If you are an IP
> holder, good luck getting the VPS provider to give you a copy of the
> fake LOA. It is not in their interest to throw their customers under
> the bus. You would have to give them a court order. So if you look
> for unrouted IP space, registered to a non-US organization (ex. Canada),
> and submit a fake LOA from another country (London, UK for instance),
> you are unlikely to get tracked down for wire fraud.

Good example, but there are also some less central
jurisdictions/coutries/territories, where local law enforcement
cooperation is even harder to get. And miscreants know this very well.



> And you might ask, well, why would a VPS provider accept an LOA from
> the UK for an IP block registered to a Canadian organization? Well,
> clearly it isn?t in the VPS provider?s interest to look into the LOAs
> too much.

While it doesn't change anything in the "interest" vector, resorting to
RPKI/ROV would probably be less work.



> As long as the IP space is unrouted, they will approve it. The LOA is
> basically just a liability shield for the VPS provider. It is not a
> crime to be deceived, though the due diligence beggars belief.

Even if the IP space is routed, can't anycast be invoked...? :-)))



> So I had this happen. There was a /24 being hijacked by a VPS
> provider. I told them this was fraud, and they asked me if I wanted to
> ?rescind the LOA?. I told them I never gave them a LOA. They dropped
> the /24 immediately. They refused to provide a copy of the LOA. So
> pretty hard to pursue any sort of wire fraud charges.

That's the thing with LOAs for routing, the only way to be sure is to
check if there is a valid ROA with the prefix, length and ASN. :-)

If the customer can't make a valid ROA, or make the legitimate owner
produce one, then the claim on the LOA is bogus...



> So a VPS provider asking for a paper LOA is basically asking you to
> lie to them, to protect them from liability. They will just drop the IP
> prefix if there is any contact from the actual IP holder.

If the legitimate IP holder has closed shop, there will not be a contact.
And miscreants also know this very well...


Cheers,
Carlos



> Tom
>
>
>
>> On Feb 26, 2024, at 10:57 AM, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
>>
>> Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
>>
>> And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
>>
>> ~Seth
>
>

On Mon, Feb 26, 2024 at 1:20?PM Joe via NANOG <nanog@nanog.org> wrote:
>
> One thing that I recently read on this mailing list, is that at least in the US, a transmitting a fraudulent LOA is a federal crime - wire fraud. [0]
> Being able to hopefully charge and convict someone performing fraud is a useful deterrent.

This would be just as true of an Emailed declaration signed with the
sender's name or other
digital representation of a signature. If there is a fraudulent
scheme, then deliberately
providing a false emailed declaration of authorization just as criminal.

My suggestion would be that a LOA should only ever be used as a
Supportive document,
it could be used for that, and Verifying the data using IRR or RPKI
after would still be necessary.
An LOA on its own should never be enough.

An LOA can still be Incorrect or Wrong due to a Typo'd ASN or IP
number, but Not fraudulent.
And even if the information is deliberately wrong it might not meet
the conditions for fraud.

It is also possible the sender of the LOA can send an erroneous
document and have No
legal responsibility for the results of incorrectly including some IP
or AS number on the form.

Surely a network service provider must have some level of duty to
verify the authenticity of
information furnished on the LOAs and confirm that the IP numbers are
Not incorrectly entered,
for example clerical errors in processing the document.

> -joe
--
-J