Mailing List Archive

RPKI validation weirdness
Hello all

First of all, sorry for the long text wall, but I thin I have a bit of
an interesting issue here. I have a router that announces a prefix
that is not RPKI signed at all, hence sould neither appear valid nor
invalid.

It _does appear valid_ though on an asr1k running 03.16.06.S. Here is the setup.

Said network (44.151.210.0/23) is announced by AS206155 to a transit
operator (AS204092) on two peerings:
- one direct from as206155 (89.234.186.158 - router id 80.67.190.204)
to 204092 ("asbr01" - asr1k - 89.234.186.153)
- one indirect from as206155 (185.1.89.27) to 204092 ("asbr02" -
linux/bird - router id 89.234.186.34) though an IX RS.
- this only happens if peering with asbr02 goes up before asbr01,
otherwise asbr02 prefers the route through asbr01.

asbr01 and 02 from as204092 are using the same two validators, one
running routinator, the other is using FORT.

44.151.210.0/23 is not signed, nor is 44.128.0.0/10. Those prefixes do
not appear as valid in the rpki table:

asbr01#show ip bgp rpki table | i 44.151.210
asbr01#

However the /23 appears signed and hence is prefered:

asbr01#show ip bgp | be 44.151.21
N* 44.151.210.0/23 80.67.167.221 20 0 57199
34019 3215 206155 i
N* 193.200.43.85 10 0 34019
3215 206155 i
N* 89.234.186.158 50 200 0 206155 i
V*>i 185.1.89.27 150 150 0 206155
i <<< BEST & ROA VALID!?

asbr01#show ip bgp 44.151.210.0
BGP routing table entry for 44.151.210.0/23, version 487298116
BGP Bestpath: deterministic-med: med
Paths: (8 available, best #7, table default)
Advertised to update-groups:
75 114 118 122
<snip>
Refresh Epoch 1
206155
89.234.186.158 from 89.234.186.158 (80.67.190.204)
Origin IGP, metric 50, localpref 200, valid, external
Community: 64496:200
path 7FA510447288 RPKI State not found <<<< ok, expected.
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
206155, (Received from a RR-client)
185.1.89.27 (metric 11) from 89.234.186.34 (89.234.186.34)
Origin IGP, metric 150, localpref 150, valid, internal, best
Community: 64496:100 64496:2150
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0003 1D3C 0000 0064 0000 0096 0003 1D3C
0003 1D3C 0000 0064
path 7FA51044F508 RPKI State valid <<<<<<< ???
rx pathid: 0, tx pathid: 0x0

The bird on asbr02 do not show the prefix as roa valid:

bird> show route for 44.151.210.0 all
Table master4:
44.151.210.0/23 unicast [bgp_breizhix_ipv4 2020-04-30 from
185.1.89.1] * (100) [AS206155i]
via 185.1.89.27 on enp3s0f0.22
Type: BGP univ
igp_metric: 20
BGP.origin: IGP
BGP.as_path: 206155
BGP.next_hop: 185.1.89.27
BGP.local_pref: 150
BGP.community: (64496,100)
BGP.ext_community: (generic, 0x43000000, 0x1) <<< ROA not found
BGP.large_community: (204092, 100, 150)
unicast [bgp_cogent_ipv4 14:13:53.541] (100) [AS206155i]


I've captured the update from asbr02 to asbr01 and there isn't the
0x43 extended community. However after this first update asbr01
reflects it to the other ibgp peers _with_ the validation state 0x00!
The capture is available here: https://paste.swordarmor.fr/tHOf

I'm quite new at RPKI, so I might be missing something entirely, but
the Cisco behaviour looks wrong at best, if not dangerous, as this
makes unsigned prefixes look valid.

I've skimmed for known rpki bugs on XE and haven't found anything
conclusive, hence my attempt at having more eyeballs looking at this
:)

What's the list view on this issue?

thanks
pierre
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hi Pierre,

I think this is well known bug on XE.

We just had a thread week or so back on this list.

You need to enable extended community to carry the validation state as
otherwise XE considers IBGP learned paths by default as VALID.

I think Cisco is already backporting the fixes for this - but as long as
you enable ext community state propagation you will be fine.

Thx
R.

On Thu, May 7, 2020 at 10:07 PM Pierre Emeriaud <petrus.lt@gmail.com> wrote:

> Hello all
>
> First of all, sorry for the long text wall, but I thin I have a bit of
> an interesting issue here. I have a router that announces a prefix
> that is not RPKI signed at all, hence sould neither appear valid nor
> invalid.
>
> It _does appear valid_ though on an asr1k running 03.16.06.S. Here is the
> setup.
>
> Said network (44.151.210.0/23) is announced by AS206155 to a transit
> operator (AS204092) on two peerings:
> - one direct from as206155 (89.234.186.158 - router id 80.67.190.204)
> to 204092 ("asbr01" - asr1k - 89.234.186.153)
> - one indirect from as206155 (185.1.89.27) to 204092 ("asbr02" -
> linux/bird - router id 89.234.186.34) though an IX RS.
> - this only happens if peering with asbr02 goes up before asbr01,
> otherwise asbr02 prefers the route through asbr01.
>
> asbr01 and 02 from as204092 are using the same two validators, one
> running routinator, the other is using FORT.
>
> 44.151.210.0/23 is not signed, nor is 44.128.0.0/10. Those prefixes do
> not appear as valid in the rpki table:
>
> asbr01#show ip bgp rpki table | i 44.151.210
> asbr01#
>
> However the /23 appears signed and hence is prefered:
>
> asbr01#show ip bgp | be 44.151.21
> N* 44.151.210.0/23 80.67.167.221 20 0 57199
> 34019 3215 206155 i
> N* 193.200.43.85 10 0 34019
> 3215 206155 i
> N* 89.234.186.158 50 200 0 206155 i
> V*>i 185.1.89.27 150 150 0 206155
> i <<< BEST & ROA VALID!?
>
> asbr01#show ip bgp 44.151.210.0
> BGP routing table entry for 44.151.210.0/23, version 487298116
> BGP Bestpath: deterministic-med: med
> Paths: (8 available, best #7, table default)
> Advertised to update-groups:
> 75 114 118 122
> <snip>
> Refresh Epoch 1
> 206155
> 89.234.186.158 from 89.234.186.158 (80.67.190.204)
> Origin IGP, metric 50, localpref 200, valid, external
> Community: 64496:200
> path 7FA510447288 RPKI State not found <<<< ok, expected.
> rx pathid: 0, tx pathid: 0
> Refresh Epoch 1
> 206155, (Received from a RR-client)
> 185.1.89.27 (metric 11) from 89.234.186.34 (89.234.186.34)
> Origin IGP, metric 150, localpref 150, valid, internal, best
> Community: 64496:100 64496:2150
> unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
> value 0003 1D3C 0000 0064 0000 0096 0003 1D3C
> 0003 1D3C 0000 0064
> path 7FA51044F508 RPKI State valid <<<<<<< ???
> rx pathid: 0, tx pathid: 0x0
>
> The bird on asbr02 do not show the prefix as roa valid:
>
> bird> show route for 44.151.210.0 all
> Table master4:
> 44.151.210.0/23 unicast [bgp_breizhix_ipv4 2020-04-30 from
> 185.1.89.1] * (100) [AS206155i]
> via 185.1.89.27 on enp3s0f0.22
> Type: BGP univ
> igp_metric: 20
> BGP.origin: IGP
> BGP.as_path: 206155
> BGP.next_hop: 185.1.89.27
> BGP.local_pref: 150
> BGP.community: (64496,100)
> BGP.ext_community: (generic, 0x43000000, 0x1) <<< ROA not found
> BGP.large_community: (204092, 100, 150)
> unicast [bgp_cogent_ipv4 14:13:53.541] (100)
> [AS206155i]
>
>
> I've captured the update from asbr02 to asbr01 and there isn't the
> 0x43 extended community. However after this first update asbr01
> reflects it to the other ibgp peers _with_ the validation state 0x00!
> The capture is available here: https://paste.swordarmor.fr/tHOf
>
> I'm quite new at RPKI, so I might be missing something entirely, but
> the Cisco behaviour looks wrong at best, if not dangerous, as this
> makes unsigned prefixes look valid.
>
> I've skimmed for known rpki bugs on XE and haven't found anything
> conclusive, hence my attempt at having more eyeballs looking at this
> :)
>
> What's the list view on this issue?
>
> thanks
> pierre
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hi Pierre,

This reminds me of a case of my own while labbing RPKI on XE. Only eBGP routes are subject to RPKI validation. iBGP routes are automatically considered to be valid. Cisco's implementation in XE will automatically modify the best path selection to prefer valid over unknown over invalid very high in the selection ruleset. This is what I assume happens :

If asbr02 goes up first, it gets the prefix, considers it a best path, sends it to asbr01 via iBGP. Then asbr01 goes up, compares an unknown external path to a valid internal one and chooses the second. Thus, traffic flows through there.

If asbr01 goes up first, it gets the prefix from its external neighbor, considers it best, sends it to asbr02. Asbr02 comes up but I'm guessing BIRD is actually preferring the route from asbr01. Thus, it never sends its own external route to asbr01. So, asbr01 keeps preferring its own external unknown one.

If I understand your design correctly, you might want to research whether BIRD can signal RPKI state via iBGP, as this would cause eventual consistency.


Regarding the extcommunity, I'm not sure if it's the best of ideas to announce state on iBGP routes, let alone reflected ones. I'd have to check whether the RFC actually specifies this before I form an opinion on what's happening. Assuming you do have asbr01 configured to announce rpki state though, it could be the expected behavior.



My thoughts and words are my own.

Spyros



-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces@puck.nether.net> On Behalf Of Pierre Emeriaud
Sent: Thursday, May 7, 2020 11:03 PM
To: cisco-nsp@puck.nether.net
Subject: [c-nsp] RPKI validation weirdness

Hello all

First of all, sorry for the long text wall, but I thin I have a bit of an interesting issue here. I have a router that announces a prefix that is not RPKI signed at all, hence sould neither appear valid nor invalid.

It _does appear valid_ though on an asr1k running 03.16.06.S. Here is the setup.

Said network (44.151.210.0/23) is announced by AS206155 to a transit operator (AS204092) on two peerings:
- one direct from as206155 (89.234.186.158 - router id 80.67.190.204) to 204092 ("asbr01" - asr1k - 89.234.186.153)
- one indirect from as206155 (185.1.89.27) to 204092 ("asbr02" - linux/bird - router id 89.234.186.34) though an IX RS.
- this only happens if peering with asbr02 goes up before asbr01, otherwise asbr02 prefers the route through asbr01.

asbr01 and 02 from as204092 are using the same two validators, one running routinator, the other is using FORT.

44.151.210.0/23 is not signed, nor is 44.128.0.0/10. Those prefixes do not appear as valid in the rpki table:

asbr01#show ip bgp rpki table | i 44.151.210 asbr01#

However the /23 appears signed and hence is prefered:

asbr01#show ip bgp | be 44.151.21
N* 44.151.210.0/23 80.67.167.221 20 0 57199
34019 3215 206155 i
N* 193.200.43.85 10 0 34019
3215 206155 i
N* 89.234.186.158 50 200 0 206155 i
V*>i 185.1.89.27 150 150 0 206155
i <<< BEST & ROA VALID!?

asbr01#show ip bgp 44.151.210.0
BGP routing table entry for 44.151.210.0/23, version 487298116 BGP Bestpath: deterministic-med: med
Paths: (8 available, best #7, table default)
Advertised to update-groups:
75 114 118 122
<snip>
Refresh Epoch 1
206155
89.234.186.158 from 89.234.186.158 (80.67.190.204)
Origin IGP, metric 50, localpref 200, valid, external
Community: 64496:200
path 7FA510447288 RPKI State not found <<<< ok, expected.
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
206155, (Received from a RR-client)
185.1.89.27 (metric 11) from 89.234.186.34 (89.234.186.34)
Origin IGP, metric 150, localpref 150, valid, internal, best
Community: 64496:100 64496:2150
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0003 1D3C 0000 0064 0000 0096 0003 1D3C
0003 1D3C 0000 0064
path 7FA51044F508 RPKI State valid <<<<<<< ???
rx pathid: 0, tx pathid: 0x0

The bird on asbr02 do not show the prefix as roa valid:

bird> show route for 44.151.210.0 all
Table master4:
44.151.210.0/23 unicast [bgp_breizhix_ipv4 2020-04-30 from
185.1.89.1] * (100) [AS206155i]
via 185.1.89.27 on enp3s0f0.22
Type: BGP univ
igp_metric: 20
BGP.origin: IGP
BGP.as_path: 206155
BGP.next_hop: 185.1.89.27
BGP.local_pref: 150
BGP.community: (64496,100)
BGP.ext_community: (generic, 0x43000000, 0x1) <<< ROA not found
BGP.large_community: (204092, 100, 150)
unicast [bgp_cogent_ipv4 14:13:53.541] (100) [AS206155i]


I've captured the update from asbr02 to asbr01 and there isn't the
0x43 extended community. However after this first update asbr01 reflects it to the other ibgp peers _with_ the validation state 0x00!
The capture is available here: https://paste.swordarmor.fr/tHOf

I'm quite new at RPKI, so I might be missing something entirely, but the Cisco behaviour looks wrong at best, if not dangerous, as this makes unsigned prefixes look valid.

I've skimmed for known rpki bugs on XE and haven't found anything conclusive, hence my attempt at having more eyeballs looking at this
:)

What's the list view on this issue?

thanks
pierre
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Le jeu. 7 mai 2020 à 22:55, Robert Raszuk <robert@raszuk.net> a écrit :
>
> Hi Pierre,
>
> I think this is well known bug on XE.

Great! Well, that this is known. not the bug...

> We just had a thread week or so back on this list.
woops my bad, missed it.

> You need to enable extended community to carry the validation state as otherwise XE considers IBGP learned paths by default as VALID.
From what I've seen in the capture this should be already the case. XE
do sends the validation state, but do not receive one from bird.

thanks Raszuk.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Le jeu. 7 mai 2020 à 22:56, Spyros Kakaroukas
<s.kakaroukas@connecticore.com> a écrit :
>
> Hi Pierre,
>
> This reminds me of a case of my own while labbing RPKI on XE. Only eBGP routes are subject to RPKI validation. iBGP routes are automatically considered to be valid. Cisco's implementation in XE will automatically modify the best path selection to prefer valid over unknown over invalid very high in the selection ruleset. This is what I assume happens :
>
> If asbr02 goes up first, it gets the prefix, considers it a best path, sends it to asbr01 via iBGP. Then asbr01 goes up, compares an unknown external path to a valid internal one and chooses the second. Thus, traffic flows through there.
>
> If asbr01 goes up first, it gets the prefix from its external neighbor, considers it best, sends it to asbr02. Asbr02 comes up but I'm guessing BIRD is actually preferring the route from asbr01. Thus, it never sends its own external route to asbr01. So, asbr01 keeps preferring its own external unknown one.

This is exactly what's happening. But why did Cisco rpki algorithm
chose to trust ibgp relationship over the validators, even though
extcommunity wasn't sent, this is weird...

> If I understand your design correctly, you might want to research whether BIRD can signal RPKI state via iBGP, as this would cause eventual consistency.
Yes, my fellow netadmin Alarig at as204092 just asked on bird's
mailing list why didn't bird sent the extcommunity.
https://bird.network.cz/pipermail/bird-users/2020-May/014559.html for
the interested.

> Regarding the extcommunity, I'm not sure if it's the best of ideas to announce state on iBGP routes, let alone reflected ones. I'd have to check whether the RFC actually specifies this before I form an opinion on what's happening. Assuming you do have asbr01 configured to announce rpki state though, it could be the expected behavior.

While I can grasp why one could announce (and trust) rpki state over
ibgp, in this situation the asr1k had both a validator and no
extcommunity whatsoever received, this I don't understand why it would
validate such a prefix...

Anyhow, thanks a lot for the debugging Spyros. I'll follow up with the
bird folks on this matter.

regards
pierre
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 7/May/20 22:02, Pierre Emeriaud wrote:

>
> What's the list view on this issue?

Known stupidity from Cisco with IOS and IOS XE... all iBGP routes are
automatically marked as Valid.

Stupid!

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 7/May/20 22:55, Robert Raszuk wrote:

> Hi Pierre,
>
> I think this is well known bug on XE.

In Cisco-land, this is a feature, not a bug.

That said, there are some warm bodies on the case.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 7/May/20 23:29, Pierre Emeriaud wrote:

> This is exactly what's happening. But why did Cisco rpki algorithm
> chose to trust ibgp relationship over the validators, even though
> extcommunity wasn't sent, this is weird...

I spent a whole week in 2014 trying to figure out why Cisco would think
this is useful, despite the RFC's saying "Don't do such". I gave up and
focused on better-written implementations.


> While I can grasp why one could announce (and trust) rpki state over
> ibgp, in this situation the asr1k had both a validator and no
> extcommunity whatsoever received, this I don't understand why it would
> validate such a prefix...

Stupid. Don't bang your head against a wall trying to figure out how
Cisco reached this conclusion in their interpretation and implementation
of the RFC.

And what's even more annoying - IOS XR is well implemented, i.e., it
does not have this stupidity. Makes you wonder.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hey Mark,

Well this is not a feature (even in Cisco land - it is a design bug). Just
a few days ago I had a phone chat with a person who implemented origin
validation in Cisco and he actually confirmed.

But your point is very valid. Unfortunately, when you hire excellent
developers who develop features without ever getting full picture on
what it is that they are actually developing and without deep understanding
how those things are going to work in the real world you will like to see
such fruits. And frankly this goes way beyond just cisco these days ...

Best,
Robert.

On Fri, May 8, 2020 at 1:13 AM Mark Tinka <mark.tinka@seacom.mu> wrote:

>
>
> On 7/May/20 22:55, Robert Raszuk wrote:
>
> > Hi Pierre,
> >
> > I think this is well known bug on XE.
>
> In Cisco-land, this is a feature, not a bug.
>
> That said, there are some warm bodies on the case.
>
> Mark.
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 01:25, Robert Raszuk wrote:

> Well this is not a feature (even in Cisco land - it is a design bug).
> Just a few days ago I had a phone chat with a person who implemented
> origin validation in Cisco and he actually confirmed.

It's all mangled, but specifically, the part that Cisco interpreted as a
feature (which is actually a bug against the RFC) is automatic
application of policy based on RPKI state, which then forms the initial
stages of the BGP best path algorithm. It is even officially documented
in the manual at:

   
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-3s/irg-xe-3s-book/bgp-origin-as-validation.html


Use of the Validation State in BGP Best Path Determination

There are two ways you can modify the default BGP best path selection
process when using RPKI validation states:

*

You can completely disable the validation of prefixes by the RPKI
server and the storage of that validation information. This is done
by configuring the bgp bestpath prefix-validate disable command. You
might want to do this for configuration testing. The router will
still connect to the RPKI server and download the validation
information, but will not use the information.

*

You can allow an invalid prefix to be used as the BGP best path,
even if valid prefixes are available. This is the default behavior.
The command to allow a BGP best path to be an invalid prefix, as
determined by the BGP Origin AS Validation feature, is the bgp
bestpath prefix-validate allow-invalid command. The prefix
validation state will still be assigned to paths, and will still be
communicated to iBGP neighbors that have been configured to receive
RPKI state information. You can use a route map to set a local
preference, metric, or other property based on the validation state.

During BGP best path selection, the default behavior, if neither of the
above options is configured, is that the system will prefer prefixes in
the following order:

*

Those with a validation state of valid.

*

Those with a validation state of not found.

*

Those with a validation state of invalid (which, by default, will
not be installed in the routing table).

These preferences override metric, local preference, and other choices
made during the bestpath computation. The standard bestpath decision
tree applies only if the validation state of the two paths is the same.

If both commands are configured, the bgp bestpath prefix-validate
disable command will prevent the validation state from being assigned to
paths, so the bgp bestpath prefix-validate allow-invalid command will
have no effect.

These configurations can be in either router configuration mode or in
address family configuration mode for the IPv4 unicast or IPv6 unicast
address families.



>
> But your point is very valid. Unfortunately, when you hire excellent
> developers who develop features without ever getting full picture on
> what it is that they are actually developing and without deep
> understanding how those things are going to work in the real world you
> will like to see such fruits. And frankly this goes way beyond just
> cisco these days ...

Which we can all appreciate. But we brought this to Cisco's attention
all the way back in 2016/2017. Yes, it's getting fixed now, but we
weren't asking to make EIGRP an inter-domain routing protocol.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hi,

On Thu 07 May 2020 23:29:03 GMT, Pierre Emeriaud wrote:
> Yes, my fellow netadmin Alarig at as204092 just asked on bird's
> mailing list why didn't bird sent the extcommunity.
> https://bird.network.cz/pipermail/bird-users/2020-May/014559.html for
> the interested.

I had a patch to test and it works, now the community is well reflected.
Refresh Epoch 1
206155
89.234.186.158 from 89.234.186.158 (80.67.190.204)
Origin IGP, metric 50, localpref 200, valid, external, best
Community: 64496:200
path 7FA510447288 RPKI State not found
rx pathid: 0, tx pathid: 0x0

In case of curiosity, this is the ebuild modification I had to make:
https://git.grifon.fr/alarig/SwordArMor-gentoo-overlay/commit/a4367e648124f151381ac18c555e4d4caa02d947

So we were facing a double bug here: bird not sending the community and
XE bugging as expected.

Thanks for your help!

Regards,
--
Alarig
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 11:23, Robert Raszuk wrote:

>
> But in what you pasted in I do not see statement that paths received
> over IBGP with no state information should be considered VALID. I
> think they just thought North - South then if you validate on ingress
> rest of the domain should assume the IBGP path received is VALID.

Yes, agreed on that. The e-mail I got from someone back in 2016 on that
issue was "this is designed as per normal".

The link I sent was specific to them documenting a mis-interpretation of
the RFC on policy application, despite being called out on it years ago.


> With that they apparently  never thought about iBGP also going
> East-West between ASBRs or via RRs where you are using add-paths or
> best external :)

We just disabled RPKI on our ASR1000 edge routers, especially since they
only handle about 2 customers across the entire backbone. Much of that
heavy-lifting is done by our MX480's, which implement ROV correctly.

One of the issues we hit that led us to this decision was when you learn
a non-ROA'd customer route via eBGP in one location, but that specific
edge router decides to send the traffic back to them via iBGP to another
site where they may have a second connection to you or via a peering
partner, because on the local edge router, Cisco automatically applies
NotFound for the route, but Valid for the iBGP-learned route. Since
Cisco will automatically apply policy by default the moment you enable
RPKI, the local eBGP route will never be chosen as the best path, even
though it is.

Utter madness!

Mark.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Yes your story how you found it is exactly as mine - with just one
exception that I was looking at my upstream ISPs across two ASBRs and
debugging what is going on :) But the core of the issue was precisely
identical !

Well I am still running it .. just enabled ext comms.

Frankly this was ugly as quite unexpected - but relatively easy to see what
is going on. In this space I am much more worried about RPKI db accuracy
then any of the implementation issues. I found number of cases so far where
what is in RPKI is just plain wrong - read INVALID :). And that is supposed
to be the source of truth ...

Now I am considering actually automation of detection of RPKI bad info and
some sort of publishing it.

See when you sign a block then sell this block without removing your RPKI
signature, then the block gets cutted into chunks and sold further - and no
one in this process of transaction chain cares about RPKI - this entire
story of using this for validation becomes pretty weak. And this is no
longer NOT-FOUND. You get false INVALIDs which some may apply to suppress
or drop.

Best,
R.


On Fri, May 8, 2020 at 11:32 AM Mark Tinka <mark.tinka@seacom.mu> wrote:

>
>
> On 8/May/20 11:23, Robert Raszuk wrote:
>
> >
> > But in what you pasted in I do not see statement that paths received
> > over IBGP with no state information should be considered VALID. I
> > think they just thought North - South then if you validate on ingress
> > rest of the domain should assume the IBGP path received is VALID.
>
> Yes, agreed on that. The e-mail I got from someone back in 2016 on that
> issue was "this is designed as per normal".
>
> The link I sent was specific to them documenting a mis-interpretation of
> the RFC on policy application, despite being called out on it years ago.
>
>
> > With that they apparently never thought about iBGP also going
> > East-West between ASBRs or via RRs where you are using add-paths or
> > best external :)
>
> We just disabled RPKI on our ASR1000 edge routers, especially since they
> only handle about 2 customers across the entire backbone. Much of that
> heavy-lifting is done by our MX480's, which implement ROV correctly.
>
> One of the issues we hit that led us to this decision was when you learn
> a non-ROA'd customer route via eBGP in one location, but that specific
> edge router decides to send the traffic back to them via iBGP to another
> site where they may have a second connection to you or via a peering
> partner, because on the local edge router, Cisco automatically applies
> NotFound for the route, but Valid for the iBGP-learned route. Since
> Cisco will automatically apply policy by default the moment you enable
> RPKI, the local eBGP route will never be chosen as the best path, even
> though it is.
>
> Utter madness!
>
> Mark.
>
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hello Robert,

On Fri, 8 May 2020 at 11:42, Robert Raszuk <robert@raszuk.net> wrote:
> See when you sign a block then sell this block without removing your RPKI
> signature, then the block gets cutted into chunks and sold further - and no
> one in this process of transaction chain cares about RPKI - this entire
> story of using this for validation becomes pretty weak. And this is no
> longer NOT-FOUND. You get false INVALIDs which some may apply to suppress
> or drop.

Well it's the IRR's job to get this right, and update RPKI data and/or
remove obsolete delegations. Just like with reverse-DNS objects.

It's not like when you are buying a new block, you can't use reverse
DNS on those new IPs. And RPKI needs to be updated just the same, by
the IRR.

I'd assume some IRR's are better than others when it comes to handling
those things.


Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 11:42, Robert Raszuk wrote:

> Frankly this was ugly as quite unexpected - but relatively easy to see
> what is going on. In this space I am much more worried about RPKI db
> accuracy then any of the implementation issues. I found number of
> cases so far where what is in RPKI is just plain wrong - read INVALID
> :). And that is supposed to be the source of truth ...

Validator issue, perhaps?

We saw some discrepancies between FORT and Routinator back in February
in Melbourne. Possible those have now been fixed, though.


>
> Now I am considering actually automation of detection of RPKI bad info
> and some sort of publishing it. 
>
> See when you sign a block then sell this block without removing your
> RPKI signature, then the block gets cutted into chunks and sold
> further - and no one in this process of transaction chain cares about
> RPKI - this entire story of using this for validation becomes pretty
> weak. And this is no longer NOT-FOUND. You get false INVALIDs which
> some may apply to suppress or drop.

One of the reasons you should now enable automatic ROA certification of
longer prefixes, if you only really meant to sign the parent.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On Fri 08 May 2020 11:42:51 GMT, Robert Raszuk wrote:
> See when you sign a block then sell this block without removing your RPKI
> signature, then the block gets cutted into chunks and sold further - and no
> one in this process of transaction chain cares about RPKI - this entire
> story of using this for validation becomes pretty weak. And this is no
> longer NOT-FOUND. You get false INVALIDs which some may apply to suppress
> or drop.

Well… if your LIR isn’t careful enough to take care of RPKI, then change
your LIR. And if the customer isn’t careful enough to verify the RPKI
state of its prefixes, some bad things will happen, one day or an other.
And this may not necessary involve RPKI.

--
Alarig
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Lukas,

True. But I am actually not sure why RPKI state could not just expire by
itself say every 12 months unless renewed by the owner ? Just like DNS name
fee :)

Thx,
R.



On Fri, May 8, 2020 at 12:02 PM Lukas Tribus <lists@ltri.eu> wrote:

> Hello Robert,
>
> On Fri, 8 May 2020 at 11:42, Robert Raszuk <robert@raszuk.net> wrote:
> > See when you sign a block then sell this block without removing your RPKI
> > signature, then the block gets cutted into chunks and sold further - and
> no
> > one in this process of transaction chain cares about RPKI - this entire
> > story of using this for validation becomes pretty weak. And this is no
> > longer NOT-FOUND. You get false INVALIDs which some may apply to suppress
> > or drop.
>
> Well it's the IRR's job to get this right, and update RPKI data and/or
> remove obsolete delegations. Just like with reverse-DNS objects.
>
> It's not like when you are buying a new block, you can't use reverse
> DNS on those new IPs. And RPKI needs to be updated just the same, by
> the IRR.
>
> I'd assume some IRR's are better than others when it comes to handling
> those things.
>
>
> Lukas
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 12:05, Mark Tinka wrote:

>
> One of the reasons you should now enable automatic ROA certification of
> longer prefixes, if you only really meant to sign the parent.

*not enable, I meant to say.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 12:06, Alarig Le Lay wrote:

> Well… if your LIR isn’t careful enough to take care of RPKI, then change
> your LIR. And if the customer isn’t careful enough to verify the RPKI
> state of its prefixes, some bad things will happen, one day or an other.
> And this may not necessary involve RPKI.

In our experience so far, outages caused by RPKI mistakes have been
fixed inside of 24hrs, even when involving operators two rounds around
the earth.

Let's hope it stays that way.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: RPKI validation weirdness [ In reply to ]
Hi,

On Fri, May 08, 2020 at 11:42:51AM +0200, Robert Raszuk wrote:
> See when you sign a block then sell this block without removing your RPKI
> signature, then the block gets cutted into chunks and sold further - and no
> one in this process of transaction chain cares about RPKI - this entire
> story of using this for validation becomes pretty weak. And this is no
> longer NOT-FOUND. You get false INVALIDs which some may apply to suppress
> or drop.

This is why you use a broker who understands their trade to facilitate
the transfer. Ensure that such details are not overlooked.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 12:24, Gert Doering wrote:

>
> This is why you use a broker who understands their trade to facilitate
> the transfer. Ensure that such details are not overlooked.

What are the chances these brokers know about IPv6? And that's just the
basic requirement to graduate to RPKI, and DNSSEC :-).

Mark.
Re: RPKI validation weirdness [ In reply to ]
Hi,

On Fri, May 08, 2020 at 12:44:14PM +0200, Mark Tinka wrote:
> On 8/May/20 12:24, Gert Doering wrote:
>
> > This is why you use a broker who understands their trade to facilitate
> > the transfer. Ensure that such details are not overlooked.
>
> What are the chances these brokers know about IPv6? And that's just the
> basic requirement to graduate to RPKI, and DNSSEC :-).

If a broker doesn't know about that, I wouldn't use them.

(I'm a bit picky anyway, because I tend to only trust people I've met
and worked with for a while, and since there are two brokers among
them that well understand RIR policies and all that - these are the ones
I'd use and recommend :-) )

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: RPKI validation weirdness [ In reply to ]
On 8/May/20 12:48, Gert Doering wrote:

> If a broker doesn't know about that, I wouldn't use them.
>
> (I'm a bit picky anyway, because I tend to only trust people I've met
> and worked with for a while, and since there are two brokers among
> them that well understand RIR policies and all that - these are the ones
> I'd use and recommend :-) )

Guys, you heard the man :-).

Gert, a small commission for your referral services would not be totally
out of order, hehe.

Mark.