Mailing List Archive

On 04/11/2021 18:26, Robert Hass wrote:
> What IPv4 FIB scale I can expect from ASR9001 ?
> BGP table is growing and I want to predict how much lifespan those
> boxes still have.

It's Typhoon (2nd Gen) so 4M v4, or 2M v6.

Assuming you don't use any of it for anything else, e.g. labels. You're
likely to run into other limits first (PPS, RAM, cXR discontinued...)

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/4/21 21:40, Tom Hill wrote:

> It's Typhoon (2nd Gen) so 4M v4, or 2M v6.
>
> Assuming you don't use any of it for anything else, e.g. labels. You're
> likely to run into other limits first (PPS, RAM, cXR discontinued...)

As Tom said.

We are retiring ours, because CPU performance for just 2x IPv4 + 2x IPv6
full sessions is too much for the Freescale PPC CPU.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

We have a couple of these in production also are you planning on going with the 9902 or 9903 or are you switching altogether?

I would love it if they could just make an ASR that has 4xQSFP28 ports and a slot to add 4-6 more.

Would be perfect for us assuming that it wasn't $80,000

Thanks,
-Drew


-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces@puck.nether.net> On Behalf Of Mark Tinka
Sent: Friday, November 5, 2021 3:54 AM
To: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] FIB scale on ASR9001



On 11/4/21 21:40, Tom Hill wrote:

> It's Typhoon (2nd Gen) so 4M v4, or 2M v6.
>
> Assuming you don't use any of it for anything else, e.g. labels.
> You're likely to run into other limits first (PPS, RAM, cXR
> discontinued...)

As Tom said.

We are retiring ours, because CPU performance for just 2x IPv4 + 2x IPv6 full sessions is too much for the Freescale PPC CPU.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_cisco-2Dnsp&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=C7wKxKAk_X4Dj694Z1yw_7Xb6Ikb6geQAHieGuLY3m8&s=OT9np_a5fIrYimJQHaBdBYxjCv4XflIOQlPlsp7f2kU&e=
archive at https://urldefense.proofpoint.com/v2/url?u=http-3A__puck.nether.net_pipermail_cisco-2Dnsp_&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=C7wKxKAk_X4Dj694Z1yw_7Xb6Ikb6geQAHieGuLY3m8&s=LG52djHtxAe9ZmsIoF5Ns724PGtiOn4hiPWec0O_4pk&e=
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Tue, Nov 09, 2021 at 04:49:54PM +0000, Drew Weaver wrote:
> We have a couple of these in production also are you planning on going with the 9902 or 9903 or are you switching altogether?
>
> I would love it if they could just make an ASR that has 4xQSFP28 ports and a slot to add 4-6 more.
>
> Would be perfect for us assuming that it wasn't $80,000

Looking at the prices for 9901/02/03/04, and the licensing models for
NCS5700, I can assure you, "it won't be $80,000".

More like "$100k for the Chassis, and then another $300k for all required
licenses, to be renewed every 3 years" (subject to bazaar style discount
negotiations that make any sort of budget planning impossible)...

Cisco is a textbook example how to drive away a truly loyal user base,
and then blaim it on stock market analysts ("they said that any company
without a recurring revenue software model will be dead soon").

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

>
> More like "$100k for the Chassis, and then another $300k for all required
> licenses, to be renewed every 3 years" (subject to bazaar style discount
> negotiations that make any sort of budget planning impossible)...
>
>
Ahh the bazaar style negotiations. I have flashbacks to Marrakesh and
everybody saying "Welcome my friend!" and "Good deal for you!".
Did I *really* just get a good deal on licenses or did I just get taken
advantage of? When I have to ask, it's usually the latter.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Tue, 9 Nov 2021 at 19:13, Gert Doering <gert@greenie.muc.de> wrote:

> Cisco is a textbook example how to drive away a truly loyal user base,
> and then blaim it on stock market analysts ("they said that any company
> without a recurring revenue software model will be dead soon").

Ranting and raving follows.

All (smart) executives claim the upside is because of their leadership
and downside is because of the market. While no data supports that
replacing executive A with executive B improved or reduced company
performance, that is, we don't know what qualities make companies fail
or succeed.

But I admire the beauty of something like this: https://exainfra.net/about-us/
'Under Xs’ leadership, GTT bought over 40 companies and grew annual
revenues from $65 million to over $1.6 billion during his tenure'

You have <150 words to highlight your most important achievements in
your career. And you choose to focus on the time when not only
shareholders but many classes of debt holders got completely wiped due
to over-extending.

In most other cases you just can't do that. Crane operator can't brag
about that one time when his mistake caused the building to collapse,
in fact he'll struggle to get hired by anyone aware of it. But
management has no metrics, so you are as competent and valuable as you
confidently say you are (which is why being tall helps being a
successful manager, as it's a metric we are able to compare easily and
being tall means to us being more competent).

Having said that, 5y performance:
SP500: 110%
CSCO: 90%
NOK: 20%
JNPR: 10%
PANW: 300%
ANET: 450%

So Cisco is losing to the wide market only very little, and is
outperforming other SP vendors (Huawei excluded). So the market
doesn't entirely agree with your assertion of user base attrition.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/9/21 18:49, Drew Weaver wrote:

> We have a couple of these in production also are you planning on going with the 9902 or 9903 or are you switching altogether?

We moved to the MX204.

Not really that interested in Cisco anymore.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/9/21 19:04, Gert Doering wrote:

> Looking at the prices for 9901/02/03/04, and the licensing models for
> NCS5700, I can assure you, "it won't be $80,000".
>
> More like "$100k for the Chassis, and then another $300k for all required
> licenses, to be renewed every 3 years" (subject to bazaar style discount
> negotiations that make any sort of budget planning impossible)...
>
> Cisco is a textbook example how to drive away a truly loyal user base,
> and then blaim it on stock market analysts ("they said that any company
> without a recurring revenue software model will be dead soon").

Gert nailed it.

This is one of the (many) reasons we aren't interested in Cisco anymore.

We are phasing out all the kit we have with them. We will, however, keep
the CSR1000v, as we like that, and can support ourselves on it.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/10/21 08:48, Saku Ytti wrote:

> Ranting and raving follows.
>
> All (smart) executives claim the upside is because of their leadership
> and downside is because of the market. While no data supports that
> replacing executive A with executive B improved or reduced company
> performance, that is, we don't know what qualities make companies fail
> or succeed.
>
> But I admire the beauty of something like this: https://exainfra.net/about-us/
> 'Under Xs’ leadership, GTT bought over 40 companies and grew annual
> revenues from $65 million to over $1.6 billion during his tenure'

I call this the "bottom line factor". People do not care about details
anymore, both on the consumer side, and on the provider side. They just
bottom line their experience based on who was in charge.

Odd, then, that Steve Jobs had success, failure and success, at Apple,
with the last bout totally blowing the whole game out the water. After
all, if Apple nearly collapsed under him, then the metrics suggest he'd
have no chance of reviving it.

But...


> Having said that, 5y performance:
> SP500: 110%
> CSCO: 90%
> NOK: 20%
> JNPR: 10%
> PANW: 300%
> ANET: 450%
>
> So Cisco is losing to the wide market only very little, and is
> outperforming other SP vendors (Huawei excluded). So the market
> doesn't entirely agree with your assertion of user base attrition.

... I think the question was about loyal customers. Cisco know how to
drum up new business, and they know how to increase cash output from
existing business. But none of this suggests that this business is loyal.

There does come a time where you can't keep adding new business, to keep
showing growth. Unless, of course, you buy everything up and grow that
way, which we know Cisco love to do. But even that has a shelf life.
Dodge it as you may, but eventually, you need to provide value to the
paying customer base, and that always catches up with you.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 05/11/2021 07:53, Mark Tinka wrote:
> We are retiring ours, because CPU performance for just 2x IPv4 + 2x IPv6
> full sessions is too much for the Freescale PPC CPU.

I may live to regret asking this, but...

I've run a lot more than that on a 9001, and it handled it all with
aplomb. They're not as fast as an RSP440 (or RSP880) but in no way did I
find them to be liabilities when running alongside measurably faster
routers in the same ASN. They were extremely competent, in fact.

So how are you measuring this, and/or how is the issue manifesting?

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

I'll second Tom on this one. I actually have 2 9001S routers (the S is the
port / memory limited license). They have 2 full feeds (V4 and V6) and
some EIGRP, and that's about it. Currently handling things fine. Though
only doing ~ 5gig in/out at this point.

Shawn

On Wed, Nov 10, 2021 at 1:04 PM Tom Hill <tom@ninjabadger.net> wrote:

> On 05/11/2021 07:53, Mark Tinka wrote:
> > We are retiring ours, because CPU performance for just 2x IPv4 + 2x IPv6
> > full sessions is too much for the Freescale PPC CPU.
>
> I may live to regret asking this, but...
>
> I've run a lot more than that on a 9001, and it handled it all with
> aplomb. They're not as fast as an RSP440 (or RSP880) but in no way did I
> find them to be liabilities when running alongside measurably faster
> routers in the same ASN. They were extremely competent, in fact.
>
> So how are you measuring this, and/or how is the issue manifesting?
>
> --
> Tom
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/10/21 20:00, Tom Hill wrote:

> I may live to regret asking this, but...
> I've run a lot more than that on a 9001, and it handled it all with
> aplomb. They're not as fast as an RSP440 (or RSP880) but in no way did I
> find them to be liabilities when running alongside measurably faster
> routers in the same ASN. They were extremely competent, in fact.
>
> So how are you measuring this, and/or how is the issue manifesting?

At various peering locations where the ASR9001 had been running for some
years, it was starting to struggle to manage several hundred sessions
(none of which were a full table). The router kept sending too many
Refresh notices to neighbors, across a number of versions of IOS XR
code. Cisco couldn't figure it out, and we kept losing sessions due to
the nuisance we'd become across the exchange fabric. What was clear was
that the issue kept growing as peers increased routes, or as we added
neighbors.

In applications where we had just 2x full BGP sessions, IS-IS would hang
and blackhole traffic. The issue was hard to reproduce, but every time
it happened, we knew that rebooting the router was the only solution. We
still see this issue in the few nodes we have running. The good news it
does not happen often, but also, it shouldn't.

Is the ASR9001 CPU as slow as the MX80? I would say no, but considering
it's the only non-x86 box we have, the performance delta of the control
and management plane is visible.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/10/21 20:47, Shawn L wrote:

> I'll second Tom on this one. I actually have 2 9001S routers (the S is the
> port / memory limited license). They have 2 full feeds (V4 and V6) and
> some EIGRP, and that's about it. Currently handling things fine. Though
> only doing ~ 5gig in/out at this point.

We have nothing against the forwarding performance of the ASR9001. It's
the control/management plane that seems to be slowing down (at least for
us, anyway) with newer code and a growing Internet DFZ.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Thu, Nov 11, 2021 at 08:27:44AM +0200, Mark Tinka wrote:
> We have nothing against the forwarding performance of the ASR9001. It's
> the control/management plane that seems to be slowing down (at least for
> us, anyway) with newer code and a growing Internet DFZ.

"newer code" might be the key issue here - what version are you on?

Our 9001 have been extremely well-behaved in regards to BGP performance
and robustness. Not as fast as the RSP440, but still well up to the
job.

We're on 6.5.3 ("nothing interesting in newer versions, and still
supported").

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

On 11/11/21 09:18, Gert Doering wrote:

> "newer code" might be the key issue here - what version are you on?
>
> Our 9001 have been extremely well-behaved in regards to BGP performance
> and robustness. Not as fast as the RSP440, but still well up to the
> job.
>
> We're on 6.5.3 ("nothing interesting in newer versions, and still
> supported").

We are currently running 6.7.1, mainly to fix some LDPv6 issues.

However, we started seeing this "too many BGP refresh messages" issue as
far back as 6.4.2.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 09:38, Mark Tinka <mark@tinka.africa> wrote:

> We are currently running 6.7.1, mainly to fix some LDPv6 issues.
>
> However, we started seeing this "too many BGP refresh messages" issue as
> far back as 6.4.2.

Did you run RPKI? Did you have softinbound disabled?

This would cause a refresh on every RTR update. Basically a
misconfiguration, if you run RPKI you must keep policy rejected
routes.
--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 09:42, Saku Ytti wrote:

>
> Did you run RPKI?

Yes.


> Did you have softinbound disabled?

Yes.


> This would cause a refresh on every RTR update. Basically a
> misconfiguration, if you run RPKI you must keep policy rejected
> routes.

What's the thought-process here?

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 09:50, Mark Tinka <mark@tinka.africa> wrote:

> What's the thought-process here?

When you receive an RTR update, you change your ingress policy, and
you don't know what is the correct result without re-evaluating.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Thu, Nov 11, 2021 at 09:26:12AM +0200, Mark Tinka wrote:
> We are currently running 6.7.1, mainly to fix some LDPv6 issues.
>
> However, we started seeing this "too many BGP refresh messages" issue as
> far back as 6.4.2.

I seem to remember it's related to RPKI ROV (every time the RPKI server
sends new data the BGP implementation asks for a refresh to re-validate
the incoming feed).

Or at least there is something in the back of my head that says "I have
heard someone talk about this unintended side-effect of RPKI ROV, together
with a BGP setup without 'soft in always'".

Might there be a correlation in your environment?

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

On 11/11/21 09:51, Saku Ytti wrote:

> When you receive an RTR update, you change your ingress policy, and
> you don't know what is the correct result without re-evaluating.

I'm with you now - makes sense.

And Junos defaults to always maintaining a copy of Adj-RIB-In, which is
why we don't have that issue there, non?

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 10:08, Mark Tinka <mark@tinka.africa> wrote:

> And Junos defaults to always maintaining a copy of Adj-RIB-In, which is
> why we don't have that issue there, non?

Correct. Add 'keep none' to junos, and you'll have the same issue.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 09:51, Gert Doering wrote:

> I seem to remember it's related to RPKI ROV (every time the RPKI server
> sends new data the BGP implementation asks for a refresh to re-validate
> the incoming feed).
>
> Or at least there is something in the back of my head that says "I have
> heard someone talk about this unintended side-effect of RPKI ROV, together
> with a BGP setup without 'soft in always'".
>
> Might there be a correlation in your environment?

Yes, this makes sense.

We, generally, did not run "soft-reconfiguration inbound" since the IOS
days, to save on RAM. Also, Route Refresh was the gold standard. I'm not
sure the RAM issue is a big deal anymore, considering how large it is in
routers these days...

But I can see how this creates an undesired side effect with ROV, which
then puts pressure on Route Refresh.

I have to say, we did not consider it; which hardly surprises me since
TAC didn't either.

I have read a lot of Cisco documentation on configuring ROV on IOS XE
and IOS XR, and unless something has been recently updated, this is not
one of the explicit recommendations in that documentation. I can see how
easy it is to overlook (or if you do have 'soft-reconfiguration inbound'
configured, how that significance can also be overlooked).

All that notwithstanding, the move to 100Gbps peering/transit + faster
CPU's on the MX204 is still a worthy reason to move away from the
ASR9001, for us anyway :-).

Mark.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 10:10, Saku Ytti wrote:

> Correct. Add 'keep none' to junos, and you'll have the same issue.

Since we started running ROV in IOS XE in 2014, we would have hit this
issue then and allowed for it if the BGP best path evaluation process in
IOS XE did not do strange things by default, which I believe have not
yet been fixed. So we turned it off then.

We always used Juniper (MX80 included) for peering back then, so didn't
run into this given Junos' default policy.

We ran the ASR9001 for peering/transit for a long time before coming up
against this, but it makes sense that the complaints only picked up in
the last 2 years when ROV was on the rise, globally.

Thanks for the clue, Saku. Hopefully someone here has the energy to ask
Cisco to update their documentation, to make this a recommendation. I
can't be asked :-).

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Thu, Nov 11, 2021 at 10:14:16AM +0200, Mark Tinka wrote:
> I have read a lot of Cisco documentation on configuring ROV on IOS XE
> and IOS XR, and unless something has been recently updated, this is not
> one of the explicit recommendations in that documentation. I can see how
> easy it is to overlook (or if you do have 'soft-reconfiguration inbound'
> configured, how that significance can also be overlooked).

I've been extremely underwhelmed with the quality of IOS XR documentation
in many respects, ROV being one of them.

(And ROV on IOS XE is full of different eels... as if a totally different
company has implemented it, without ever speaking to someone who has
done this before, or understand what it's supposed to do)

> All that notwithstanding, the move to 100Gbps peering/transit + faster
> CPU's on the MX204 is still a worthy reason to move away from the
> ASR9001, for us anyway :-).

Yeah, not argueing that decision, just curious about the problems you
saw.

We're moving towards Arista 7280R, which is not without its own set of
surprises...

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

On 11/11/21 10:26, Gert Doering wrote:

> I've been extremely underwhelmed with the quality of IOS XR documentation
> in many respects, ROV being one of them.
>
> (And ROV on IOS XE is full of different eels... as if a totally different
> company has implemented it, without ever speaking to someone who has
> done this before, or understand what it's supposed to do)

I and the usual friends you know of have been fighting with Cisco to fix
ROV's default behaviour in IOS XE since 2015. They refuse to listen.
Personally, I gave up.

We don't run RPKI on our CSR1000v route reflectors, so I'm not bothered.

We don't run RPKI on the few ASR1000's that are in the network, and
those are also being replaced by the MX204.

We are also working on replacing our ASR920's with something better. We
ran RPKI on those for all of 3 days until we turned it off. The box has
bigger problems, like insufficient FIB, that we are already dealing with.


> We're moving towards Arista 7280R, which is not without its own set of
> surprises...

We use this box for customer Layer 2 aggregation in the data centre.
Very happy with it.

As far as its maturity goes as an IP/MPLS box goes, I cannot tell you.
But, knowing you, I'll keep my eyes on arista-nsp open wide open :-).
It's an awfully quiet list - haven't got anything new on there since
February of this year.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 10:19, Mark Tinka <mark@tinka.africa> wrote:

> Thanks for the clue, Saku. Hopefully someone here has the energy to ask
> Cisco to update their documentation, to make this a recommendation. I
> can't be asked :-).

I think it should just be a config error. You're not just cucking
yourself, but your peers and customers. So it shouldn't be a choice
you can make.

We can also imagine improvements
1) by default keep all RPKI rejects, and have 'soft-inbound never'
optionally to turn that off
2) have 1 bit per neighbor indicating policy had rpki rejects and 2
bits for validation database update iindicating database become
less/more permissive
IFF database became more permissive and neighbor has rpki
rejects and we have soft-inbound never, then refresh





--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 11:22, Saku Ytti wrote:

> I think it should just be a config error. You're not just cucking
> yourself, but your peers and customers. So it shouldn't be a choice
> you can make.

I don't disagree, especially as there are likely several other operators
working this way, and not knowing it because the neighbor either hasn't
complained, or isn't detecting for Route Refresh noise.

However, the documentation should still be updated for folk running old
code earlier than the new code which would have this improvement.


>
> We can also imagine improvements
> 1) by default keep all RPKI rejects, and have 'soft-inbound never'
> optionally to turn that off

Similar to how Junos does it, but specifically for RPKI. That would make
sense.

Of course, if someone already uses 'soft-reconfiguration inbound' for
historical reasons, then keeping it as they enable ROV works out for
them anyway.


> 2) have 1 bit per neighbor indicating policy had rpki rejects and 2
> bits for validation database update iindicating database become
> less/more permissive
> IFF database became more permissive and neighbor has rpki
> rejects and we have soft-inbound never, then refresh

Reasonable.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hello,

On Thu, 11 Nov 2021 at 10:22, Saku Ytti <saku@ytti.fi> wrote:
>
> On Thu, 11 Nov 2021 at 10:19, Mark Tinka <mark@tinka.africa> wrote:
>
> > Thanks for the clue, Saku. Hopefully someone here has the energy to ask
> > Cisco to update their documentation, to make this a recommendation. I
> > can't be asked :-).
>
> I think it should just be a config error. You're not just cucking
> yourself, but your peers and customers. So it shouldn't be a choice
> you can make.
>
> We can also imagine improvements
> 1) by default keep all RPKI rejects, and have 'soft-inbound never'
> optionally to turn that off

When enabling ROV on the ASR9001, I set everything to
"soft-reconfiguration inbound always", because I couldn't imagine how
ROV would work in a reliable way otherwise. I didn't think people
would complain about periodic route-refreshes, I just didn't trust
that it would work reliably on huge internet exchanges with many peers
and questionable BGP implementations on the other side. I wanted my
EBGP behaviour to remain *exactly* the same, just as before-ROV.

For ROV to work reliably it needs to be able to reconsider previously
rejected invalids, so I would not recommend disabling
soft-reconfiguartion inbound, instead I'd suggest to set it to always.

Of course it would be better to store ROV-rejects separately, instead
of rejecting them. Better yet: add a new drop statement in RPL, which
makes the route not eligible for path selection, but doesn't remove it
from memory - this way the operator has full flexibility. I can't
believe I'm saying this, but a famous CPE vendor from Latvia actually
supports this (routing filter action: reject vs discard) [1], maybe
the XR BGP team could steal some ideas from them ;)

I don't like to depend on optional or not commonly used BGP features
and code paths in other people's networks for changes of any kind on
my end.

Memory consumption was negligible for my use-case, iirc it was 200 -
300 MB for 30 - 40 peers, one of which was transit (so full table -
this was about a year ago). I believe we had this conversation before,
and you mentioned that this could be a DoS vector, which is true but
all the solutions that we can possibly suggest simply implement a
"selective" soft-reconfig inbound always" anyway, so the DoS vector
needs to be addressed in a different way, in my opinion.


cheers,
lukas

[1] https://wiki.mikrotik.com/wiki/Manual:Routing/Routing_filters
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 15:43, Lukas Tribus wrote:

> For ROV to work reliably it needs to be able to reconsider previously
> rejected invalids, so I would not recommend disabling
> soft-reconfiguartion inbound, instead I'd suggest to set it to always.

If my router vendor is not automatically applying BGP policy based on
RPKI state, it shouldn't matter what ended up in RIB if I have not set
an explicit policy to act on RPKI state.

So if a previously-Invalid route now becomes a VRP, and is then good to
go for export toward a neighbor based on existing policy that matches,
why can we not leverage Route Refresh to only cater for that change?

I'm wondering if we can carry a loose relationship between the VRP
database, and RIB state.


> Of course it would be better to store ROV-rejects separately, instead
> of rejecting them. Better yet: add a new drop statement in RPL, which
> makes the route not eligible for path selection, but doesn't remove it
> from memory - this way the operator has full flexibility.

I still don't get why we need to worry about Invalid routes, if the
operator is doing ROV to drop them. As soon as they become Valid, the
RTR update will tell us that and we can allow for incremental changes
for what we ask our neighbors to accept.


> I can't
> believe I'm saying this, but a famous CPE vendor from Latvia actually
> supports this (routing filter action: reject vs discard) [1], maybe
> the XR BGP team could steal some ideas from them ;)

:-).


> I don't like to depend on optional or not commonly used BGP features
> and code paths in other people's networks for changes of any kind on
> my end.

In Juniper's case, their default to keep a copy of Adj-RIB-In had the
unintended side-effect of making ROV deployment less exciting. However,
I still feel not being able to leverage Route Refresh and avoid this
clumsiness with ROV is below par for what we can design. After all, that
was the whole point of Route Refresh...


> Memory consumption was negligible for my use-case, iirc it was 200 -
> 300 MB for 30 - 40 peers, one of which was transit (so full table -
> this was about a year ago). I believe we had this conversation before,
> and you mentioned that this could be a DoS vector, which is true but
> all the solutions that we can possibly suggest simply implement a
> "selective" soft-reconfig inbound always" anyway, so the DoS vector
> needs to be addressed in a different way, in my opinion.

Well, we had several peers disable sessions with us because their CPU
was being impacted by all the refresh messages we were sending. So we
have seen it become a DoS vector toward others, and then by extension,
toward us when we have to lose shorter paths to those peers due to the
session tear-down.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hello Gert,

On Thu, 11 Nov 2021 at 08:18, Gert Doering <gert@greenie.muc.de> wrote:
>
> Hi,
>
> On Thu, Nov 11, 2021 at 08:27:44AM +0200, Mark Tinka wrote:
> > We have nothing against the forwarding performance of the ASR9001. It's
> > the control/management plane that seems to be slowing down (at least for
> > us, anyway) with newer code and a growing Internet DFZ.
>
> "newer code" might be the key issue here - what version are you on?
>
> Our 9001 have been extremely well-behaved in regards to BGP performance
> and robustness. Not as fast as the RSP440, but still well up to the
> job.
>
> We're on 6.5.3 ("nothing interesting in newer versions, and still
> supported").

When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
specifically the RTR session would hang in certain scenarios (router
restart, RTR server reboot, etc), requiring manual intervention with a
"clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
releases, but 6.5.3 was not part of those better releases. I still did
not have a lot of trust in XR for this reason, so I wrote a NETCONF
script that would make some sanity checks on the RTR session state on
XR boxes (nagios friendly):

https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3

(minimum number of absolute v4 and v6 VRPs on each RTR server, maximum
v4/v6 VRP deviation between the RTR servers, RTR servers not in
"connected" state).


cheers,
lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 16:02, Lukas Tribus wrote:

> When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
> specifically the RTR session would hang in certain scenarios (router
> restart, RTR server reboot, etc), requiring manual intervention with a
> "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
> releases, but 6.5.3 was not part of those better releases. I still did
> not have a lot of trust in XR for this reason, so I wrote a NETCONF
> script that would make some sanity checks on the RTR session state on
> XR boxes (nagios friendly):
>
> https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3
>
> (minimum number of absolute v4 and v6 VRPs on each RTR server, maximum
> v4/v6 VRP deviation between the RTR servers, RTR servers not in
> "connected" state).

And now, with the code you are currently running for IOS XR?

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 15:12, Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/11/21 16:02, Lukas Tribus wrote:
>
> > When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
> > specifically the RTR session would hang in certain scenarios (router
> > restart, RTR server reboot, etc), requiring manual intervention with a
> > "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
> > releases, but 6.5.3 was not part of those better releases. I still did
> > not have a lot of trust in XR for this reason, so I wrote a NETCONF
> > script that would make some sanity checks on the RTR session state on
> > XR boxes (nagios friendly):
> >
> > https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3
> >
> > (minimum number of absolute v4 and v6 VRPs on each RTR server, maximum
> > v4/v6 VRP deviation between the RTR servers, RTR servers not in
> > "connected" state).
>
> And now, with the code you are currently running for IOS XR?

I haven't been able to reproduce the hung RTR session issue due to
reboot/reloads in newer code. 6.7.3 (32-bit)/7.1.3 (64-bit) should be
fine. I don't recall if 6.6.X was good or bad, but 6.5 certainly was.
Still, the monitoring script makes sense.

Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 16:26, Lukas Tribus wrote:

> Still, the monitoring script makes sense.

Yep - lots of monitoring is needed.

I'm monitoring some odd behaviour where Fort will, after a few days,
grow its delta in the number of VRP's vs. rpki-client.

Probably one of the best reasons to run different validators.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Thu, 11 Nov 2021 at 15:01, Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/11/21 15:43, Lukas Tribus wrote:
>
> > For ROV to work reliably it needs to be able to reconsider previously
> > rejected invalids, so I would not recommend disabling
> > soft-reconfiguartion inbound, instead I'd suggest to set it to always.
>
> If my router vendor is not automatically applying BGP policy based on
> RPKI state, it shouldn't matter what ended up in RIB if I have not set
> an explicit policy to act on RPKI state.
>
> So if a previously-Invalid route now becomes a VRP, and is then good to
> go for export toward a neighbor based on existing policy that matches,
> why can we not leverage Route Refresh to only cater for that change?

I believe with the amount of RAM we have on those boxes nowadays,
keeping a copy of everything should be a non-issue.

On the other hand, leveraging route-refresh changes your EBGP
behaviour, which can trigger remote and local bugs, or, as in your
case, trigger humans with most likely a little over-dramatic
monitoring. I won't trust other peoples BGP routers and
implementations more than I absolutely have to and I don't think my
time is well spent arguing with other people about their
underdimensioned control plane CPU, oversensitive CPU load monitoring
or troubleshooting corner cases in their BGP implementation that
trigger bugs in route refresh code. And then the need to explain in a
RFO why your network heavily uses route-refresh which triggered that
remote bug in the first place, while your competitor didn't change
anything in their BGP configuration in the last decade, so "they
didn't have any issue with this, only your network has issues''.

Those are all rabbit holes that I will gladly trade for a little bit
of RAM usage in a heartbeat.


Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Thu, Nov 11, 2021 at 03:02:35PM +0100, Lukas Tribus wrote:
> > We're on 6.5.3 ("nothing interesting in newer versions, and still
> > supported").
>
> When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
> specifically the RTR session would hang in certain scenarios (router
> restart, RTR server reboot, etc), requiring manual intervention with a
> "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
> releases, but 6.5.3 was not part of those better releases.

Yeah. That bug hits here as well, every now and then one of the two
sessions (or both) get stuck. We have a script that walks all our XR
boxes and verifies that they all see the same amount of prefixes on
both sessions, across all boxes... and then alerts, and we go "clear".

Not exactly displaying best of software quality, but more of a minor
nuisance (for us).

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

On 11/11/21 16:33, Lukas Tribus wrote:

> I believe with the amount of RAM we have on those boxes nowadays,
> keeping a copy of everything should be a non-issue.
>
> On the other hand, leveraging route-refresh changes your EBGP
> behaviour, which can trigger remote and local bugs, or, as in your
> case, trigger humans with most likely a little over-dramatic
> monitoring. I won't trust other peoples BGP routers and
> implementations more than I absolutely have to and I don't think my
> time is well spent arguing with other people about their
> underdimensioned control plane CPU, oversensitive CPU load monitoring
> or troubleshooting corner cases in their BGP implementation that
> trigger bugs in route refresh code. And then the need to explain in a
> RFO why your network heavily uses route-refresh which triggered that
> remote bug in the first place, while your competitor didn't change
> anything in their BGP configuration in the last decade, so "they
> didn't have any issue with this, only your network has issues''.
>
> Those are all rabbit holes that I will gladly trade for a little bit
> of RAM usage in a heartbeat.

So some friends and I are working on an RFC draft to fix this:

https://datatracker.ietf.org/doc/html/draft-ymbk-sidrops-rov-no-rr

Comments and contributions are most welcome.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Sat, 13 Nov 2021 at 13:48, Mark Tinka <mark@tinka.africa> wrote:
>

> So some friends and I are working on an RFC draft to fix this:
>
> https://datatracker.ietf.org/doc/html/draft-ymbk-sidrops-rov-no-rr
>
> Comments and contributions are most welcome.

I chose my words carefully when I said 'RPKI rejects', instead of 'invalid'.

The problem only cursorily relates to a specific RPKI validation
state. We may reject RPKI 'unknown', we may even imagine policies
which reject based on some criteria AND RPKI 'valid' (maybe I have my
own motivations for how I use VRP and want to capitalise on all three
states arbitrarily, maybe I'm rejecting valids, because I'm collecting
invalids to some separate RIB for research purposes).

That is:
soft-reconfiguration inbound never # don't keep anything
soft-reconfiguration inbound rpki ## default, keep if policy
rejected route while using validation database state (may have used
something else, but as long as reject policy used validation state,
regardless of state, we need to keep it).



--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/11/21 16:49, Gert Doering wrote:
> Not exactly displaying best of software quality, but more of a minor
> nuisance (for us).

Fundamentally, we just need to provide a lot more feedback to router
vendors as well as validator coders, about real world experiences.

Especially now since, really, there are effectively just 3 - 4
validators in the wild:

https://docs.google.com/spreadsheets/d/1uuDlO6g1DLATV5OVCa20kU9OOiX9XWBFoZT2OkOezi8/edit#gid=0

Mark.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/13/21 17:20, Saku Ytti wrote:

> I chose my words carefully when I said 'RPKI rejects', instead of 'invalid'.

Well, this only really happens on IOS XE since Cisco apply policy by
default.

On IOS XR, you'll need 'bgp bestpath origin-as allow invalid' for
Invalids not to be automatically dropped.


> The problem only cursorily relates to a specific RPKI validation
> state. We may reject RPKI 'unknown', we may even imagine policies
> which reject based on some criteria AND RPKI 'valid' (maybe I have my
> own motivations for how I use VRP and want to capitalise on all three
> states arbitrarily, maybe I'm rejecting valids, because I'm collecting
> invalids to some separate RIB for research purposes).

And that is all fine, provided YOU, as the operator, are deciding policy.

The problem is that Cisco seem to want to automatically apply policy,
particularly on IOS XE. We've hounded them about this since 2015, and
nothing has changed.

IOS XR is a little better in this specific regard, but not by much when
compared against Junos.


> soft-reconfiguration inbound rpki ## default, keep if policy
> rejected route while using validation database state (may have used
> something else, but as long as reject policy used validation state,
> regardless of state, we need to keep it).

This is what we are trying to write the RFC for - to decouple the
historical need to keep or drop Adj-RIB-In from the operational
requirements of RTR dynamics, i.e., leverage the value of Route Refresh
to its fullest extent.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Sat, 13 Nov 2021 at 21:23, Mark Tinka <mark@tinka.africa> wrote:

> And that is all fine, provided YOU, as the operator, are deciding policy.

I don't think IETF is making standards for specific implementation.
The implementation agnostic solution is to keep all routes which we
rejected due to consulting validation database, regardless of
validation state.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/14/21 08:58, Saku Ytti wrote:

> I don't think IETF is making standards for specific implementation.
> The implementation agnostic solution is to keep all routes which we
> rejected due to consulting validation database, regardless of
> validation state.

Agreed, and covered in the draft:

https://datatracker.ietf.org/doc/html/draft-ymbk-sidrops-rov-no-rr-02

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Mon, 15 Nov 2021 at 16:12, Mark Tinka <mark@tinka.africa> wrote:
>
>
>
> On 11/14/21 08:58, Saku Ytti wrote:
>
> > I don't think IETF is making standards for specific implementation.
> > The implementation agnostic solution is to keep all routes which we
> > rejected due to consulting validation database, regardless of
> > validation state.
>
> Agreed, and covered in the draft:

You better clarify:

- when this is implemented, STOP triggering route-refreshes from RTR
updates (otherwise there is no point to it in the first place)
- implement the above without BREAKING existing route-refresh
functionality for non-RTR related code paths (like other RPL changes)


Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Wed, Nov 10, 2021 at 8:15 AM Mark Tinka <mark@tinka.africa> wrote:
> Not really that interested in Cisco anymore.

We will keep our ASR 9001 until support will expire, but for small Edge nodes.

Well it is hard to trust Cisco currently.

I can recall our CSR 1000V story (permament licenses). CSR 1000V
permament licenses are EoL/EoS. But subscription CSR 1000V are not, so
you need to pay again for something you already paid for.
Next move was Catalyst 8000V which is CSR 1000V but with just new name,

How to deceive a customer who bought licenses?
1. Release a newer version under a newer name (Catalyst 8000V).
2. Retiring the previous product/version - EoS/EoL - CSR 1000V.
3. By that you simply stop discussion regarding perpetual ->
subscrption model change
4. Done. Dear customers - pay again for same piece of software!

Rob
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On 11/22/21 14:10, Robert Hass wrote:

> We will keep our ASR 9001 until support will expire, but for small Edge nodes.
>
> Well it is hard to trust Cisco currently.
>
> I can recall our CSR 1000V story (permament licenses). CSR 1000V
> permament licenses are EoL/EoS. But subscription CSR 1000V are not, so
> you need to pay again for something you already paid for.
> Next move was Catalyst 8000V which is CSR 1000V but with just new name,
>
> How to deceive a customer who bought licenses?
> 1. Release a newer version under a newer name (Catalyst 8000V).
> 2. Retiring the previous product/version - EoS/EoL - CSR 1000V.
> 3. By that you simply stop discussion regarding perpetual ->
> subscrption model change
> 4. Done. Dear customers - pay again for same piece of software!

Oh dear, sorry to hear that. But that's exactly the kind of nonsense we
are running away from Cisco for.

We love the CSR1000v, but we are not running the latest & greatest. We
use them primarily for iBGP route reflection, and have settled on
15.5(3)S9 - a.k.a. 3.16(09)S - for some time now, even though it shipped
in March of 2019. It has all the relevant BGP features we need now, and
into the future, so we aren't worried about running old code.

We did attempt upgrading them to Amsterdam - 17.3(4a) - but that was as
cluster-f* of note. First, due to some issue between later versions of
CSR1000v (16.x and higher) and how VMware identifies interface drivers,
you lose the ability to set Jumbo frames, even if the host supports it.
This totally broke iBGP sessions because IS-IS was confused, without
looking confused.

Secondly, the license drama went to hell, same way you describe. But
what I didn't know is that Cisco make you pay again for it; what horror!
I just assumed they'd honour your previous permanent license, and port
you over. Thanks for that!

Anyway, we downgraded back to what we are running today, and are happy.
Our perpetual licenses are still valid, for the code we run.

If you need the newer code, I'd recommend taking a hard look at vMX or
vSR, just for a better account management situation, if nothing else.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/