Mailing List Archive

Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
On 4/2/23 23:16, Mike Hammett via cisco-nsp wrote:
> We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
> I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
> However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
> If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?

A couple points of view:
a) what you are explaining is what you see in the 'gui'
b) what you are explaining is what you see in the individual sflow records
c) routing information base has something else

For a)

Sflow may be sending out an interface index.

Interface indexes may change unless forced not to do so.

An interface name comes from a different command.

Index and name may not be in sync.

For b)

No idea.

for c)

Did you look in the routing table / cef table to see what the actual
destination is? Rather than what BGP shows you?
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
>
> Midwest Internet Exchange
>
> The Brothers WISP
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
What started this investigation was a client complained of traffic coming from another upstream instead of our direct connection.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett via cisco-nsp" <cisco-nsp@puck.nether.net>
To: "Cisco Network Service Providers" <cisco-nsp@puck.nether.net>
Sent: Monday, April 3, 2023 12:16:52 AM
Subject: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
NANOG did point me to a couple other commands (show ip route and show forwarding) that yielded the desired result, but not what's happening.


I did see an individual sFlow record that showed source\destination IPs, interfaces, etc, but it mirrors what the big picture graphs show.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Raymond Burkholder via cisco-nsp" <cisco-nsp@puck.nether.net>
To: cisco-nsp@puck.nether.net
Sent: Monday, April 3, 2023 12:31:59 AM
Subject: Re: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

On 4/2/23 23:16, Mike Hammett via cisco-nsp wrote:
> We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
> I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
> However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
> If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?

A couple points of view:
a) what you are explaining is what you see in the 'gui'
b) what you are explaining is what you see in the individual sflow records
c) routing information base has something else

For a)

Sflow may be sending out an interface index.

Interface indexes may change unless forced not to do so.

An interface name comes from a different command.

Index and name may not be in sync.

For b)

No idea.

for c)

Did you look in the routing table / cef table to see what the actual
destination is? Rather than what BGP shows you?
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
>
> Midwest Internet Exchange
>
> The Brothers WISP
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
The snmp if index reported by the Nexus matches the if index in ElastiFlow.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Raymond Burkholder via cisco-nsp" <cisco-nsp@puck.nether.net>
To: cisco-nsp@puck.nether.net
Sent: Monday, April 3, 2023 12:31:59 AM
Subject: Re: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

On 4/2/23 23:16, Mike Hammett via cisco-nsp wrote:
> We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
> I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
> However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
> If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?

A couple points of view:
a) what you are explaining is what you see in the 'gui'
b) what you are explaining is what you see in the individual sflow records
c) routing information base has something else

For a)

Sflow may be sending out an interface index.

Interface indexes may change unless forced not to do so.

An interface name comes from a different command.

Index and name may not be in sync.

For b)

No idea.

for c)

Did you look in the routing table / cef table to see what the actual
destination is? Rather than what BGP shows you?
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
>
> Midwest Internet Exchange
>
> The Brothers WISP
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
On Mon, 3 Apr 2023, Mike Hammett via cisco-nsp wrote:

> We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
>
>
> I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
>
>
> However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
>
>
> If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?

Flowspec rules? Policy routing? Bugs? :)

----------------------------------------------------------------------
Jon Lewis, MCP :) | I route
StackPath, Sr. Neteng | therefore you are
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
We did have our common upstream provider perform maintenance that then afterwards, had the traffic flowing on the right path. Later activity on our direct connection pushed it back to the common upstream. We haven't yet had the opportunity to bump our BGP session with the common upstream provider, but I suspect that will put the traffic back onto the right path. Seems like the router is just hanging onto the oldest BGP session it has, regardless of any other parameter or configuration.


This seems like a bug. We do intend on upgrading NX-OS, but that's on someone else's schedule.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett via cisco-nsp" <cisco-nsp@puck.nether.net>
To: "Cisco Network Service Providers" <cisco-nsp@puck.nether.net>
Sent: Monday, April 3, 2023 12:16:52 AM
Subject: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
We have upgraded NX-OS to a new major version and have the same results.



Apr 20 09:36:05 UTC: %UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot 1
Apr 19 13:55:57 UTC: %IPFIB-2-FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table

show forwarding inconsistency
...
69. slot(1), vrf(default), prefix(198.XXX.XXX.0/YY), Route inconsistent in FIB Software.


Well, those seem like problems.


EASTGATE401-BGP-02# show ip route summary
IP Route Table for VRF "default"
Total number of routes: 292
Total number of paths: 292

Unicast paths:
Best paths per protocol: Backup paths per protocol:
am : 48 None
local : 8
direct : 8
static : 5
broadcast : 19
bgp-XXXXX : 204

Number of routes per mask-length:
/0 : 1 /8 : 1 /18: 2 /19: 4 /20: 26
/21: 24 /22: 35 /23: 18 /24: 100 /27: 1
/28: 1 /29: 1 /30: 4 /32: 74




It's choking on only 292 routes?


Error Message FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table
Explanation The TCAM device in the Layer 3 forwarding ASIC has reached its system limits for IPv4 entries in the LPM table.

Recommended Action No action is required.



TCAM is exhausted and no action is recommended?


Error Message UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot [chars]
Explanation FIB Ipv4 route consistency checker Failed. Route database is consistent with hardware

Recommended Action No action is required.


The FIB is inconsistent and no action is recommended?


EASTGATE401-BGP-02# show system internal forwarding route summary

slot 1
=======


IPv4 hosts & routes summary on module 1
---------------------------------------------

Max host route entries : 8192
Total number of IPv4 host routes used: 72
Max LPM table entries : 7167
Total number of IPv4 LPM routes used : 16



I seem to be out of my depth here in that 292 is less than 7167, but yet it still fails.




-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]

----- Original Message -----
From: "Mike Hammett via cisco-nsp" <cisco-nsp@puck.nether.net>
To: "Cisco Network Service Providers" <cisco-nsp@puck.nether.net>
Sent: Monday, April 3, 2023 12:16:52 AM
Subject: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging [ In reply to ]
"partial tables"

There were only 292 IPv4 routes, which was expected. The IPv6 routes were expected to be less. There were 118k IPv6 routes in a box that could only handle... 4k?


Fixed that errant IPv6 fill and all is well. Well, seemingly anyway. I won't rule out a coincidence, but these seems like a direct line from problem to resolution.



-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett via cisco-nsp" <cisco-nsp@puck.nether.net>
To: "Cisco Network Service Providers" <cisco-nsp@puck.nether.net>
Sent: Monday, April 3, 2023 12:16:52 AM
Subject: [c-nsp] Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/