Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Cisco>
  3. NSP
cisco VPC problem on nx 3064
cedric.bassaget.ml at gmail
Oct 9, 2019, 7:13 AM
Post #1 of 5 (685 views)
Permalink
Hello,

I'm facing something I can't explain and I don't understand what is the
problem. I have a lab setup with two n3k switches and 2 linux hosts. If
somebody can tell me if there's something wrong with my setup, it would be
very helpfull.

Here's my setup :

on each n3k :
vpc domain 1, consistency OK, peer-link port-channel 4091, keepalive-link
port-channel 4090
int eth1/13 = port-channel 13 = vpc 13
int eth1/14 = port-channel 14 = vpc 14

on each linux host : iface bond1, mode 802.3ad
n3k-1 iface enp130s0f1 <-> n3k-1 eth1/13
n3k-1 iface enp130s0f0 <-> n3k-2 eth1/13
n3k-2 iface enp130s0f1 <-> n3k-1 eth1/14
n3k-2 iface enp130s0f0 <-> n3k-2 eth1/14

test : ping linux1 (3c:fd:fe:a7:9c:58) -> linux2 (3c:fd:fe:10:02:a0)

If I unplug one of the two n3k, everything continue to work as expected.
if I unplug n3k-1 eth1/13 : linux hosts can't see each other.

### before unplugging n3k-1 eth1/13 :
################ n3k-1 : mac-address table vlan 40 :
+ 40 3cfd.fe10.02a0 dynamic 0 F F Po14
+ 40 3cfd.fea7.9c58 dynamic 0 F F Po13

################ n3k-2 : mac-address table vlan 40 :
+ 40 3cfd.fe10.02a0 dynamic 0 F F Po14
+ 40 3cfd.fea7.9c58 dynamic 0 F F Po13


### after unplugging n3k-1 eth1/13
################ n3k-1 : sh mac-address table vlan 40
+ 40 3cfd.fe10.02a0 dynamic 0 F F Po14
40 3cfd.fea7.9c58 dynamic 0 F F Po4091
################ n3k-2 : sh mac-address table vlan 40
+ 40 3cfd.fe10.02a0 dynamic 0 F F Po14
+ 40 3cfd.fea7.9c58 dynamic 0 F F Po13

ICMP echo request :
linux1 (3c:fd:fe:a7:9c:58) -> n3k-2 eth1/13 -> n3k-2 eth1/14 -> linux2
(3c:fd:fe:10:02:a0)
ICMP echo reply :
linux2 (3c:fd:fe:10:02:a0) -> n3k-1 eth1/14 -> ????

ICMP echo reply is sent on the interface which is still UP on n3k-1. I
guess n3k-1 sends it to n3k-2 via VPC peer-link (Po4091). Then I don't know
what happens, but ICMP reply is never received by linux1 host.

################################
################ Configs
################################

################ n3k-1 :
vpc domain 1
peer-switch
role priority 32000
peer-keepalive destination 10.10.10.2 source 10.10.10.1 vrf vpc_keepalive
delay restore 250
auto-recovery reload-delay 300

interface port-channel4090
no switchport
vrf member vpc_keepalive
ip address 10.10.10.1/30

interface port-channel4091
switchport mode trunk
spanning-tree port type network
vpc peer-link


interface port-channel13
switchport mode trunk
switchport trunk allowed vlan 40
vpc 13

interface port-channel14
switchport mode trunk
switchport trunk allowed vlan 40
vpc 14

interface Ethernet1/13
switchport mode trunk
switchport trunk allowed vlan 40
channel-group 13 mode active
no shutdown

interface Ethernet1/14
switchport mode trunk
switchport trunk allowed vlan 40
channel-group 14 mode active
no shutdown


################ n3k-2 :
vpc domain 1
peer-switch
role priority 33000
peer-keepalive destination 10.10.10.1 source 10.10.10.2 vrf vpc_keepalive
delay restore 250
auto-recovery reload-delay 300

interface port-channel4090
no switchport
vrf member vpc_keepalive
ip address 10.10.10.2/30

interface port-channel4091
switchport mode trunk
spanning-tree port type network
vpc peer-link


interface port-channel13
switchport mode trunk
switchport trunk allowed vlan 40
vpc 13

interface port-channel14
switchport mode trunk
switchport trunk allowed vlan 40
vpc 14

interface Ethernet1/13
switchport mode trunk
switchport trunk allowed vlan 40
channel-group 13 mode active
no shutdown

interface Ethernet1/14
switchport mode trunk
switchport trunk allowed vlan 40
channel-group 14 mode active
no shutdown


################ linux 1 & linux 2 :

#cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 3c:fd:fe:a7:9c:58
Active Aggregator Info:
Aggregator ID: 2
Number of ports: 2
Actor Key: 15
Partner Key: 32781
Partner Mac Address: 00:23:04:ee:be:01

Slave Interface: enp130s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 3c:fd:fe:a7:9c:58
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 3c:fd:fe:a7:9c:58
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:01
oper key: 32781
port priority: 32768
port number: 269
port state: 61

Slave Interface: enp130s0f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 16
Permanent HW addr: 3c:fd:fe:a7:9c:59
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 3c:fd:fe:a7:9c:58
port key: 15
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:01
oper key: 32781
port priority: 32768
port number: 16653
port state: 61

################ linux 1 :
# ifconfig bond1.40
bond1.40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet x.x.x.x netmask 255.255.255.224 broadcast x.x.x.x
inet6 fe80::3efd:feff:fea7:9c58 prefixlen 64 scopeid 0x20<link>
ether 3c:fd:fe:a7:9c:58 txqueuelen 1000 (Ethernet)

################ linux 2 :
# ifconfig bond1.40
bond1.40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet x.x.x.x netmask 255.255.255.224 broadcast x.x.x.x
inet6 fe80::3efd:feff:fe10:2a0 prefixlen 64 scopeid 0x20<link>
ether 3c:fd:fe:10:02:a0 txqueuelen 1000 (Ethernet)


Thanks for your help.
Regards,
Cédric
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: cisco VPC problem on nx 3064 [ In reply to ]
cedric.bassaget.ml at gmail
Oct 9, 2019, 11:35 PM
Post #2 of 5 (677 views)
Permalink
Hello aain,
It seems my problem is related to STP.
After rebooting a switch, VPC peer-link is disabled by spanning tree :

N3K-eqx-pa3-2(config)# show spanning-tree detail | inc Peer-link
Port 8186 (port-channel4091, vPC Peer-link) of MST0000 is alternate
blocking


any reason to that ?
I'm using version 7.0(3)I7(6) on both switches.

Regards,
Cédric

Le mer. 9 oct. 2019 à 16:13, BASSAGET Cédric <cedric.bassaget.ml@gmail.com>
a écrit :

> Hello,
>
> I'm facing something I can't explain and I don't understand what is the
> problem. I have a lab setup with two n3k switches and 2 linux hosts. If
> somebody can tell me if there's something wrong with my setup, it would be
> very helpfull.
>
> Here's my setup :
>
> on each n3k :
> vpc domain 1, consistency OK, peer-link port-channel 4091,
> keepalive-link port-channel 4090
> int eth1/13 = port-channel 13 = vpc 13
> int eth1/14 = port-channel 14 = vpc 14
>
> on each linux host : iface bond1, mode 802.3ad
> n3k-1 iface enp130s0f1 <-> n3k-1 eth1/13
> n3k-1 iface enp130s0f0 <-> n3k-2 eth1/13
> n3k-2 iface enp130s0f1 <-> n3k-1 eth1/14
> n3k-2 iface enp130s0f0 <-> n3k-2 eth1/14
>
> test : ping linux1 (3c:fd:fe:a7:9c:58) -> linux2 (3c:fd:fe:10:02:a0)
>
> If I unplug one of the two n3k, everything continue to work as expected.
> if I unplug n3k-1 eth1/13 : linux hosts can't see each other.
>
> ### before unplugging n3k-1 eth1/13 :
> ################ n3k-1 : mac-address table vlan 40 :
> + 40 3cfd.fe10.02a0 dynamic 0 F F Po14
> + 40 3cfd.fea7.9c58 dynamic 0 F F Po13
>
> ################ n3k-2 : mac-address table vlan 40 :
> + 40 3cfd.fe10.02a0 dynamic 0 F F Po14
> + 40 3cfd.fea7.9c58 dynamic 0 F F Po13
>
>
> ### after unplugging n3k-1 eth1/13
> ################ n3k-1 : sh mac-address table vlan 40
> + 40 3cfd.fe10.02a0 dynamic 0 F F Po14
> 40 3cfd.fea7.9c58 dynamic 0 F F Po4091
> ################ n3k-2 : sh mac-address table vlan 40
> + 40 3cfd.fe10.02a0 dynamic 0 F F Po14
> + 40 3cfd.fea7.9c58 dynamic 0 F F Po13
>
> ICMP echo request :
> linux1 (3c:fd:fe:a7:9c:58) -> n3k-2 eth1/13 -> n3k-2 eth1/14 -> linux2
> (3c:fd:fe:10:02:a0)
> ICMP echo reply :
> linux2 (3c:fd:fe:10:02:a0) -> n3k-1 eth1/14 -> ????
>
> ICMP echo reply is sent on the interface which is still UP on n3k-1. I
> guess n3k-1 sends it to n3k-2 via VPC peer-link (Po4091). Then I don't know
> what happens, but ICMP reply is never received by linux1 host.
>
> ################################
> ################ Configs
> ################################
>
> ################ n3k-1 :
> vpc domain 1
> peer-switch
> role priority 32000
> peer-keepalive destination 10.10.10.2 source 10.10.10.1 vrf vpc_keepalive
> delay restore 250
> auto-recovery reload-delay 300
>
> interface port-channel4090
> no switchport
> vrf member vpc_keepalive
> ip address 10.10.10.1/30
>
> interface port-channel4091
> switchport mode trunk
> spanning-tree port type network
> vpc peer-link
>
>
> interface port-channel13
> switchport mode trunk
> switchport trunk allowed vlan 40
> vpc 13
>
> interface port-channel14
> switchport mode trunk
> switchport trunk allowed vlan 40
> vpc 14
>
> interface Ethernet1/13
> switchport mode trunk
> switchport trunk allowed vlan 40
> channel-group 13 mode active
> no shutdown
>
> interface Ethernet1/14
> switchport mode trunk
> switchport trunk allowed vlan 40
> channel-group 14 mode active
> no shutdown
>
>
> ################ n3k-2 :
> vpc domain 1
> peer-switch
> role priority 33000
> peer-keepalive destination 10.10.10.1 source 10.10.10.2 vrf vpc_keepalive
> delay restore 250
> auto-recovery reload-delay 300
>
> interface port-channel4090
> no switchport
> vrf member vpc_keepalive
> ip address 10.10.10.2/30
>
> interface port-channel4091
> switchport mode trunk
> spanning-tree port type network
> vpc peer-link
>
>
> interface port-channel13
> switchport mode trunk
> switchport trunk allowed vlan 40
> vpc 13
>
> interface port-channel14
> switchport mode trunk
> switchport trunk allowed vlan 40
> vpc 14
>
> interface Ethernet1/13
> switchport mode trunk
> switchport trunk allowed vlan 40
> channel-group 13 mode active
> no shutdown
>
> interface Ethernet1/14
> switchport mode trunk
> switchport trunk allowed vlan 40
> channel-group 14 mode active
> no shutdown
>
>
> ################ linux 1 & linux 2 :
>
> #cat /proc/net/bonding/bond1
> Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
>
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer3+4 (1)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
>
> 802.3ad info
> LACP rate: slow
> Min links: 0
> Aggregator selection policy (ad_select): stable
> System priority: 65535
> System MAC address: 3c:fd:fe:a7:9c:58
> Active Aggregator Info:
> Aggregator ID: 2
> Number of ports: 2
> Actor Key: 15
> Partner Key: 32781
> Partner Mac Address: 00:23:04:ee:be:01
>
> Slave Interface: enp130s0f0
> MII Status: up
> Speed: 10000 Mbps
> Duplex: full
> Link Failure Count: 2
> Permanent HW addr: 3c:fd:fe:a7:9c:58
> Slave queue ID: 0
> Aggregator ID: 2
> Actor Churn State: none
> Partner Churn State: none
> Actor Churned Count: 0
> Partner Churned Count: 0
> details actor lacp pdu:
> system priority: 65535
> system mac address: 3c:fd:fe:a7:9c:58
> port key: 15
> port priority: 255
> port number: 1
> port state: 61
> details partner lacp pdu:
> system priority: 32667
> system mac address: 00:23:04:ee:be:01
> oper key: 32781
> port priority: 32768
> port number: 269
> port state: 61
>
> Slave Interface: enp130s0f1
> MII Status: up
> Speed: 10000 Mbps
> Duplex: full
> Link Failure Count: 16
> Permanent HW addr: 3c:fd:fe:a7:9c:59
> Slave queue ID: 0
> Aggregator ID: 2
> Actor Churn State: none
> Partner Churn State: none
> Actor Churned Count: 0
> Partner Churned Count: 0
> details actor lacp pdu:
> system priority: 65535
> system mac address: 3c:fd:fe:a7:9c:58
> port key: 15
> port priority: 255
> port number: 2
> port state: 61
> details partner lacp pdu:
> system priority: 32667
> system mac address: 00:23:04:ee:be:01
> oper key: 32781
> port priority: 32768
> port number: 16653
> port state: 61
>
> ################ linux 1 :
> # ifconfig bond1.40
> bond1.40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet x.x.x.x netmask 255.255.255.224 broadcast x.x.x.x
> inet6 fe80::3efd:feff:fea7:9c58 prefixlen 64 scopeid 0x20<link>
> ether 3c:fd:fe:a7:9c:58 txqueuelen 1000 (Ethernet)
>
> ################ linux 2 :
> # ifconfig bond1.40
> bond1.40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet x.x.x.x netmask 255.255.255.224 broadcast x.x.x.x
> inet6 fe80::3efd:feff:fe10:2a0 prefixlen 64 scopeid 0x20<link>
> ether 3c:fd:fe:10:02:a0 txqueuelen 1000 (Ethernet)
>
>
> Thanks for your help.
> Regards,
> Cédric
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: cisco VPC problem on nx 3064 [ In reply to ]
nathan.lannine at gmail
Oct 10, 2019, 2:15 AM
Post #3 of 5 (677 views)
Permalink
On Thu, Oct 10, 2019 at 2:36 AM BASSAGET Cédric <
cedric.bassaget.ml@gmail.com> wrote:

> Hello aain,
> It seems my problem is related to STP.
> After rebooting a switch, VPC peer-link is disabled by spanning tree :
>
> > interface port-channel13
> > switchport mode trunk
> > switchport trunk allowed vlan 40
> > vpc 13
> >
> > interface port-channel14
> > switchport mode trunk
> > switchport trunk allowed vlan 40
> > vpc 14
> >
>
>
This really only makes sense if the linux boxen are sending bpdus, but you
could try setting the etherchannel interfaces to "spanning-tree port type
edge trunk". Of course, if you do so, you should probably run "brctl show"
on your linux endpoints just to ensure they are not somehow running their
bond interfaces in a bridge and sending bpdus.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: cisco VPC problem on nx 3064 [ In reply to ]
cedric.bassaget.ml at gmail
Oct 10, 2019, 3:27 AM
Post #4 of 5 (677 views)
Permalink
Hi Nathan,
My linux boxes are not sending BPDUs. They're running bonds in bond-mode
802.3ad. No bridge stuff here.
My last test was :

N3K-eqx-pa3-1# show running-config spanning-tree
spanning-tree mode mst
spanning-tree mst configuration
instance 1 vlan 2-4094
interface port-channel4091
spanning-tree port type network


After re-checking my config I saw that I had a priority set for vlan 40 on
switch 1. I removed it, put back all my vlans un MST0 and it seems it fixed
my problem...

Regards,
Cédric

Le jeu. 10 oct. 2019 à 11:16, Nathan Lannine <nathan.lannine@gmail.com> a
écrit :

> On Thu, Oct 10, 2019 at 2:36 AM BASSAGET Cédric <
> cedric.bassaget.ml@gmail.com> wrote:
>
>> Hello aain,
>> It seems my problem is related to STP.
>> After rebooting a switch, VPC peer-link is disabled by spanning tree :
>>
>> > interface port-channel13
>> > switchport mode trunk
>> > switchport trunk allowed vlan 40
>> > vpc 13
>> >
>> > interface port-channel14
>> > switchport mode trunk
>> > switchport trunk allowed vlan 40
>> > vpc 14
>> >
>>
>>
> This really only makes sense if the linux boxen are sending bpdus, but you
> could try setting the etherchannel interfaces to "spanning-tree port type
> edge trunk". Of course, if you do so, you should probably run "brctl show"
> on your linux endpoints just to ensure they are not somehow running their
> bond interfaces in a bridge and sending bpdus.
>
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: cisco VPC problem on nx 3064 [ In reply to ]
Friedrich at pdv-sachsen
Oct 21, 2019, 6:48 AM
Post #5 of 5 (652 views)
Permalink
Hello all

For the vpc feature peer-switch the spanning tree config has to be *exactly the same* on both switches!

Also I would suggest to change the STP priority from default 32768 , otherwise some other new connected switch could become the new root bridge...

Regards Gregor


-----Ursprüngliche Nachricht-----
Von: cisco-nsp <cisco-nsp-bounces@puck.nether.net> Im Auftrag von BASSAGET Cédric
Gesendet: Donnerstag, 10. Oktober 2019 12:28
An: Nathan Lannine <nathan.lannine@gmail.com>
Cc: Cisco-nsp <cisco-nsp@puck.nether.net>
Betreff: Re: [c-nsp] cisco VPC problem on nx 3064

Hi Nathan,
My linux boxes are not sending BPDUs. They're running bonds in bond-mode 802.3ad. No bridge stuff here.
My last test was :

N3K-eqx-pa3-1# show running-config spanning-tree spanning-tree mode mst spanning-tree mst configuration
instance 1 vlan 2-4094
interface port-channel4091
spanning-tree port type network


After re-checking my config I saw that I had a priority set for vlan 40 on switch 1. I removed it, put back all my vlans un MST0 and it seems it fixed my problem...

Regards,
Cédric

Le jeu. 10 oct. 2019 à 11:16, Nathan Lannine <nathan.lannine@gmail.com> a écrit :

> On Thu, Oct 10, 2019 at 2:36 AM BASSAGET Cédric <
> cedric.bassaget.ml@gmail.com> wrote:
>
>> Hello aain,
>> It seems my problem is related to STP.
>> After rebooting a switch, VPC peer-link is disabled by spanning tree :
>>
>> > interface port-channel13
>> > switchport mode trunk
>> > switchport trunk allowed vlan 40
>> > vpc 13
>> >
>> > interface port-channel14
>> > switchport mode trunk
>> > switchport trunk allowed vlan 40
>> > vpc 14
>> >
>>
>>
> This really only makes sense if the linux boxen are sending bpdus, but
> you could try setting the etherchannel interfaces to "spanning-tree
> port type edge trunk". Of course, if you do so, you should probably run "brctl show"
> on your linux endpoints just to ensure they are not somehow running
> their bond interfaces in a bridge and sending bpdus.
>
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal