Mailing List Archive

On Sep 17, 2013, at 2:17 PM, Frank Bulk wrote:

> Are there any compatibility issues between the two cards? Placement
> requirements? Any obvious reasons why we experienced all that traffic loss?
>

Any of the circuits part of LAGs?



_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Not that Brocade is aware of. (I've asked the question to them)

Do you get the same packet loss on other ports on slot 2? First blush
would be a bad packet processor. Do you have a spare you can drop in to
test?

Aaron



On 9/17/2013 3:17 PM, Frank Bulk wrote:
> We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
> well. We saw most IP video traffic (1.3 Gbps) not flow through, and when we
> moved an inter-border router link from eth 1/2 to eth 2/2 we saw significant
> packet loss on other Internet VLANs (all on eth 1/x). Moving it back
> resolved the issue.
>
> We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4 and
> the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
> that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
> correct releases.
>
> The ring goes in one 10G port on the one 10G card and through another 10G
> port on the other 10G card.
>
> Are there any compatibility issues between the two cards? Placement
> requirements? Any obvious reasons why we experienced all that traffic loss?
>
> Regards,
>
> Frank
>
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

If you're getting loss between cards but not while traffic stays on a card, check your (h)SFMs. You can usually take one out of service without impact. You can see their load with "sh sfm-ut all" and then walk them off one at a time ("power-off snm [#]") to see if the packet loss goes away. You can also see if they're having issues with "sh sfm-link all err" but I believe that would have shown up in logs.

We've found a couple bad ones this way and they can be an absolute nightmare to discover since they'll report everything fine.

The other option could be a bad TM on a card but those usually throw errors as well.

Regards,
Mike


On Sep 17, 2013, at 1:17 PM, Frank Bulk <frnkblk@iname.com> wrote:

> We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
> well. We saw most IP video traffic (1.3 Gbps) not flow through, and when we
> moved an inter-border router link from eth 1/2 to eth 2/2 we saw significant
> packet loss on other Internet VLANs (all on eth 1/x). Moving it back
> resolved the issue.
>
> We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4 and
> the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
> that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
> correct releases.
>
> The ring goes in one 10G port on the one 10G card and through another 10G
> port on the other 10G card.
>
> Are there any compatibility issues between the two cards? Placement
> requirements? Any obvious reasons why we experienced all that traffic loss?
>
> Regards,
>
> Frank
>
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

*----------- H U R R I C A N E - E L E C T R I C ---------->>
| Mike Tindle | Senior Network Engineer | mtindle@he.net
| ASN 6939 | http://www.he.net | 510-580-4126
*--------------------------------------------------->>

Thanks for the suggestions. I believe I checked the sfm utilization during
the maintenance window, and there were no sfm errors. But we did not try
powering off one at a time. We'll have to try that.



TM's didn't throw errors and appeared clean.



Regards,



Frank Bulk



From: Mike Tindle [mailto:mtindle@he.net]
Sent: Tuesday, September 17, 2013 4:43 PM
To: Frank Bulk
Cc: foundry-nsp@puck.nether.net
Subject: Re: [f-nsp] Packet loss on MLXe-4



If you're getting loss between cards but not while traffic stays on a card,
check your (h)SFMs. You can usually take one out of service without impact.
You can see their load with "sh sfm-ut all" and then walk them off one at a
time ("power-off snm [#]") to see if the packet loss goes away. You can
also see if they're having issues with "sh sfm-link all err" but I believe
that would have shown up in logs.



We've found a couple bad ones this way and they can be an absolute nightmare
to discover since they'll report everything fine.



The other option could be a bad TM on a card but those usually throw errors
as well.



Regards,

Mike





On Sep 17, 2013, at 1:17 PM, Frank Bulk <frnkblk@iname.com
<mailto:frnkblk@iname.com> > wrote:





We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
well. We saw most IP video traffic (1.3 Gbps) not flow through, and when we
moved an inter-border router link from eth 1/2 to eth 2/2 we saw significant
packet loss on other Internet VLANs (all on eth 1/x). Moving it back
resolved the issue.

We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4 and
the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
correct releases.

The ring goes in one 10G port on the one 10G card and through another 10G
port on the other 10G card.

Are there any compatibility issues between the two cards? Placement
requirements? Any obvious reasons why we experienced all that traffic loss?

Regards,

Frank



_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net <mailto:foundry-nsp@puck.nether.net>
http://puck.nether.net/mailman/listinfo/foundry-nsp



*----------- H U R R I C A N E - E L E C T R I C ---------->>
| Mike Tindle | Senior Network Engineer | mtindle@he.net
<mailto:mtindle@he.net>
| ASN 6939 | http://www.he.net | 510-580-4126
*--------------------------------------------------->>

No, they're not.

Frank

-----Original Message-----
From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of
Steven Raymond
Sent: Tuesday, September 17, 2013 3:24 PM
To: foundry-nsp@puck.nether.net
Subject: Re: [f-nsp] Packet loss on MLXe-4


On Sep 17, 2013, at 2:17 PM, Frank Bulk wrote:

> Are there any compatibility issues between the two cards? Placement
> requirements? Any obvious reasons why we experienced all that traffic
loss?
>

Any of the circuits part of LAGs?



_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp


_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

We didn't try other ports. The IP video ingress of the ring was on the new
card, eth 2/1, and the inter-border router link was bad when moved to eth
2/2.

That card is the spare, so we can't pop another one in.

Could we theoretically swap the two cards around, as part of our
troubleshooting?

Frank

-----Original Message-----
From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of
Aaron Wendel
Sent: Tuesday, September 17, 2013 3:33 PM
To: foundry-nsp@puck.nether.net
Subject: Re: [f-nsp] Packet loss on MLXe-4

Not that Brocade is aware of. (I've asked the question to them)

Do you get the same packet loss on other ports on slot 2? First blush
would be a bad packet processor. Do you have a spare you can drop in to
test?

Aaron



On 9/17/2013 3:17 PM, Frank Bulk wrote:
> We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
> well. We saw most IP video traffic (1.3 Gbps) not flow through, and when
we
> moved an inter-border router link from eth 1/2 to eth 2/2 we saw
significant
> packet loss on other Internet VLANs (all on eth 1/x). Moving it back
> resolved the issue.
>
> We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4
and
> the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
> that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
> correct releases.
>
> The ring goes in one 10G port on the one 10G card and through another 10G
> port on the other 10G card.
>
> Are there any compatibility issues between the two cards? Placement
> requirements? Any obvious reasons why we experienced all that traffic
loss?
>
> Regards,
>
> Frank
>
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp


_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

If you're sparing a -X with a vanilla non -X (or the other way around), be aware that the chassis reports them as two different cards and you have to blow away the port config for one before inserting the other. We spare both types of 4x10g cards for this reason.

Regards,
Mike

On Sep 17, 2013, at 2:53 PM, "Frank Bulk" <frnkblk@iname.com> wrote:

> We didn't try other ports. The IP video ingress of the ring was on the new
> card, eth 2/1, and the inter-border router link was bad when moved to eth
> 2/2.
>
> That card is the spare, so we can't pop another one in.
>
> Could we theoretically swap the two cards around, as part of our
> troubleshooting?
>
> Frank
>
> -----Original Message-----
> From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of
> Aaron Wendel
> Sent: Tuesday, September 17, 2013 3:33 PM
> To: foundry-nsp@puck.nether.net
> Subject: Re: [f-nsp] Packet loss on MLXe-4
>
> Not that Brocade is aware of. (I've asked the question to them)
>
> Do you get the same packet loss on other ports on slot 2? First blush
> would be a bad packet processor. Do you have a spare you can drop in to
> test?
>
> Aaron
>
>
>
> On 9/17/2013 3:17 PM, Frank Bulk wrote:
>> We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
>> well. We saw most IP video traffic (1.3 Gbps) not flow through, and when
> we
>> moved an inter-border router link from eth 1/2 to eth 2/2 we saw
> significant
>> packet loss on other Internet VLANs (all on eth 1/x). Moving it back
>> resolved the issue.
>>
>> We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4
> and
>> the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
>> that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
>> correct releases.
>>
>> The ring goes in one 10G port on the one 10G card and through another 10G
>> port on the other 10G card.
>>
>> Are there any compatibility issues between the two cards? Placement
>> requirements? Any obvious reasons why we experienced all that traffic
> loss?
>>
>> Regards,
>>
>> Frank
>>
>>
>>
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp@puck.nether.net
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
>>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

*----------- H U R R I C A N E - E L E C T R I C ---------->>
| Mike Tindle | Senior Network Engineer | mtindle@he.net
| ASN 6939 | http://www.he.net | 510-580-4126
*--------------------------------------------------->>

Thanks for the feedback received.

We just completed a scheduled maintenance window with Brocade tech support
where both issues were resolved.

When we inserted the MLX into the ring we saw IP video traffic at some CES'
on the metro ring drop from ~700 Mbps to ~170 Mbps. The Brocade tech
noticed that we didn't have the "router pim" command configured at the
global level on the MLX. We applied that command, but it didn't resolve the
IP video issue. But while the Brocade tech was poking around, my colleague
looked at the only other MLX on the ring and noticed that we didn't have
"multicast flooding" configured on the two VLANs carrying the (multicast) IP
video traffic. He applied that to the two IP video VLANs on the problem MLX
and the IP video traffic levels restored back to normal. We learned from
Brocade TAC that without this command the LP's CPU is impacted, and so to
protect it some traffic is thrown away. If we had run "show cpu lp" we
might have seen the LP's spiked traffic levels, but alas, we only ran "show
cpu". On the CES' this "multicast flooding" feature is on by default, but
with the MLX you have to explicitly configure that.

We also moved the inter-border router link from eth 1/2 to eth 2/2 and that
was just fine now, too We theorize that the LP's CPU was hit so hard with
the multicast traffic that while the BGP and OSPF packets were properly
processed (probably because they have higher priority), unicast and other
traffic was affected.

Frank

-----Original Message-----
From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of
Frank Bulk
Sent: Tuesday, September 17, 2013 3:17 PM
To: foundry-nsp@puck.nether.net
Subject: [f-nsp] Packet loss on MLXe-4

We inserted an MLXe-4 into a Brocade ring this morning and it didn't go so
well. We saw most IP video traffic (1.3 Gbps) not flow through, and when we
moved an inter-border router link from eth 1/2 to eth 2/2 we saw significant
packet loss on other Internet VLANs (all on eth 1/x). Moving it back
resolved the issue.

We have two SFMs and have two four-port 10G cards, one is a NI-XMR-10Gx4 and
the other is a BR-MLX-10Gx4-X. We also have a BR-MLX-1GCx24-X in slot 3
that we're not using, yet. We're running 5.4.0d and the FPGAs are at the
correct releases.

The ring goes in one 10G port on the one 10G card and through another 10G
port on the other 10G card.

Are there any compatibility issues between the two cards? Placement
requirements? Any obvious reasons why we experienced all that traffic loss?

Regards,

Frank



_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp


_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp