Mailing List Archive

ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX)
Hi folks, any ideas about this?

The switches affected by this include ICX6540, 6610 and 6650 all of which
were involved in transporting the VLAN described below.

IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a
mirror port) on VLAN682 was causing >40% CPU load on all switches where
this VLAN was configured, even though there is no IP virtual interface in
this VLAN. At one point there was a brief but serious OSPF failure whilst
this condition was present.

With the ingress port shut down the CPU load returned to 1%.

We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect. We
then added a router-interface so we could add an IP6 ACL to filter *all*
IP6 traffic - again no effect

vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 682 <- added later so we could implement an ACL
multicast disable-igmp-snoop <- did not help
multicast6 disable-mld-snoop <- did not help

*We need a way to make sure that IP6 multicasts on a VLAN won't overload
the CPU on any switch with that VLAN present - ideally filter that VLAN
from the CPU altogether!*

Any ideas?

Thanks

Justin
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
*Suggestion from Ronald and Rajesh THANKS- more comments below*

*From Ronald:* Take a look at these:
http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf


*That's definitely better documentation than I've found before, thanks a
lot.We did put in commands to disable multicast IGMP (v4) and MLD (v6)
snooping.*
*It seems not to have worked - Is there something else we're missing?*

vlan 682 by port
tagged ethe 1/2/1 to 1/2/3
multicast disable-igmp-snoop <- did not help
multicast6 disable-mld-snoop <- did not help


*Rajesh: *"If you have genuine multicast traffic in your network then you
can apply Broadcast and multicast limit on the up links. Else stop the
cast by ACL."

The granularity seems to be that we can't set a limit of less than
64Mbit/sec (traffic is less than that). We tried to block IP6 altogether
via ACL - no effect.

*Is it possible that we need to remove/rebuild the VLAN or disable/enable
the interface before the Multicast or ACL settings will take effect?*

*Is there some way to simply forward the multicast traffic as layer 2 and
force the CPU to ignore it, which is what we want!*


On 19 November 2014 12:31, Ronald Esveld <ronald.esveld@qi.nl> wrote:

> Hi Justin,
>
>
>
> Take a look at these:
> http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
>
>
>
> This one helps out.
>
> Ronald
>
>
>
> *Van:* foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] *Namens *Justin
> Keery
> *Verzonden:* woensdag 19 november 2014 11:04
> *Aan:* foundry-nsp@puck.nether.net
> *Onderwerp:* [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU
> load issue (on ICX)
>
>
>
>
> Hi folks, any ideas about this?
>
> The switches affected by this include ICX6540, 6610 and 6650 all of which
> were involved in transporting the VLAN described below.
>
> IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a
> mirror port) on VLAN682 was causing >40% CPU load on all switches where
> this VLAN was configured, even though there is no IP virtual interface in
> this VLAN. At one point there was a brief but serious OSPF failure whilst
> this condition was present.
>
> With the ingress port shut down the CPU load returned to 1%.
>
> We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect.
> We then added a router-interface so we could add an IP6 ACL to filter *all*
> IP6 traffic - again no effect
>
> vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
> tagged ethe 1/2/1 to 1/2/3
> router-interface ve 682 <- added later so we could implement an ACL
> multicast disable-igmp-snoop <- did not help
> multicast6 disable-mld-snoop <- did not help
>
>
>
> *We need a way to make sure that IP6 multicasts on a VLAN won't overload
> the CPU on any switch with that VLAN present - ideally filter that VLAN
> from the CPU altogether!*
>
>
>
> Any ideas?
>
>
>
> Thanks
>
>
>
> Justin
>
>
>
>
>
> Met vriendelijke groet, With kind regards,
>
> [image: http://www.qi.nl]
>
> Ronald Esveld
> senior network engineer
>
> *Qi ict*
> Delftechpark 35-37
> Postbus 402, 2600 AK Delft
>
> T : +31 15 888 0 444 F : +31 15 888 0 445 E : ronald.esveld@qi.nl I :
> http://www.qi.nl
>
> Qi ict neemt strategisch belang in INOVATIV
> <https://www.qi.nl/actueel/qi-ict-neemt-strategisch-belang-in-inovativ>
>
>
>
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
Can you get more details on what in particular in cpu it is doing? "sh
cpu detail" or "sh proc cpu" for example (can't remember which are
supported on whic platform)?

Jethro.



On Wed, 19 Nov 2014, Justin Keery wrote:

> *Suggestion from Ronald and Rajesh THANKS- more comments below*
>
> *From Ronald:* Take a look at these:
> http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
>
>
> *That's definitely better documentation than I've found before, thanks a
> lot.We did put in commands to disable multicast IGMP (v4) and MLD (v6)
> snooping.*
> *It seems not to have worked - Is there something else we're missing?*
>
> vlan 682 by port
> tagged ethe 1/2/1 to 1/2/3
> multicast disable-igmp-snoop <- did not help
> multicast6 disable-mld-snoop <- did not help
>
>
> *Rajesh: *"If you have genuine multicast traffic in your network then you
> can apply Broadcast and multicast limit on the up links. Else stop the
> cast by ACL."
>
> The granularity seems to be that we can't set a limit of less than
> 64Mbit/sec (traffic is less than that). We tried to block IP6 altogether
> via ACL - no effect.
>
> *Is it possible that we need to remove/rebuild the VLAN or disable/enable
> the interface before the Multicast or ACL settings will take effect?*
>
> *Is there some way to simply forward the multicast traffic as layer 2 and
> force the CPU to ignore it, which is what we want!*
>
>
> On 19 November 2014 12:31, Ronald Esveld <ronald.esveld@qi.nl> wrote:
>
> > Hi Justin,
> >
> >
> >
> > Take a look at these:
> > http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> >
> >
> >
> > This one helps out.
> >
> > Ronald
> >
> >
> >
> > *Van:* foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] *Namens *Justin
> > Keery
> > *Verzonden:* woensdag 19 november 2014 11:04
> > *Aan:* foundry-nsp@puck.nether.net
> > *Onderwerp:* [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU
> > load issue (on ICX)
> >
> >
> >
> >
> > Hi folks, any ideas about this?
> >
> > The switches affected by this include ICX6540, 6610 and 6650 all of which
> > were involved in transporting the VLAN described below.
> >
> > IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a
> > mirror port) on VLAN682 was causing >40% CPU load on all switches where
> > this VLAN was configured, even though there is no IP virtual interface in
> > this VLAN. At one point there was a brief but serious OSPF failure whilst
> > this condition was present.
> >
> > With the ingress port shut down the CPU load returned to 1%.
> >
> > We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect.
> > We then added a router-interface so we could add an IP6 ACL to filter *all*
> > IP6 traffic - again no effect
> >
> > vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
> > tagged ethe 1/2/1 to 1/2/3
> > router-interface ve 682 <- added later so we could implement an ACL
> > multicast disable-igmp-snoop <- did not help
> > multicast6 disable-mld-snoop <- did not help
> >
> >
> >
> > *We need a way to make sure that IP6 multicasts on a VLAN won't overload
> > the CPU on any switch with that VLAN present - ideally filter that VLAN
> > from the CPU altogether!*
> >
> >
> >
> > Any ideas?
> >
> >
> >
> > Thanks
> >
> >
> >
> > Justin
> >
> >
> >
> >
> >
> > Met vriendelijke groet, With kind regards,
> >
> > [image: http://www.qi.nl]
> >
> > Ronald Esveld
> > senior network engineer
> >
> > *Qi ict*
> > Delftechpark 35-37
> > Postbus 402, 2600 AK Delft
> >
> > T : +31 15 888 0 444 F : +31 15 888 0 445 E : ronald.esveld@qi.nl I :
> > http://www.qi.nl
> >
> > Qi ict neemt strategisch belang in INOVATIV
> > <https://www.qi.nl/actueel/qi-ict-neemt-strategisch-belang-in-inovativ>
> >
> >
> >
>

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Network Manager,
Information Services Directorate, University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
The platform is ICX - the traffic in fact passed through three models and
all have the same symptoms (over 40% CPU load and occasional OSPF issues as
a result)

ICX6450, ICX6610 and ICX6650

The ICX platform does not offer granular CPU info - it just describes all
activity as "application".

Frustratingly therefore there is no good info about what the CPU is doing -
all we know is that there's IP6 multicast traffic, and when we shut the
port down the CPU load goes back to normal :-(

*All we want to do is pass through and not process the traffic. No
snooping, no CPU processing at all.*

The continued suggestions are much appreciated!

Thanks!


On 19 November 2014 13:34, Jethro R Binks <jethro.binks@strath.ac.uk> wrote:

> Can you get more details on what in particular in cpu it is doing? "sh
> cpu detail" or "sh proc cpu" for example (can't remember which are
> supported on whic platform)?
>
> Jethro.
>
>
>
> On Wed, 19 Nov 2014, Justin Keery wrote:
>
> > *Suggestion from Ronald and Rajesh THANKS- more comments below*
> >
> > *From Ronald:* Take a look at these:
> >
> http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> >
> >
> > *That's definitely better documentation than I've found before, thanks a
> > lot.We did put in commands to disable multicast IGMP (v4) and MLD (v6)
> > snooping.*
> > *It seems not to have worked - Is there something else we're missing?*
> >
> > vlan 682 by port
> > tagged ethe 1/2/1 to 1/2/3
> > multicast disable-igmp-snoop <- did not help
> > multicast6 disable-mld-snoop <- did not help
> >
> >
> > *Rajesh: *"If you have genuine multicast traffic in your network then
> you
> > can apply Broadcast and multicast limit on the up links. Else stop the
> > cast by ACL."
> >
> > The granularity seems to be that we can't set a limit of less than
> > 64Mbit/sec (traffic is less than that). We tried to block IP6 altogether
> > via ACL - no effect.
> >
> > *Is it possible that we need to remove/rebuild the VLAN or disable/enable
> > the interface before the Multicast or ACL settings will take effect?*
> >
> > *Is there some way to simply forward the multicast traffic as layer 2 and
> > force the CPU to ignore it, which is what we want!*
> >
> >
> > On 19 November 2014 12:31, Ronald Esveld <ronald.esveld@qi.nl> wrote:
> >
> > > Hi Justin,
> > >
> > >
> > >
> > > Take a look at these:
> > >
> http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> > >
> > >
> > >
> > > This one helps out.
> > >
> > > Ronald
> > >
> > >
> > >
> > > *Van:* foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net]
> *Namens *Justin
> > > Keery
> > > *Verzonden:* woensdag 19 november 2014 11:04
> > > *Aan:* foundry-nsp@puck.nether.net
> > > *Onderwerp:* [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe
> CPU
> > > load issue (on ICX)
> > >
> > >
> > >
> > >
> > > Hi folks, any ideas about this?
> > >
> > > The switches affected by this include ICX6540, 6610 and 6650 all of
> which
> > > were involved in transporting the VLAN described below.
> > >
> > > IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark
> on a
> > > mirror port) on VLAN682 was causing >40% CPU load on all switches where
> > > this VLAN was configured, even though there is no IP virtual interface
> in
> > > this VLAN. At one point there was a brief but serious OSPF failure
> whilst
> > > this condition was present.
> > >
> > > With the ingress port shut down the CPU load returned to 1%.
> > >
> > > We tried to disable IP4 and IP6 igmp / mld snooping, this had no
> effect.
> > > We then added a router-interface so we could add an IP6 ACL to filter
> *all*
> > > IP6 traffic - again no effect
> > >
> > > vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
> > > tagged ethe 1/2/1 to 1/2/3
> > > router-interface ve 682 <- added later so we could implement an ACL
> > > multicast disable-igmp-snoop <- did not help
> > > multicast6 disable-mld-snoop <- did not help
> > >
> > >
> > >
> > > *We need a way to make sure that IP6 multicasts on a VLAN won't
> overload
> > > the CPU on any switch with that VLAN present - ideally filter that VLAN
> > > from the CPU altogether!*
> > >
> > >
> > >
> > > Any ideas?
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > > Justin
> > >
> > >
> > >
> > >
> > >
> > > Met vriendelijke groet, With kind regards,
> > >
> > > [image: http://www.qi.nl]
> > >
> > > Ronald Esveld
> > > senior network engineer
> > >
> > > *Qi ict*
> > > Delftechpark 35-37
> > > Postbus 402, 2600 AK Delft
> > >
> > > T : +31 15 888 0 444 F : +31 15 888 0 445 E : ronald.esveld@qi.nl
> I :
> > > http://www.qi.nl
> > >
> > > Qi ict neemt strategisch belang in INOVATIV
> > > <https://www.qi.nl/actueel/qi-ict-neemt-strategisch-belang-in-inovativ
> >
> > >
> > >
> > >
> >
>
> . . . . . . . . . . . . . . . . . . . . . . . . .
> Jethro R Binks, Network Manager,
> Information Services Directorate, University Of Strathclyde, Glasgow, UK
>
> The University of Strathclyde is a charitable body, registered in
> Scotland, number SC015263.
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
There are frequently software fixes for high CPU conditions, check the
release notes of later releases and see if anything springs out at you?

Jethro.


On Wed, 19 Nov 2014, Justin Keery wrote:

> The platform is ICX - the traffic in fact passed through three models and
> all have the same symptoms (over 40% CPU load and occasional OSPF issues as
> a result)
>
> ICX6450, ICX6610 and ICX6650
>
> The ICX platform does not offer granular CPU info - it just describes all
> activity as "application".
>
> Frustratingly therefore there is no good info about what the CPU is doing -
> all we know is that there's IP6 multicast traffic, and when we shut the
> port down the CPU load goes back to normal :-(
>
> *All we want to do is pass through and not process the traffic. No
> snooping, no CPU processing at all.*
>
> The continued suggestions are much appreciated!
>
> Thanks!
>
>
> On 19 November 2014 13:34, Jethro R Binks <jethro.binks@strath.ac.uk> wrote:
>
> > Can you get more details on what in particular in cpu it is doing? "sh
> > cpu detail" or "sh proc cpu" for example (can't remember which are
> > supported on whic platform)?
> >
> > Jethro.
> >
> >
> >
> > On Wed, 19 Nov 2014, Justin Keery wrote:
> >
> > > *Suggestion from Ronald and Rajesh THANKS- more comments below*
> > >
> > > *From Ronald:* Take a look at these:
> > >
> > http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> > >
> > >
> > > *That's definitely better documentation than I've found before, thanks a
> > > lot.We did put in commands to disable multicast IGMP (v4) and MLD (v6)
> > > snooping.*
> > > *It seems not to have worked - Is there something else we're missing?*
> > >
> > > vlan 682 by port
> > > tagged ethe 1/2/1 to 1/2/3
> > > multicast disable-igmp-snoop <- did not help
> > > multicast6 disable-mld-snoop <- did not help
> > >
> > >
> > > *Rajesh: *"If you have genuine multicast traffic in your network then
> > you
> > > can apply Broadcast and multicast limit on the up links. Else stop the
> > > cast by ACL."
> > >
> > > The granularity seems to be that we can't set a limit of less than
> > > 64Mbit/sec (traffic is less than that). We tried to block IP6 altogether
> > > via ACL - no effect.
> > >
> > > *Is it possible that we need to remove/rebuild the VLAN or disable/enable
> > > the interface before the Multicast or ACL settings will take effect?*
> > >
> > > *Is there some way to simply forward the multicast traffic as layer 2 and
> > > force the CPU to ignore it, which is what we want!*
> > >
> > >
> > > On 19 November 2014 12:31, Ronald Esveld <ronald.esveld@qi.nl> wrote:
> > >
> > > > Hi Justin,
> > > >
> > > >
> > > >
> > > > Take a look at these:
> > > >
> > http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> > > >
> > > >
> > > >
> > > > This one helps out.
> > > >
> > > > Ronald
> > > >
> > > >
> > > >
> > > > *Van:* foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net]
> > *Namens *Justin
> > > > Keery
> > > > *Verzonden:* woensdag 19 november 2014 11:04
> > > > *Aan:* foundry-nsp@puck.nether.net
> > > > *Onderwerp:* [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe
> > CPU
> > > > load issue (on ICX)
> > > >
> > > >
> > > >
> > > >
> > > > Hi folks, any ideas about this?
> > > >
> > > > The switches affected by this include ICX6540, 6610 and 6650 all of
> > which
> > > > were involved in transporting the VLAN described below.
> > > >
> > > > IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark
> > on a
> > > > mirror port) on VLAN682 was causing >40% CPU load on all switches where
> > > > this VLAN was configured, even though there is no IP virtual interface
> > in
> > > > this VLAN. At one point there was a brief but serious OSPF failure
> > whilst
> > > > this condition was present.
> > > >
> > > > With the ingress port shut down the CPU load returned to 1%.
> > > >
> > > > We tried to disable IP4 and IP6 igmp / mld snooping, this had no
> > effect.
> > > > We then added a router-interface so we could add an IP6 ACL to filter
> > *all*
> > > > IP6 traffic - again no effect
> > > >
> > > > vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
> > > > tagged ethe 1/2/1 to 1/2/3
> > > > router-interface ve 682 <- added later so we could implement an ACL
> > > > multicast disable-igmp-snoop <- did not help
> > > > multicast6 disable-mld-snoop <- did not help
> > > >
> > > >
> > > >
> > > > *We need a way to make sure that IP6 multicasts on a VLAN won't
> > overload
> > > > the CPU on any switch with that VLAN present - ideally filter that VLAN
> > > > from the CPU altogether!*
> > > >
> > > >
> > > >
> > > > Any ideas?
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > Justin
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Met vriendelijke groet, With kind regards,
> > > >
> > > > [image: http://www.qi.nl]
> > > >
> > > > Ronald Esveld
> > > > senior network engineer
> > > >
> > > > *Qi ict*
> > > > Delftechpark 35-37
> > > > Postbus 402, 2600 AK Delft
> > > >
> > > > T : +31 15 888 0 444 F : +31 15 888 0 445 E : ronald.esveld@qi.nl
> > I :
> > > > http://www.qi.nl
> > > >
> > > > Qi ict neemt strategisch belang in INOVATIV
> > > > <https://www.qi.nl/actueel/qi-ict-neemt-strategisch-belang-in-inovativ
> > >
> > > >
> > > >
> > > >
> > >
> >
> > . . . . . . . . . . . . . . . . . . . . . . . . .
> > Jethro R Binks, Network Manager,
> > Information Services Directorate, University Of Strathclyde, Glasgow, UK
> >
> > The University of Strathclyde is a charitable body, registered in
> > Scotland, number SC015263.
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
>

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Network Manager,
Information Services Directorate, University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
Do you have MLD snooping turned on? If so, that could be an issue.



Frank



From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of Justin Keery
Sent: Wednesday, November 19, 2014 4:04 AM
To: foundry-nsp@puck.nether.net
Subject: [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX)




Hi folks, any ideas about this?

The switches affected by this include ICX6540, 6610 and 6650 all of which were involved in transporting the VLAN described below.

IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a mirror port) on VLAN682 was causing >40% CPU load on all switches where this VLAN was configured, even though there is no IP virtual interface in this VLAN. At one point there was a brief but serious OSPF failure whilst this condition was present.

With the ingress port shut down the CPU load returned to 1%.

We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect. We then added a router-interface so we could add an IP6 ACL to filter *all* IP6 traffic - again no effect

vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 682 <- added later so we could implement an ACL
multicast disable-igmp-snoop <- did not help
multicast6 disable-mld-snoop <- did not help



We need a way to make sure that IP6 multicasts on a VLAN won't overload the CPU on any switch with that VLAN present - ideally filter that VLAN from the CPU altogether!



Any ideas?



Thanks



Justin
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
Thanks. Have you read this blog and discussion?

http://blog.ipspace.net/2014/09/ipv6-neighbor-discovery-nd-and.html

http://www.ietf.org/mail-archive/web/v6ops/current/msg19877.html [.very long breaks into a couple of sub-threads, but worth reading]



Frank



From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of Justin Keery
Sent: Wednesday, November 19, 2014 8:01 AM
To: Jethro R Binks; foundry-nsp@puck.nether.net
Subject: Re: [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX)





The platform is ICX - the traffic in fact passed through three models and all have the same symptoms (over 40% CPU load and occasional OSPF issues as a result)



ICX6450, ICX6610 and ICX6650



The ICX platform does not offer granular CPU info - it just describes all activity as "application".



Frustratingly therefore there is no good info about what the CPU is doing - all we know is that there's IP6 multicast traffic, and when we shut the port down the CPU load goes back to normal :-(



All we want to do is pass through and not process the traffic. No snooping, no CPU processing at all.




The continued suggestions are much appreciated!



Thanks!



On 19 November 2014 13:34, Jethro R Binks <jethro.binks@strath.ac.uk <mailto:jethro.binks@strath.ac.uk> > wrote:

Can you get more details on what in particular in cpu it is doing? "sh
cpu detail" or "sh proc cpu" for example (can't remember which are
supported on whic platform)?

Jethro.



On Wed, 19 Nov 2014, Justin Keery wrote:

> *Suggestion from Ronald and Rajesh THANKS- more comments below*
>
> *From Ronald:* Take a look at these:
> http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
>
>
> *That's definitely better documentation than I've found before, thanks a
> lot.We did put in commands to disable multicast IGMP (v4) and MLD (v6)
> snooping.*
> *It seems not to have worked - Is there something else we're missing?*
>
> vlan 682 by port
> tagged ethe 1/2/1 to 1/2/3
> multicast disable-igmp-snoop <- did not help
> multicast6 disable-mld-snoop <- did not help
>
>
> *Rajesh: *"If you have genuine multicast traffic in your network then you
> can apply Broadcast and multicast limit on the up links. Else stop the
> cast by ACL."
>
> The granularity seems to be that we can't set a limit of less than
> 64Mbit/sec (traffic is less than that). We tried to block IP6 altogether
> via ACL - no effect.
>
> *Is it possible that we need to remove/rebuild the VLAN or disable/enable
> the interface before the Multicast or ACL settings will take effect?*
>
> *Is there some way to simply forward the multicast traffic as layer 2 and
> force the CPU to ignore it, which is what we want!*
>
>
> On 19 November 2014 12:31, Ronald Esveld <ronald.esveld@qi.nl <mailto:ronald.esveld@qi.nl> > wrote:
>
> > Hi Justin,
> >
> >
> >
> > Take a look at these:
> > http://www.brocade.com/downloads/documents/product_manuals/B_FastIron/FastIron_08000a_MulticastGuide.pdf
> >
> >
> >
> > This one helps out.
> >
> > Ronald
> >
> >
> >
> > *Van:* foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net <mailto:foundry-nsp-bounces@puck.nether.net> ] *Namens *Justin
> > Keery
> > *Verzonden:* woensdag 19 november 2014 11:04
> > *Aan:* foundry-nsp@puck.nether.net <mailto:foundry-nsp@puck.nether.net>
> > *Onderwerp:* [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU
> > load issue (on ICX)
> >
> >
> >
> >
> > Hi folks, any ideas about this?
> >
> > The switches affected by this include ICX6540, 6610 and 6650 all of which
> > were involved in transporting the VLAN described below.
> >
> > IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a
> > mirror port) on VLAN682 was causing >40% CPU load on all switches where
> > this VLAN was configured, even though there is no IP virtual interface in
> > this VLAN. At one point there was a brief but serious OSPF failure whilst
> > this condition was present.
> >
> > With the ingress port shut down the CPU load returned to 1%.
> >
> > We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect.
> > We then added a router-interface so we could add an IP6 ACL to filter *all*
> > IP6 traffic - again no effect
> >
> > vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
> > tagged ethe 1/2/1 to 1/2/3
> > router-interface ve 682 <- added later so we could implement an ACL
> > multicast disable-igmp-snoop <- did not help
> > multicast6 disable-mld-snoop <- did not help
> >
> >
> >
> > *We need a way to make sure that IP6 multicasts on a VLAN won't overload
> > the CPU on any switch with that VLAN present - ideally filter that VLAN
> > from the CPU altogether!*
> >
> >
> >
> > Any ideas?
> >
> >
> >
> > Thanks
> >
> >
> >
> > Justin
> >
> >
> >
> >
> >
> > Met vriendelijke groet, With kind regards,
> >
> > [image: http://www.qi.nl]
> >
> > Ronald Esveld
> > senior network engineer
> >
> > *Qi ict*
> > Delftechpark 35-37
> > Postbus 402, 2600 AK Delft
> >
> > T : +31 15 888 0 444 <tel:%2B31%2015%20888%200%20444> F : +31 15 888 0 445 <tel:%2B31%2015%20888%200%20445> E : ronald.esveld@qi.nl <mailto:ronald.esveld@qi.nl> I :
> > http://www.qi.nl
> >
> > Qi ict neemt strategisch belang in INOVATIV
> > <https://www.qi.nl/actueel/qi-ict-neemt-strategisch-belang-in-inovativ>
> >
> >
> >
>

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Network Manager,
Information Services Directorate, University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net <mailto:foundry-nsp@puck.nether.net>
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
Hi Justin,
 
I had a similar issue some weeks ago on a FESX. TAC told me they have CPU issues with IPv6 Multicasts caused by intel network cards. See link below:
 

https://communities.intel.com/message/220048#220048

 
Solution was to update the network card driver.
 
Regards,
Frank
 
-----Ursprüngliche Nachricht-----
Von: "Justin Keery" [justin.keery@venus.co.uk]
Gesendet: Mi. 19.11.2014 11:03
An: "foundry-nsp@puck.nether.net" [foundry-nsp@puck.nether.net]
Betreff: [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX)


Hi folks, any ideas about this?

The switches affected by this include ICX6540, 6610 and 6650 all of which were involved in transporting the VLAN described below.

IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a mirror port) on VLAN682 was causing >40% CPU load on all switches where this VLAN was configured, even though there is no IP virtual interface in this VLAN. At one point there was a brief but serious OSPF failure whilst this condition was present.

With the ingress port shut down the CPU load returned to 1%.

We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect. We then added a router-interface so we could add an IP6 ACL to filter *all* IP6 traffic - again no effect

vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
 tagged ethe 1/2/1 to 1/2/3
 router-interface ve 682 <- added later so we could implement an ACL
 multicast disable-igmp-snoop <- did not help
 multicast6 disable-mld-snoop <- did not help
We need a way to make sure that IP6 multicasts on a VLAN won't overload the CPU on any switch with that VLAN present - ideally filter that VLAN from the CPU altogether!Any ideas?ThanksJustin-----Ursprüngliche Nachricht Ende-----


---
Alle Postfächer an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen! Rundum glücklich mit freenetMail
Re: ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX) [ In reply to ]
Reading this the other way, BTAC are admitting that FESX boxes will die if you hit them with certain types of multicast through-traffic?  That's not good.

Nick

On 20/11/2014 12:33, dd7zt@freenet.de wrote:


Hi Justin,

 

I had a similar issue some weeks ago on a FESX. TAC told me they have CPU issues with IPv6 Multicasts caused by intel network cards. See link below:

 





https://communities.intel.com/message/220048#220048"]https://communities.intel.com/message/220048#220048

 

Solution was to update the network card driver.

 

Regards,

Frank

  -----Ursprüngliche Nachricht-----
Von: "Justin Keery" [justin.keery@venus.co.uk]
Gesendet: Mi. 19.11.2014 11:03
An: "foundry-nsp@puck.nether.net" [foundry-nsp@puck.nether.net]
Betreff: [f-nsp] ANY IDEAS - IP6 multicast traffic causing severe CPU load issue (on ICX)


Hi folks, any ideas about this?

The switches affected by this include ICX6540, 6610 and 6650 all of which were involved in transporting the VLAN described below.

IP6 multcast traffic (less than 20Mbit/sec, discovered with wireshark on a mirror port) on VLAN682 was causing >40% CPU load on all switches where this VLAN was configured, even though there is no IP virtual interface in this VLAN. At one point there was a brief but serious OSPF failure whilst this condition was present.

With the ingress port shut down the CPU load returned to 1%.

We tried to disable IP4 and IP6 igmp / mld snooping, this had no effect. We then added a router-interface so we could add an IP6 ACL to filter *all* IP6 traffic - again no effect

vlan 682 name KARMARAMA_L2_ONEA809159_682 by port
 tagged ethe 1/2/1 to 1/2/3
 router-interface ve 682 <- added later so we could implement an ACL
 multicast disable-igmp-snoop <- did not help
 multicast6 disable-mld-snoop <- did not help
We need a way to make sure that IP6 multicasts on a VLAN won't overload the CPU on any switch with that VLAN present - ideally filter that VLAN from the CPU altogether! Any ideas? Thanks Justin

-----Ursprüngliche Nachricht Ende-----

---
Alle Postfächer an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen! http://email.freenet.de/basic/Informationen"]Rundum glücklich mit freenetMail

_______________________________________________ foundry-nsp mailing list foundry-nsp@puck.nether.net http://puck.nether.net/mailman/listinfo/foundry-nsp"]http://puck.nether.net/mailman/listinfo/foundry-nsp