Mailing List Archive

x650 and MAC addresses missing in FDB ????
hello,

I have noticed an issue, that sometimes some mac addresses are not
visible in FDB on x650 stack.

I have several x450 in chain-patch from customer and then x650 where
router is connected to.

I could see MAC addresses for particular hosts in all x450 chain, but it
was missing in x650 for most of the time. Then sometimes it appeard.

How do I catch them ? I'am monitoring unicast flooding on all vlans and
I have a lot of such cases...


EXOS release is 12.5.4.5 v1254b5-patch1-4
numer of MAC addresses in FDB osscilates around 14k max, but this is not
a problem because fdb capacity in x650 is 32k.

Nothing in logs..

It is not tooo nasty, but I'am wondering if it is not some bug that may
affect some other services, i.e. multicasts.


Regards,
Marcin


_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Marcin Kuczera wrote:
> hello,
>
> I have noticed an issue, that sometimes some mac addresses are not
> visible in FDB on x650 stack.
>
> I have several x450 in chain-patch from customer and then x650 where
> router is connected to.
>
> I could see MAC addresses for particular hosts in all x450 chain, but it
> was missing in x650 for most of the time. Then sometimes it appeard.
>
> How do I catch them ? I'am monitoring unicast flooding on all vlans and
> I have a lot of such cases...
>
>
> EXOS release is 12.5.4.5 v1254b5-patch1-4
> numer of MAC addresses in FDB osscilates around 14k max, but this is not
> a problem because fdb capacity in x650 is 32k.
>
> Nothing in logs..
>
> It is not tooo nasty, but I'am wondering if it is not some bug that may
> affect some other services, i.e. multicasts.

well, I spoke to an extreme engineer and he mentioned, that this might
be some issue related to fdb hashing mechanism..

It is supposed to be multi-level hashing, however it might be, that the
lowest level "container" capacity, that contains whole MAC address - is
limited.
So, if there are some MAC addresses that will match this lowest level
"container", and let's say - the limit per "container" is 16, and we
have 17th MAC matching - it will not be placed there due to no space,
and the traffic will be flooded...


Is there anyone here that could verify this thesis ?
How do I debug it ?


Regards,
Marcin
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Well,
I'am recalling my issue, maybe someone has a proper contact to someone
from Extreme R&D.

at the moment we have about 13k in FDB, and up to 25Mbit/s of
unknown-unicasts traffic...
This is getting nasty...

I need to know if x650 has a probablity that not all MAC may be
represented in FDB (still below 32k limit).

Regards,
Marcin




Marcin Kuczera wrote:
> Marcin Kuczera wrote:
>> hello,
>>
>> I have noticed an issue, that sometimes some mac addresses are not
>> visible in FDB on x650 stack.
>>
>> I have several x450 in chain-patch from customer and then x650 where
>> router is connected to.
>>
>> I could see MAC addresses for particular hosts in all x450 chain, but
>> it was missing in x650 for most of the time. Then sometimes it appeard.
>>
>> How do I catch them ? I'am monitoring unicast flooding on all vlans
>> and I have a lot of such cases...
>>
>>
>> EXOS release is 12.5.4.5 v1254b5-patch1-4
>> numer of MAC addresses in FDB osscilates around 14k max, but this is
>> not a problem because fdb capacity in x650 is 32k.
>>
>> Nothing in logs..
>>
>> It is not tooo nasty, but I'am wondering if it is not some bug that
>> may affect some other services, i.e. multicasts.
>
> well, I spoke to an extreme engineer and he mentioned, that this might
> be some issue related to fdb hashing mechanism..
>
> It is supposed to be multi-level hashing, however it might be, that the
> lowest level "container" capacity, that contains whole MAC address - is
> limited.
> So, if there are some MAC addresses that will match this lowest level
> "container", and let's say - the limit per "container" is 16, and we
> have 17th MAC matching - it will not be placed there due to no space,
> and the traffic will be flooded...
>
>
> Is there anyone here that could verify this thesis ?
> How do I debug it ?
>
>
> Regards,
> Marcin
> _______________________________________________
> extreme-nsp mailing list
> extreme-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/extreme-nsp
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Hi Marcin,

Have you checked what kind of traffic it might be ?
Did you enable IGMP snoop and MLD snoop

Is the traffic limited to 1 single vlan or is it a specific vlan / type traffic that causes this ? Microsoft NLB comes to mind ...

Regards,
Erik


-----Original Message-----
From: extreme-nsp-bounces@puck.nether.net [mailto:extreme-nsp-bounces@puck.nether.net] On Behalf Of Marcin Kuczera
Sent: woensdag 28 december 2011 20:39
To: extreme-nsp@puck.nether.net
Subject: Re: [e-nsp] x650 and MAC addresses missing in FDB ????

Well,
I'am recalling my issue, maybe someone has a proper contact to someone from Extreme R&D.

at the moment we have about 13k in FDB, and up to 25Mbit/s of unknown-unicasts traffic...
This is getting nasty...

I need to know if x650 has a probablity that not all MAC may be represented in FDB (still below 32k limit).

Regards,
Marcin




Marcin Kuczera wrote:
> Marcin Kuczera wrote:
>> hello,
>>
>> I have noticed an issue, that sometimes some mac addresses are not
>> visible in FDB on x650 stack.
>>
>> I have several x450 in chain-patch from customer and then x650 where
>> router is connected to.
>>
>> I could see MAC addresses for particular hosts in all x450 chain, but
>> it was missing in x650 for most of the time. Then sometimes it appeard.
>>
>> How do I catch them ? I'am monitoring unicast flooding on all vlans
>> and I have a lot of such cases...
>>
>>
>> EXOS release is 12.5.4.5 v1254b5-patch1-4 numer of MAC addresses in
>> FDB osscilates around 14k max, but this is not a problem because fdb
>> capacity in x650 is 32k.
>>
>> Nothing in logs..
>>
>> It is not tooo nasty, but I'am wondering if it is not some bug that
>> may affect some other services, i.e. multicasts.
>
> well, I spoke to an extreme engineer and he mentioned, that this might
> be some issue related to fdb hashing mechanism..
>
> It is supposed to be multi-level hashing, however it might be, that
> the lowest level "container" capacity, that contains whole MAC address
> - is limited.
> So, if there are some MAC addresses that will match this lowest level
> "container", and let's say - the limit per "container" is 16, and we
> have 17th MAC matching - it will not be placed there due to no space,
> and the traffic will be flooded...
>
>
> Is there anyone here that could verify this thesis ?
> How do I debug it ?
>
>
> Regards,
> Marcin
> _______________________________________________
> extreme-nsp mailing list
> extreme-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/extreme-nsp
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Erik Bais wrote:
> Hi Marcin,
>
> Have you checked what kind of traffic it might be ?
> Did you enable IGMP snoop and MLD snoop

Both enabled

> Is the traffic limited to 1 single vlan or is it a specific vlan / type traffic that causes this ? Microsoft NLB comes to mind ...

It is not specific vlan, it is random behaviour.

traffic is regular IPv4:
21:09:25.276836 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
(0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
53, id 49068, offset 0, flags [DF], proto TCP (6), length 1500)
216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
322060:323520, ack 1, win 29, length 1460
21:09:25.276839 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
(0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
53, id 49069, offset 0, flags [DF], proto TCP (6), length 1500)
216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
323520:324980, ack 1, win 29, length 1460
21:09:25.276841 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
(0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
53, id 49070, offset 0, flags [DF], proto TCP (6), length 1500)
216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
324980:326440, ack 1, win 29, length 1460

i.e. 98:fc:11:cb:7b:ce is not in FDB and after several seconds it
appears, u-unicast traffic is gone for that host, but then I have
another hosts with such traffic with no entry in FDB.
Ok, in many cases it would be correct but not in ours..

When particular DST MAC is broadcasted (unknown-unicast) it apears in
FDB of next switch in chain towards DST (x450), then also I could see
that age timer refreshes ! and - this MAC is still not in FDB of x650.


On x650 I have almost 500 VLANs, 13k MAC addresses in FDB.

I don't see this behaviour on x450 switches...

Regards,
Marcin







> Regards,
> Erik
>
>
> -----Original Message-----
> From: extreme-nsp-bounces@puck.nether.net [mailto:extreme-nsp-bounces@puck.nether.net] On Behalf Of Marcin Kuczera
> Sent: woensdag 28 december 2011 20:39
> To: extreme-nsp@puck.nether.net
> Subject: Re: [e-nsp] x650 and MAC addresses missing in FDB ????
>
> Well,
> I'am recalling my issue, maybe someone has a proper contact to someone from Extreme R&D.
>
> at the moment we have about 13k in FDB, and up to 25Mbit/s of unknown-unicasts traffic...
> This is getting nasty...
>
> I need to know if x650 has a probablity that not all MAC may be represented in FDB (still below 32k limit).
>
> Regards,
> Marcin
>
>
>
>
> Marcin Kuczera wrote:
>> Marcin Kuczera wrote:
>>> hello,
>>>
>>> I have noticed an issue, that sometimes some mac addresses are not
>>> visible in FDB on x650 stack.
>>>
>>> I have several x450 in chain-patch from customer and then x650 where
>>> router is connected to.
>>>
>>> I could see MAC addresses for particular hosts in all x450 chain, but
>>> it was missing in x650 for most of the time. Then sometimes it appeard.
>>>
>>> How do I catch them ? I'am monitoring unicast flooding on all vlans
>>> and I have a lot of such cases...
>>>
>>>
>>> EXOS release is 12.5.4.5 v1254b5-patch1-4 numer of MAC addresses in
>>> FDB osscilates around 14k max, but this is not a problem because fdb
>>> capacity in x650 is 32k.
>>>
>>> Nothing in logs..
>>>
>>> It is not tooo nasty, but I'am wondering if it is not some bug that
>>> may affect some other services, i.e. multicasts.
>> well, I spoke to an extreme engineer and he mentioned, that this might
>> be some issue related to fdb hashing mechanism..
>>
>> It is supposed to be multi-level hashing, however it might be, that
>> the lowest level "container" capacity, that contains whole MAC address
>> - is limited.
>> So, if there are some MAC addresses that will match this lowest level
>> "container", and let's say - the limit per "container" is 16, and we
>> have 17th MAC matching - it will not be placed there due to no space,
>> and the traffic will be flooded...
>>
>>
>> Is there anyone here that could verify this thesis ?
>> How do I debug it ?
>>
>>
>> Regards,
>> Marcin
>> _______________________________________________
>> extreme-nsp mailing list
>> extreme-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/extreme-nsp
>>
>
> _______________________________________________
> extreme-nsp mailing list
> extreme-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/extreme-nsp
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Do you have specific type of filtering in place that puts the mac learning from hw on that switch to sw based learning ? Like port security or someting else ?

That would explain the slower mac learning, the traffic flooding etc. Eapecially in a large fdb environment.

Erik

You say it is originating from

Verstuurd vanaf mijn iPad

Op Dec 28, 2011 om 21:33 heeft "Marcin Kuczera" <marcin@leon.pl> het volgende geschreven:

> Erik Bais wrote:
>> Hi Marcin,
>>
>> Have you checked what kind of traffic it might be ?
>> Did you enable IGMP snoop and MLD snoop
>
> Both enabled
>
>> Is the traffic limited to 1 single vlan or is it a specific vlan / type traffic that causes this ? Microsoft NLB comes to mind ...
>
> It is not specific vlan, it is random behaviour.
>
> traffic is regular IPv4:
> 21:09:25.276836 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
> 53, id 49068, offset 0, flags [DF], proto TCP (6), length 1500)
> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
> 322060:323520, ack 1, win 29, length 1460
> 21:09:25.276839 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
> 53, id 49069, offset 0, flags [DF], proto TCP (6), length 1500)
> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
> 323520:324980, ack 1, win 29, length 1460
> 21:09:25.276841 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
> 53, id 49070, offset 0, flags [DF], proto TCP (6), length 1500)
> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
> 324980:326440, ack 1, win 29, length 1460
>
> i.e. 98:fc:11:cb:7b:ce is not in FDB and after several seconds it
> appears, u-unicast traffic is gone for that host, but then I have
> another hosts with such traffic with no entry in FDB.
> Ok, in many cases it would be correct but not in ours..
>
> When particular DST MAC is broadcasted (unknown-unicast) it apears in
> FDB of next switch in chain towards DST (x450), then also I could see
> that age timer refreshes ! and - this MAC is still not in FDB of x650.
>
>
> On x650 I have almost 500 VLANs, 13k MAC addresses in FDB.
>
> I don't see this behaviour on x450 switches...
>
> Regards,
> Marcin
>
>
>
>
>
>
>
>> Regards,
>> Erik
>>
>>
>> -----Original Message-----
>> From: extreme-nsp-bounces@puck.nether.net [mailto:extreme-nsp-bounces@puck.nether.net] On Behalf Of Marcin Kuczera
>> Sent: woensdag 28 december 2011 20:39
>> To: extreme-nsp@puck.nether.net
>> Subject: Re: [e-nsp] x650 and MAC addresses missing in FDB ????
>>
>> Well,
>> I'am recalling my issue, maybe someone has a proper contact to someone from Extreme R&D.
>>
>> at the moment we have about 13k in FDB, and up to 25Mbit/s of unknown-unicasts traffic...
>> This is getting nasty...
>>
>> I need to know if x650 has a probablity that not all MAC may be represented in FDB (still below 32k limit).
>>
>> Regards,
>> Marcin
>>
>>
>>
>>
>> Marcin Kuczera wrote:
>>> Marcin Kuczera wrote:
>>>> hello,
>>>>
>>>> I have noticed an issue, that sometimes some mac addresses are not
>>>> visible in FDB on x650 stack.
>>>>
>>>> I have several x450 in chain-patch from customer and then x650 where
>>>> router is connected to.
>>>>
>>>> I could see MAC addresses for particular hosts in all x450 chain, but
>>>> it was missing in x650 for most of the time. Then sometimes it appeard.
>>>>
>>>> How do I catch them ? I'am monitoring unicast flooding on all vlans
>>>> and I have a lot of such cases...
>>>>
>>>>
>>>> EXOS release is 12.5.4.5 v1254b5-patch1-4 numer of MAC addresses in
>>>> FDB osscilates around 14k max, but this is not a problem because fdb
>>>> capacity in x650 is 32k.
>>>>
>>>> Nothing in logs..
>>>>
>>>> It is not tooo nasty, but I'am wondering if it is not some bug that
>>>> may affect some other services, i.e. multicasts.
>>> well, I spoke to an extreme engineer and he mentioned, that this might
>>> be some issue related to fdb hashing mechanism..
>>>
>>> It is supposed to be multi-level hashing, however it might be, that
>>> the lowest level "container" capacity, that contains whole MAC address
>>> - is limited.
>>> So, if there are some MAC addresses that will match this lowest level
>>> "container", and let's say - the limit per "container" is 16, and we
>>> have 17th MAC matching - it will not be placed there due to no space,
>>> and the traffic will be flooded...
>>>
>>>
>>> Is there anyone here that could verify this thesis ?
>>> How do I debug it ?
>>>
>>>
>>> Regards,
>>> Marcin
>>> _______________________________________________
>>> extreme-nsp mailing list
>>> extreme-nsp@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/extreme-nsp
>>>
>>
>> _______________________________________________
>> extreme-nsp mailing list
>> extreme-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/extreme-nsp
>>
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp
Re: x650 and MAC addresses missing in FDB ???? [ In reply to ]
Erik Bais wrote:
> Do you have specific type of filtering in place that puts the mac learning from hw on that switch to sw based learning ? Like port security or someting else ?
>
> That would explain the slower mac learning, the traffic flooding etc. Eapecially in a large fdb environment.

No, have some protocol filtering on some vlans, but no mac filtering..
Or maybe I could check it somehow if someone didn't set something like that.

Regards,
Marcin


>
> Erik
>
> You say it is originating from
>
> Verstuurd vanaf mijn iPad
>
> Op Dec 28, 2011 om 21:33 heeft "Marcin Kuczera" <marcin@leon.pl> het volgende geschreven:
>
>> Erik Bais wrote:
>>> Hi Marcin,
>>>
>>> Have you checked what kind of traffic it might be ?
>>> Did you enable IGMP snoop and MLD snoop
>> Both enabled
>>
>>> Is the traffic limited to 1 single vlan or is it a specific vlan / type traffic that causes this ? Microsoft NLB comes to mind ...
>> It is not specific vlan, it is random behaviour.
>>
>> traffic is regular IPv4:
>> 21:09:25.276836 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
>> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
>> 53, id 49068, offset 0, flags [DF], proto TCP (6), length 1500)
>> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
>> 322060:323520, ack 1, win 29, length 1460
>> 21:09:25.276839 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
>> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
>> 53, id 49069, offset 0, flags [DF], proto TCP (6), length 1500)
>> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
>> 323520:324980, ack 1, win 29, length 1460
>> 21:09:25.276841 00:30:88:19:32:a5 > 98:fc:11:cb:7b:ce, ethertype 802.1Q
>> (0x8100), length 1518: vlan 3408, p 0, ethertype IPv4, (tos 0x20, ttl
>> 53, id 49070, offset 0, flags [DF], proto TCP (6), length 1500)
>> 216.155.151.93.80 > 188.137.52.217.49460: Flags [.], seq
>> 324980:326440, ack 1, win 29, length 1460
>>
>> i.e. 98:fc:11:cb:7b:ce is not in FDB and after several seconds it
>> appears, u-unicast traffic is gone for that host, but then I have
>> another hosts with such traffic with no entry in FDB.
>> Ok, in many cases it would be correct but not in ours..
>>
>> When particular DST MAC is broadcasted (unknown-unicast) it apears in
>> FDB of next switch in chain towards DST (x450), then also I could see
>> that age timer refreshes ! and - this MAC is still not in FDB of x650.
>>
>>
>> On x650 I have almost 500 VLANs, 13k MAC addresses in FDB.
>>
>> I don't see this behaviour on x450 switches...
>>
>> Regards,
>> Marcin
>>
>>
>>
>>
>>
>>
>>
>>> Regards,
>>> Erik
>>>
>>>
>>> -----Original Message-----
>>> From: extreme-nsp-bounces@puck.nether.net [mailto:extreme-nsp-bounces@puck.nether.net] On Behalf Of Marcin Kuczera
>>> Sent: woensdag 28 december 2011 20:39
>>> To: extreme-nsp@puck.nether.net
>>> Subject: Re: [e-nsp] x650 and MAC addresses missing in FDB ????
>>>
>>> Well,
>>> I'am recalling my issue, maybe someone has a proper contact to someone from Extreme R&D.
>>>
>>> at the moment we have about 13k in FDB, and up to 25Mbit/s of unknown-unicasts traffic...
>>> This is getting nasty...
>>>
>>> I need to know if x650 has a probablity that not all MAC may be represented in FDB (still below 32k limit).
>>>
>>> Regards,
>>> Marcin
>>>
>>>
>>>
>>>
>>> Marcin Kuczera wrote:
>>>> Marcin Kuczera wrote:
>>>>> hello,
>>>>>
>>>>> I have noticed an issue, that sometimes some mac addresses are not
>>>>> visible in FDB on x650 stack.
>>>>>
>>>>> I have several x450 in chain-patch from customer and then x650 where
>>>>> router is connected to.
>>>>>
>>>>> I could see MAC addresses for particular hosts in all x450 chain, but
>>>>> it was missing in x650 for most of the time. Then sometimes it appeard.
>>>>>
>>>>> How do I catch them ? I'am monitoring unicast flooding on all vlans
>>>>> and I have a lot of such cases...
>>>>>
>>>>>
>>>>> EXOS release is 12.5.4.5 v1254b5-patch1-4 numer of MAC addresses in
>>>>> FDB osscilates around 14k max, but this is not a problem because fdb
>>>>> capacity in x650 is 32k.
>>>>>
>>>>> Nothing in logs..
>>>>>
>>>>> It is not tooo nasty, but I'am wondering if it is not some bug that
>>>>> may affect some other services, i.e. multicasts.
>>>> well, I spoke to an extreme engineer and he mentioned, that this might
>>>> be some issue related to fdb hashing mechanism..
>>>>
>>>> It is supposed to be multi-level hashing, however it might be, that
>>>> the lowest level "container" capacity, that contains whole MAC address
>>>> - is limited.
>>>> So, if there are some MAC addresses that will match this lowest level
>>>> "container", and let's say - the limit per "container" is 16, and we
>>>> have 17th MAC matching - it will not be placed there due to no space,
>>>> and the traffic will be flooded...
>>>>
>>>>
>>>> Is there anyone here that could verify this thesis ?
>>>> How do I debug it ?
>>>>
>>>>
>>>> Regards,
>>>> Marcin
>>>> _______________________________________________
>>>> extreme-nsp mailing list
>>>> extreme-nsp@puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/extreme-nsp
>>>>
>>> _______________________________________________
>>> extreme-nsp mailing list
>>> extreme-nsp@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/extreme-nsp
>>>
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp