Mailing List Archive: Strange SNMP logs/Slowdown

Strange SNMP logs/Slowdown

Sep 18, 2018, 10:01 AM

Post #1 of 5 (1428 views)

I've a strange issue - just one of my MLX polls very, very slowly for no
discernible reason. A simple snmpwalk confirms this. Fearing it was
somehow being over polled, I added log statements to my snmp acl. I did
not find evidence of excessive polling, however, I got strange log messages:

#show log | i list 1
Sep 17 16:18:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1
cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
Sep 17 16:13:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1
cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)

They didn't increment, mgmt1 is down, acl list 1 doesn't exist nor was the
mac in the table. The other messages all make sense, but this one - this
is just weird. I'm still running 5.8.0h and have no plans to go to 6 till
they add my "ipv6 loopback in vrf can't ping" fix to the code.

I suppose I simply will reboot it, as my last tac case was easily recreated
yet caused me to beat my head against the wall till my coworkers questioned
my sanity. (A week to explain, a week for them to lab it up and then I had
to argue that it was wrong because they didn't even use a VRF, much less
MPLS) Can't reboot till next week and my head is still sore from the last
tac case.

Hence, in despair, I throw myself upon the mercy of the f-nsp list. thkx

--

E-Mail to and from me, in connection with the transaction
of public
business, is subject to the Wyoming Public Records
Act and may be
disclosed to third parties.

Re: Strange SNMP logs/Slowdown [ In reply to ]

lists at openunix

Sep 19, 2018, 12:27 AM

Post #2 of 5 (1427 views)

Permalink

On Di, Sep 18, 2018 at 11:01:21 -0600, Daniel Schmidt <daniel.schmidt@wyo.gov> wrote:
> I've a strange issue - just one of my MLX polls very, very slowly for no
> discernible reason. A simple snmpwalk confirms this. Fearing it was
> somehow being over polled, I added log statements to my snmp acl. I did
> not find evidence of excessive polling, however, I got strange log messages:
>
> #show log | i list 1
> Sep 17 16:18:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1
> cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
> Sep 17 16:13:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1
> cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)

Is cc4e.24XX.XXXX your router's mac?
198.51.100.1 is your router's IP on mgmt port?

For the SNMP issue you could change community and make a test poll.
Does the SNMP process take up too much CPU?

> They didn't increment, mgmt1 is down, acl list 1 doesn't exist nor was the
> mac in the table. The other messages all make sense, but this one - this
> is just weird. I'm still running 5.8.0h and have no plans to go to 6 till
> they add my "ipv6 loopback in vrf can't ping" fix to the code.

Does it work in 5.8?

> I suppose I simply will reboot it, as my last tac case was easily recreated
> yet caused me to beat my head against the wall till my coworkers questioned
> my sanity. (A week to explain, a week for them to lab it up and then I had
> to argue that it was wrong because they didn't even use a VRF, much less
> MPLS) Can't reboot till next week and my head is still sore from the last
> tac case.

Another thing is that the issue might come back even if solved after
reboot (if it is software defect).

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: Strange SNMP logs/Slowdown [ In reply to ]

daniel.schmidt at wyo

Sep 19, 2018, 9:00 AM

Post #3 of 5 (1427 views)

Permalink

>Is cc4e.24XX.XXXX your router's mac?
>198.51.100.1 is your router's IP on mgmt port?

It's a brocade mac, but does not show in the mac table. Yes, that is the
router's IP on loopback, or it would be if it weren't replaced with one in
the documentation range. Since that time, the error message has changed to
something even stranger that I can post completely unmodified:

Sep 19 06:12:54:W:ACL: ACL: list 1 denied all 0.0.0.0()(Ethernet 1/1
0000.0000.0000) -> 0.0.0.0(), 8414208 event(s)

All zeros, but, notice the counter is still exactly the same - hasn't
changed. Poll times seem better today - perhaps I'll just wait for upgrade
abiding the "It works, don't touch it!" principle.

>Does it work in 5.8?
No, they created a fix for that bug, will eventually make it in
6.(something). My apologies for getting off on a tangent with that rant.
I do sometimes get weird IPv6 mpls issues that I am still trying to track
down.

>Another thing is that the issue might come back even if solved after
>reboot (if it is software defect).
Yeah.... yeah, that definitely is the risk.

Thank you for your comments.

On Wed, Sep 19, 2018 at 1:28 AM Franz Georg Köhler <lists@openunix.de>
wrote:

> On Di, Sep 18, 2018 at 11:01:21 -0600, Daniel Schmidt <
> daniel.schmidt@wyo.gov> wrote:
> > I've a strange issue - just one of my MLX polls very, very slowly for no
> > discernible reason. A simple snmpwalk confirms this. Fearing it was
> > somehow being over polled, I added log statements to my snmp acl. I did
> > not find evidence of excessive polling, however, I got strange log
> messages:
> >
> > #show log | i list 1
> > Sep 17 16:18:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet
> mgmt1
> > cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
> > Sep 17 16:13:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet
> mgmt1
> > cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
>
> Is cc4e.24XX.XXXX your router's mac?
> 198.51.100.1 is your router's IP on mgmt port?
>
> For the SNMP issue you could change community and make a test poll.
> Does the SNMP process take up too much CPU?
>
> > They didn't increment, mgmt1 is down, acl list 1 doesn't exist nor was
> the
> > mac in the table. The other messages all make sense, but this one - this
> > is just weird. I'm still running 5.8.0h and have no plans to go to 6
> till
> > they add my "ipv6 loopback in vrf can't ping" fix to the code.
>
> Does it work in 5.8?
>
> > I suppose I simply will reboot it, as my last tac case was easily
> recreated
> > yet caused me to beat my head against the wall till my coworkers
> questioned
> > my sanity. (A week to explain, a week for them to lab it up and then I
> had
> > to argue that it was wrong because they didn't even use a VRF, much less
> > MPLS) Can't reboot till next week and my head is still sore from the
> last
> > tac case.
>
> Another thing is that the issue might come back even if solved after
> reboot (if it is software defect).
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

--

E-Mail to and from me, in connection with the transaction
of public
business, is subject to the Wyoming Public Records
Act and may be
disclosed to third parties.

Re: Strange SNMP logs/Slowdown [ In reply to ]

timoid at timoid

Sep 19, 2018, 3:09 PM

Post #4 of 5 (1426 views)

Permalink

Any 8x10G cards?

> On 19 Sep 2018, at 03:02, Daniel Schmidt <daniel.schmidt@wyo.gov> wrote:
>
> I've a strange issue - just one of my MLX polls very, very slowly for no discernible reason. A simple snmpwalk confirms this. Fearing it was somehow being over polled, I added log statements to my snmp acl. I did not find evidence of excessive polling, however, I got strange log messages:
>
> #show log | i list 1
> Sep 17 16:18:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1 cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
> Sep 17 16:13:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1 cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
>
> They didn't increment, mgmt1 is down, acl list 1 doesn't exist nor was the mac in the table. The other messages all make sense, but this one - this is just weird. I'm still running 5.8.0h and have no plans to go to 6 till they add my "ipv6 loopback in vrf can't ping" fix to the code.
>
> I suppose I simply will reboot it, as my last tac case was easily recreated yet caused me to beat my head against the wall till my coworkers questioned my sanity. (A week to explain, a week for them to lab it up and then I had to argue that it was wrong because they didn't even use a VRF, much less MPLS) Can't reboot till next week and my head is still sore from the last tac case.
>
> Hence, in despair, I throw myself upon the mercy of the f-nsp list. thkx
>
>
> E-Mail to and from me, in connection with the transaction
> of public business, is subject to the Wyoming Public Records
> Act and may be disclosed to third parties.
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: Strange SNMP logs/Slowdown [ In reply to ]

jethro.binks at strath

Sep 20, 2018, 1:20 AM

Post #5 of 5 (1423 views)

Permalink

On Wed, 19 Sep 2018, Tim Warnock wrote:

> Any 8x10G cards?

Hmm. Why do you query that? (asks a passive but interested observer to
this conversation.)

Jethro.

> > On 19 Sep 2018, at 03:02, Daniel Schmidt <daniel.schmidt@wyo.gov> wrote:
> >
> > I've a strange issue - just one of my MLX polls very, very slowly for no discernible reason. A simple snmpwalk confirms this. Fearing it was somehow being over polled, I added log statements to my snmp acl. I did not find evidence of excessive polling, however, I got strange log messages:
> >
> > #show log | i list 1
> > Sep 17 16:18:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1 cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
> > Sep 17 16:13:26:W:ACL: ACL: list 1 denied udp 192.0.2.153(0)(Ethernet mgmt1 cc4e.24XX.XXXX) -> 198.51.100.1(snmp), 8414208 event(s)
> >
> > They didn't increment, mgmt1 is down, acl list 1 doesn't exist nor was the mac in the table. The other messages all make sense, but this one - this is just weird. I'm still running 5.8.0h and have no plans to go to 6 till they add my "ipv6 loopback in vrf can't ping" fix to the code.
> >
> > I suppose I simply will reboot it, as my last tac case was easily recreated yet caused me to beat my head against the wall till my coworkers questioned my sanity. (A week to explain, a week for them to lab it up and then I had to argue that it was wrong because they didn't even use a VRF, much less MPLS) Can't reboot till next week and my head is still sore from the last tac case.
> >
> > Hence, in despair, I throw myself upon the mercy of the f-nsp list. thkx

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Network Manager,
Information Services Directorate, University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp