Mailing List Archive

ARP requests
Hi!

I've Cisco 7201 router acting as PE that floods the network with ARP requests
and I cannot understand, why?

The router is connected to the network core with an interface GigabitEthernet0/1 in trunk mode:

interface GigabitEthernet0/1
mtu 1618
no ip address
end

And there are multiple client-facing sub-interfaces configured similar to:

interface GigabitEthernet0/1.1993
encapsulation dot1Q 1993
ip unnumbered Loopback4
end

Some sub-interfaces have non-unnumbered IP address assigned. Note ARP-proxying is not disabled.
The router also uses an interface GigabitEthernet0/0 as IP up-link/MPLS:

interface GigabitEthernet0/0
mtu 1636
ip address X.X.X.X 255.255.255.252
ipv6 address Y:Y:Y:Y::Y/64
ipv6 enable
mpls label protocol ldp
mpls ip
end

The problem: the router sends lots of untagged ARP broadcasts (requests)
over its GigabitEthernet0/1 to a switch. I've enabled local SPAN mirroring
at the switch to make sure this is true. The requests are from router's MAC address
and some of its IPv4 addresses without 802.1q tags.

Lots of unneeded ARP broadcasts is no good. Why does it do so?
The router has not any interface with "encapsulation dot1Q 1".
It runs c7200p-adventerprisek9-mz.152-4.M6.bin

Switch mirroring configuration is the following:

monitor session 1 source interface Gi2/3 tx
! to the traffic analyzer
monitor session 1 destination interface Po14

It should not strip 801.1q tags while mirroring and I really have tags for other kinds of mirrored packets
seen at my traffic analyzer.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
Hi,

On Wed, Aug 19, 2020 at 08:02:13PM +0700, Eugene Grosbein wrote:
> Switch mirroring configuration is the following:
>
> monitor session 1 source interface Gi2/3 tx

"tx"? This is "outbound to the router", not "from the router"

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: ARP requests [ In reply to ]
19.08.2020 21:24, Gert Doering wrote:
> Hi,
>
> On Wed, Aug 19, 2020 at 08:02:13PM +0700, Eugene Grosbein wrote:
>> Switch mirroring configuration is the following:
>>
>> monitor session 1 source interface Gi2/3 tx
>
> "tx"? This is "outbound to the router", not "from the router"

You are right, I've pasted wrongly part show "show conf | i monitor"
instead of actual "show run | i monitor":

monitor session 1 source interface Gi3/9 rx
monitor session 1 destination interface Po14

Gi3/9 of the switch is actually the port connected to router's Gi0/1.
So, the question remains same.



_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
Moin,

On Wed, Aug 19, 2020 at 09:40:43PM +0700, Eugene Grosbein wrote:
> Gi3/9 of the switch is actually the port connected to router's Gi0/1.
> So, the question remains same.

Check the routes on the router. If you have something which is pointing
directly to an interface ("ip route 0.0.0.0 0.0.0.0 gi0/1") it will
generate proxy-ARP requests for all destinations covered by that route.

Always use interface + gateway-IP unless this is a desired property.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: ARP requests [ In reply to ]
19.08.2020 21:44, Gert Doering wrote:
> Moin,
>
> On Wed, Aug 19, 2020 at 09:40:43PM +0700, Eugene Grosbein wrote:
>> Gi3/9 of the switch is actually the port connected to router's Gi0/1.
>> So, the question remains same.
>
> Check the routes on the router. If you have something which is pointing
> directly to an interface ("ip route 0.0.0.0 0.0.0.0 gi0/1") it will
> generate proxy-ARP requests for all destinations covered by that route.
>
> Always use interface + gateway-IP unless this is a desired property.

I have not such routes: the command "show ip route | include 0/1$" shows nothing.
The interface Gi0/1 does not have any IP configuration itself, only its sub-interfaces have.



_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
Hi,

On Wed, Aug 19, 2020 at 10:03:29PM +0700, Eugene Grosbein wrote:
> I have not such routes: the command "show ip route | include 0/1$" shows nothing.
> The interface Gi0/1 does not have any IP configuration itself, only its sub-interfaces have.

Do you have a subinterface with no "encaps dot1q ..." configured? That
would be "untagged" then.

Otherwise, ask the router - "debug arp" - and for each ARP request you
see it issue, check the routing whether it points directly to an
interface.

(And never believe what switches say about tagging vs. not)

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: ARP requests [ In reply to ]
Eugene,

Did the interface had IP address assigned in the past to main interface and then changed to subinterface ones? I remember couple of nasty 7200-impacting bugs in 15.x train (so called “CEF rewrite” or “not 13.x”) that had stale IDB entries wrongly mapped to CEF structures and that could potentially result in similar behavior.

Also, can you identify if those ARP requests are valid, belonging to subinterface link space, or totally bogus? And if they happen both on the main interface and subinterface, or only on main interface?

Working with TAC would be probably best option going forward.


./

> On 19 Aug 2020, at 17:03, Eugene Grosbein <eugen@grosbein.net> wrote:
>
> 19.08.2020 21:44, Gert Doering wrote:
>> Moin,
>>
>> On Wed, Aug 19, 2020 at 09:40:43PM +0700, Eugene Grosbein wrote:
>>> Gi3/9 of the switch is actually the port connected to router's Gi0/1.
>>> So, the question remains same.
>>
>> Check the routes on the router. If you have something which is pointing
>> directly to an interface ("ip route 0.0.0.0 0.0.0.0 gi0/1") it will
>> generate proxy-ARP requests for all destinations covered by that route.
>>
>> Always use interface + gateway-IP unless this is a desired property.
>
> I have not such routes: the command "show ip route | include 0/1$" shows nothing.
> The interface Gi0/1 does not have any IP configuration itself, only its sub-interfaces have.
>
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net <mailto:cisco-nsp@puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-nsp <https://puck.nether.net/mailman/listinfo/cisco-nsp>
> archive at http://puck.nether.net/pipermail/cisco-nsp/ <http://puck.nether.net/pipermail/cisco-nsp/>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
Hi,

On Wed, Aug 19, 2020 at 06:23:36PM +0200, ??ukasz Bromirski wrote:
> Working with TAC would be probably best option going forward.

I find your optimism amazing.

(My TAC case about "ASR920 do not always honour IPv4 ACLs on SVIs"
is not getting anywhere since December. We managed to reproduce the
issue in the TAC lab around April...)

It might give some sort of satisfaction if a TAC case eventually comes
to a conclusion different from "works as designed" or "the effect is
gone after reboot" or "have you tried upgrading to the latest version?"
(and then "gone after reboot") but for ongoing operational problems,
we have been less than underwhelmed with TAC performance.

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: ARP requests [ In reply to ]
Gert,

> On 19 Aug 2020, at 18:29, Gert Doering <gert@greenie.muc.de> wrote:
>
> Hi,
>
> On Wed, Aug 19, 2020 at 06:23:36PM +0200, ??ukasz Bromirski wrote:
>> Working with TAC would be probably best option going forward.
>
> I find your optimism amazing.

Maybe that’s because I work for Cisco? :)

> (My TAC case about "ASR920 do not always honour IPv4 ACLs on SVIs"
> is not getting anywhere since December. We managed to reproduce the
> issue in the TAC lab around April...)

I’ll follow up with you offline on that, while that’s not exact name of the case, I found it ;)

> It might give some sort of satisfaction if a TAC case eventually comes
> to a conclusion different from "works as designed" or "the effect is
> gone after reboot" or "have you tried upgrading to the latest version?"
> (and then "gone after reboot") but for ongoing operational problems,
> we have been less than underwhelmed with TAC performance.

Unfortunately, everything comes down to proper escalation. While Cisco TAC is still biggest technical support organization, unmatched by other networking vendors, people are working on a number of cases. Priorities/Severities matter.

P1 - my network is down
P2 - my network functioning is severly degraded
P3 - I have a problem which is interesting
P4 - I have a question

I’d keep such request at least at P2 at all times. Even if TAC is waiting to schedule internal lab recreation or cooperation with platform developers to verify behavior.

Anyway, yeah - I’m actually pessimist and cynic, but enough about me ;)

--
?ukasz Bromirski / https://lukasz.bromirski.net
CCIE R&S/SP #15929, CCDE #2012::17, PGP Key ID: 0xFD077F6A
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
19.08.2020 23:21, Gert Doering wrote:

> On Wed, Aug 19, 2020 at 10:03:29PM +0700, Eugene Grosbein wrote:
>> I have not such routes: the command "show ip route | include 0/1$" shows nothing.
>> The interface Gi0/1 does not have any IP configuration itself, only its sub-interfaces have.
>
> Do you have a subinterface with no "encaps dot1q ..." configured? That
> would be "untagged" then.

Thanks, I did not think about it. There were some left-over subinterfaces in the configuration
with empty settings at all (and without encapsulation) and I've just removed all such interfaces.
Still, ARP requests continue to come.

> Otherwise, ask the router - "debug arp" - and for each ARP request you
> see it issue, check the routing whether it points directly to an
> interface.

Well, I have packet capture in my analyzer and I see all the details there,
like this:

# tcpdump -s0 -en -i $incoming ether dst ff:ff:ff:ff:ff:ff and ether src 00:26:99:28:e6:1a
23:59:06.629164 00:26:99:28:e6:1a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.19.0.19 tell 10.19.0.254, length 46

At the router:
#show ip route 10.19.0.19
Routing entry for 10.19.0.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via eigrp 200
Advertised by eigrp 200
Routing Descriptor Blocks:
* directly connected, via GigabitEthernet0/1.10
Route metric is 0, traffic share count is 1

#show run int GigabitEthernet0/1.10
Building configuration...

Current configuration : 260 bytes
!
interface GigabitEthernet0/1.10
encapsulation dot1Q 10
ip address 10.19.1.254 255.255.255.0 secondary
ip address 10.19.0.254 255.255.255.0
end

(interface description and ip access-groups not shown)

> (And never believe what switches say about tagging vs. not)

I've saved capture to PCAP file and verified it with Wireshark, no tags.
Still cannot understand why no tags.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
19.08.2020 23:23, ?ukasz Bromirski wrote:

I've just solved my problem, see below.

> Did the interface had IP address assigned in the past to main interface and then changed to subinterface ones?

No, never.

> I remember couple of nasty 7200-impacting bugs in 15.x train (so called “CEF rewrite” or “not 13.x”) that had stale IDB entries wrongly mapped to CEF structures and that could potentially result in similar behavior.
>
> Also, can you identify if those ARP requests are valid, belonging to subinterface link space, or totally bogus?

They seem to be completely valid with exception they seemingly lack tags...

> And if they happen both on the main interface and subinterface, or only on main interface?

Main only. Mirroring is done by Cisco 7600 acting as "switch" for this port,
so I went and re-read "Local SPAN, RSPAN, and ERSPAN Guidelines and Restrictions"
part of documentation, as I've read it long ago last time. And here it is:

> SPAN copies Layer 2 Ethernet frames, but SPAN does not copy
> source trunk port Inter-Switch Link Protocol (ISL) or 802.1Q tags.
> You can configure destinations as trunks to send locally tagged traffic to the traffic analyzer.

SPAN destination port was configured as L3 (no switchport) and still some traffic was copied with 802.1q vlan tags.
Nevertheless, I've reconfigured SPAN destination to switchport trunk mode with all vlans allowed
and voila! Now it shows that ARP request DO REALLY HAVE proper tags attached.

Sorry for noise, case closed. Thank you all for help.


_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: ARP requests [ In reply to ]
Hi,

On Thu, Aug 20, 2020 at 12:05:07AM +0700, Eugene Grosbein wrote:
> > Otherwise, ask the router - "debug arp" - and for each ARP request you
> > see it issue, check the routing whether it points directly to an
> > interface.
>
> Well, I have packet capture in my analyzer and I see all the details there,
> like this:

This is missing the point. The router will tell you more on "what it
thinks the interface is".

[..]
> > (And never believe what switches say about tagging vs. not)
>
> I've saved capture to PCAP file and verified it with Wireshark, no tags.
> Still cannot understand why no tags.

On many platforms, monitoring ports will not show the original tag.

(On Cat6500, tags will only be shown if the *destination* port is set
to "switchport mode trunk", for example)

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: ARP requests [ In reply to ]
Looks like you should've not used the $

show ip route | include 0/1$

this would've shown routes using any subints on g0/1

show ip route | include 0/1

-aaron


-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces@puck.nether.net> On Behalf Of Eugene
Grosbein
Sent: Wednesday, August 19, 2020 12:05 PM
To: Gert Doering <gert@greenie.muc.de>
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] ARP requests

19.08.2020 23:21, Gert Doering wrote:

> On Wed, Aug 19, 2020 at 10:03:29PM +0700, Eugene Grosbein wrote:
>> I have not such routes: the command "show ip route | include 0/1$" shows
nothing.
>> The interface Gi0/1 does not have any IP configuration itself, only its
sub-interfaces have.
>
> Do you have a subinterface with no "encaps dot1q ..." configured?
> That would be "untagged" then.

Thanks, I did not think about it. There were some left-over subinterfaces in
the configuration with empty settings at all (and without encapsulation) and
I've just removed all such interfaces.
Still, ARP requests continue to come.

> Otherwise, ask the router - "debug arp" - and for each ARP request you
> see it issue, check the routing whether it points directly to an
> interface.

Well, I have packet capture in my analyzer and I see all the details there,
like this:

# tcpdump -s0 -en -i $incoming ether dst ff:ff:ff:ff:ff:ff and ether src
00:26:99:28:e6:1a
23:59:06.629164 00:26:99:28:e6:1a > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 60: Request who-has 10.19.0.19 tell 10.19.0.254, length 46

At the router:
#show ip route 10.19.0.19
Routing entry for 10.19.0.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via eigrp 200
Advertised by eigrp 200
Routing Descriptor Blocks:
* directly connected, via GigabitEthernet0/1.10
Route metric is 0, traffic share count is 1

#show run int GigabitEthernet0/1.10
Building configuration...

Current configuration : 260 bytes
!
interface GigabitEthernet0/1.10
encapsulation dot1Q 10
ip address 10.19.1.254 255.255.255.0 secondary ip address 10.19.0.254
255.255.255.0 end

(interface description and ip access-groups not shown)

> (And never believe what switches say about tagging vs. not)

I've saved capture to PCAP file and verified it with Wireshark, no tags.
Still cannot understand why no tags.

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/