Mailing List Archive: Cisco IOS switch SSH connections not working

Hello everyone,

We started seeing an issue starting at 1:45am Sunday whereby we can no
longer connect to one of our switches via SSH. all the normal functions
seem fine, just can't get onto the switch.

When trying to connect to it, the session just hangs for about 30 seconds
and then says connection timed out. No login prompt.

So I did a little troubleshooting and I am not seeing the attempts even
make it to the ACL. No logs of failed or attempted connections.
Additionally, there are no active ssh or any vty sessions.

So then just to get the switch to restart ssh, I generated a new rsa key.
It stopped and restarted ssh, but nothing.

So attempted to just remove the ACL and try. Still nothing. Lastly, I
enabled telnet and tried to connect via telnet. Still nothing. I really
don't want to restart the switch if there is any other way to resolve this.

Anyone have any suggestions?

This is a 6509-e with dual SUPs, so possible to fail over to the other SUP,
but that also carries downtime with it as it causes the OSPF and BGP
sessions to reset.

Nothing in the logs either other than the last successful SSH alive check
from nagios.

Best,

-Lee
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Tue, 14 Feb 2023 at 02:21, Lee Starnes via cisco-nsp
<cisco-nsp@puck.nether.net> wrote:

> So attempted to just remove the ACL and try. Still nothing. Lastly, I
> enabled telnet and tried to connect via telnet. Still nothing. I really
> don't want to restart the switch if there is any other way to resolve this.
>
> Anyone have any suggestions?

I assume you have connectivity to the box by some means, based on the
content of your email.

If packets are getting delivered to the device port, then it seems
like they fail to enter from HW to the control-plane, a somewhat
common problem and this would require deep dive into how to debug each
step in the punt path. But as some starting points, by no means not a
complete view into the punt path.

You could try ELAM capture to see what PFC thinks of the packet, is it
going to punt it to software.
You could try pinnacle capture to see what the punt looks like.
- show plat cap buffer asic pinnacle <sup#> port 4 direction out
priority lo. ## sup => rp path
- show plat cap buffer filter data ipv4 IP_SA=<ssh saddr>
- show plat cap buffer data filt
- show plat cap buffer data sample <x>
You could try to look at input buffers on input interface, to see if
buffers are full, very often there are bugs where control-plane
packets get wedged, eventually filling the buffer precluding new
packets from entering software.
- show buffer input X dump, if so
You could review TCP/TCB for stuck sessions you might need to clear manually
- clear tcp tcb might help
You could review TTY/VTY session box thinks it is holding
- clear line might help

You might not be able to fix the problem without boot, but you can
absolutely find incriminating evidence of the anatomy of the problem,
assuming you supply enough time on the keyboard.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/