Ways to spot switch trouble?

[Anonymous]

Today we experienced a painful situation in our own office:  Someone plugged in a mini-switch to a network jack at a temporary work desk.  This was connected to our 1638 core switch.  That's fine and all, but every time the user plugged in a device to a certain port on the mini-switch, our entire LAN would "hiccup," causing loss of connectivity to servers and other resources in varying degrees.  Of course, the user didn't know that the whole office was having trouble, and we didn't know the user was able to reproduce the symptom--our team merely observed intermittent, wide-spread network instability.  We spent most of a day troubleshooting, monitoring and testing before we and said user matched up our experiences.

It turns out that the mini-switch is Layer 2 with port-mirroring capabilities, and there was a mirror in place.  The destination port was patched to the wall jack and linked to the 1638.  When connecting a PC to a source port in the mirror, mayhem ensued!    Just took a while to find that.

I'm wondering, could we have made use of any certain debug, show commands or other techniques to more rapidly catch the cause?  I have heard of (especially larger) networks experiencing bad things when some NIC goes batty, or the dreaded network loop when switches are patched twice without STP in working order.  In order to be better prepared for things like this, what can AOS devices do to help me find the needle in the haystack?  A related question:  Are there design or configuration features that might be employed to increase resiliency against these types of things?

Chris

[Anonymous · Accepted solution]

Chris,

There are several methods to troubleshoot a switching loop. It is possible that not all switches within your network have spanning-tree enabled thus leaving your network exposed to broadcast storms caused by these loops. Spanning-tree troubleshooting commands may be able to help you out to a certain degree. The command "show span" gives a good summary of what state each port is in. The commands "debug spanning-tree topology" and "debug spanning-tree events" will give you an idea if the switch is noticing any topology changes forcing spanning-tree to recalculate. The important thing to take from these commands is that spanning-tree is allowing traffic to flow through your switching network as you want it to. More details in configuring spanning-tree can be found in this document: https://supportforums.adtran.com/docs/DOC-1661

Beyond that, I find the "show arp" and "show mac address-table" commands helpful. What I am looking for in these tables are duplicate entries, which points to the fact that these entries were learned twice and from a different source. These commands allow me to trace these duplicate entries through the switch network and usually lead to the culprit device.

As far as preventing these events from hindering your network, all NetVanta switches have a feature called storm control available within them. Storm control allows the switch to monitor input and output rates on a port. If the traffic on the port exceeds a certain amount, the port begins to drop excess traffic OR, depending on the switch available, shuts the port down to prevent further network instability until traffic falls back below the threshold. The following thread on storm control provides more details on each switch's capabilities and how to go about configuring it: https://supportforums.adtran.com/message/2376

Please do not hesitate to let us know if you have any further questions.

Thanks,

Noor

[Anonymous]

Very helpful, Noor. Thanks for being so thorough!  I'll look over the spanning-tree document.

Any guidelines about setting the storm control thresholds?