Intermittent issue: 1234/1238 PoE switches won't learn specific MAC addresses

[cca651]

We've had an intermittent problem for several months now where we'll discover that one of our Netvanta switches will not place a specific MAC address in it's MAC table.

The issue is typically brought to our attention when a user mentions that their downloads are slow.   We now know to check the MAC address tables of our switches and usually find that one of the switches does not show the PC's MAC address in it's table.  Sometimes the switch is the 3rd switch the packet passes through on its route to the internet, sometimes the PC is plugged directly into the switch which won't list the MAC on the port to which it is connected.   If we ping the offending switch from the PC, the IP and MAC address are listed correctly in the ARP table but the MAC is still absent from the MAC table.

Oddly, if we manually change the MAC address on the problem PC's ethernet port (to some random address) this resolves the problem.  If we then move the problem MAC to another PC which was working fine, it experiences the same problem.

We typically show 450 or so MAC addresses in our switch tables and encounter one of these problems every few weeks or so (though we may have more but don't hear from the user for some reason).

Our switch sw is not the latest - 17.09.0x or 17.08.0x.   I've looked through the fixed-issue list for the later revs but haven't found anything promising.

Thanks in advance for any ideas!

Craig

[Anonymous]

:

Thank you for asking this question in the support community.  It appears you have done significant troubleshooting, but since the issue is so random, it will be hard to diagnose.  The first thing I would recommend is upgrading the unit to newer firmware (R10.3.2).  Furthermore, when the PC is not in the mac address-table and you ping it, does the debug arp command show replies from the PC?  If you issue the clear mac address-table command on the unit that does not show the MAC, does it then install it to the table?  You mentioned that you typically have 450 MAC addresses, which should be no problem for this unit, as it supports 8,000 MAC entries.

Again, first I recommend upgrading the firmware, and then trying some of the commands mentioned above.  Please, do not hesitate to reply to this post with additional information or questions.  I will be happy to assist you in any way I can.

Levi

[cca651]

Thanks for the response, Levi.    Since my post, I've loaded the latest gen1 sw onto this switch  (so 18.01.04 is the highest version) - the soonest I can reboot the switch will be Sunday.

I actually did try clearing the MAC table on the switch today for the first time and this did allow it to enter this particular PC's address into its table.....for awhile.  I checked back an hour or 2 later and it was missing again.  For the time being, I've added a static MAC entry on the switch as a workaround (this is an edge switch which doesn't typically terminate direct connections to user PC's so I don't have to worry about the static entry here).

If I ping the PC from the switch the entry does appear in the switch ARP table, but still not in the the MAC table.    I'll post an update after the software upgrade this weekend indicating whether or not it had any effect on this issue.

Regards,

Craig

[cca651]

Update:

Upgraded all switches to 18.01.04.00 and rebooted on Sunday.   As soon as I was able I checked the status of our problem MAC address and it was missing from this one switch MAC table.  I created a static mac entry for the device.  On Monday I removed the static entry and the MAC address switched to dynamic and remained in the table for several hours, then disappeared.   I was unable to get it back into the table as a dynamic entry either by pinging the pc from the switch or the switch from the pc.  As always, it did appear in the ARP table after taking this step.

So I placed it back into the table with a static entry...

[Anonymous]

:

Thank you for providing the update.  Is it always the same MAC address that isn't learned, and the same switch that doesn't learn it?  Do other switches in the network learn this MAC address without incident?

Levi

[cca651]

In this particular instance, yes.   Of the 14 switches on the this network this is the only one having a problem with this particular MAC address - it appears correctly in all of the other tables.  As in the past, I've moved this MAC address from the user computer and placed it on my test box and the problem as always followed the MAC.

We have run across the issue on our other switches with other MAC addresses.  The only common factor being that it's always a Netvanta switch and an HP computer (we replaced most of our HP Procurve Switches with Netvanta a year and a half or so ago when we needed PoE ports - there are a few Procurves left on the network but we've yet to experience the problem on them....or ever prior to the swap-out).   Still, it's hard to believe it's a switch issue that's only effecting us.  Likewise, I have a hard time picturing the network misconfiguration which would produce this odd result...though there may very well be one!

Regards,

Craig

[cca651]

Just a final update on this issue.  Over the weekend, I replaced the switch in question and the missing MAC issue occurred on the new switch also.  I then moved our VoIP PBX off of that switch and the issue followed the PBX to it's new switch (along with the majority of phone MAC's in our voice vlan).

It would seem the number of MAC addresses in our voice vlan has a threshold over which isolated MACs in the default vlan begin to disappear.  Yesterday I happened to notice one other switch on our network which was not displaying the problem MAC (it had approximately 20 more voice vlan connections than our other switches).   Overnight I disabled ports with phones attached one-by-one until the MAC with which we're currently having problems appeared back in the switch table.  If I reenable the voice ports the MAC drops out again.  So even at night with the reduced MAC count in our default vlan due to PC's being shut down, the issue remains on switches with higher voice vlan counts).

I'll try to redistribute our phone connections to lower the MAC count on our remaining problem switch, or swap it out with another model of switch if necessary.

[johnloj]

We are having a similar or the same issue.  In our environment we intermittantly have a PC's network become slow, so slow that the machine is almost unusable.  Sometimes this happen when a PC is initally introduced to our network and in other cases it may appear 6 months after it was deployed.  In every case we change the MAC address and the PC's performance is back to normal.  I have also gone through updating firmware on all switches and it did not resolve the issue.  We have about 500 devices on the network at this location and have had this issue pop up about 20 times.  Just curious if you have been able to resolve your issue.

[cca651]

Well, this sounds like the same issue (user complains about slow PC - changing MAC address resolves problem).  Sorry to take comfort from your pain but none of my searches turned up anyone else with this issue and I was questioning my sanity. 

Unfortunately, our 'solution' isn't exactly optimal.   I had some spare switches which I deployed and reduced the load of the switches that refused to learn the specific mac(s) until I found the working threshold - it was pretty low.  I'm waiting to see if the problem reoccurs....the issue is so intermittent that it may take awhile to know whether or not we're in the clear.   Hate to say it, but I'll be looking elsewhere when it comes time to add/replace switches.

[johnloj]

OK, I was hoping someone had a solution. I am going to keep looking. If I come up with any new info I will post it on here.

[Anonymous]

:

Are all of the ADTRAN switches 1st Generation 123x Series?  What firmware versions are the units running?  Do you know if the MAC address you are having an issue with is an HP unit?  If you move that MAC address to another switch does it follow the MAC address?

Please, let me know when you get a chance, and I will try to help.

Levi

[Anonymous]

:

When you get a chance, will you please reply and attach a copy of one of the current configurations of a switch? (Please, remember to remove any information that may be sensitive to the organization).

Levi

[johnloj]

Here is a list of my switches and their firmware versions. The one with a 17 ver. Firmware was added after I upgraded the firmware of all of my switches and the problem remained. The issue was not with a HP device. I have moved the device to a different switch and the issue remained. Only changing the MAC address of the device resolved the issue. It is very sporadic.

Switch Model Part Number Firmware

NetVanta 1544 1702544G1 18.02.03.00

NetVanta 1234 1700594G1 18.01.04.00

NetVanta 1238 1700598G1 18.01.04.00

NetVanta 1238 PoE 1700599G1 18.01.04.00

NetVanta 1234 1700594G1 18.01.04.00

NetVanta 1238 PoE 1700599G1 18.01.04.00

NetVanta 1238 PoE 1700599G1 18.01.04.00

NetVanta 1234 1700594G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1238 PoE 1700599G1 18.01.04.00

NetVanta 1238 PoE 1700599G1 18.01.04.00

NetVanta 1534P 1702591G1 18.02.01.00

NetVanta 1234 PoE 1700595G1 17.08.02.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

NetVanta 1234 PoE 1700595G1 18.01.04.00

[Anonymous]

& :

Thank you for providing the requested information.  It appears you have opened an ADTRAN Technical Support ticket regarding this issue.  ADTRAN will contact you on that ticket to obtain additional information, and hopefully get to the bottom of this.  Afterwards, we can post a summary to this forum thread.

Levi

[Anonymous · Accepted solution]

:

It has recently been discovered that 1st generation NetVanta 1234 and 1238 switches have a hardware limitation that can cause slow throughput from some LAN devices.  The simplest way to confirm whether or not this issue is affecting a LAN device is to confirm that the device's MAC address is not being actively added to the switch's MAC address table (show mac address-table).  Typically, this issue is only seen when a network design contains a large number of MAC addresses (400+) in the same broadcast domain (subnet/VLAN), which happens in a broadcast domain larger than 255.255.255.0 (/24).  With a large number of MAC addresses, the possibility of hashed MAC addresses conflicting with each other is greater, which may result in MAC addresses not being added to the MAC address table properly.

There are a few workarounds for this issue:

1. Reduce the large number of MAC addresses per broadcast domain (subnet/VLAN):  By separating a large subnet/VLAN, that contains a large number of devices (MAC addresses), into smaller subnets/VLANs (such as /24 subnets), the chances of seeing this issue are greatly reduced, if not eliminated.
2. Configure static MAC entries in the switch (arp <ip address> <mac address> arpa) for devices whose MAC addresses are not being entered into the MAC address table
3. Modify the MAC addresses on the end user devices not being entered into the MAC address table

Levi

[Anonymous]

:

I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and select another in its place with the applicable buttons.  If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

Thanks,

Levi