Slow performance after R10.8.0 firmware update

[reyahtbor]

Ever since I updated our NetVanta 1534P switches to the R10.8.0 firmware (from R10.5.0), the network has slowed considerably.

If I run Wireshark on my PC and monitor the activity from me trying to use our ERP client app (which is an RDP remote app), I see it log a ton of TCP entries that state "TCP segment of a reassembled pdu" with a packet length of 91.  I assume something changed in the processing of packets with this update, but I couldn't find anything on the forum about this, at least not by searching for the Wireshark information.

Unfortunately, I can't 100% rule out a Windows update (which I also performed on all of our servers last weekend just before upgrading the 1534P switches).

Has anyone else seen this issue, and if so, can you tell me how to resolve it?

Additional information about our network environment (none of this has physically changed):

- All client PCs attach to a single gigabit switch port of a ShoreTel phone that then patches back to the 1534P PoE switch.

- All servers are in a single rack and attach to a Dell PowerConnect 5324 gigabit switch that is in the rack, and it has a single connection to one of our 2 1534P switches that all of the clients connect to.

Thanks,

Rob

6/27/2013 Update: Backed out the firmware upgrade, no change.  It turns out the issue is not with the switch.

I'm still not sure what the issue is exactly but it's confined to my hyper visor, one of the guest OS', or the physical server these run on.

6/28/2013 Update: It looks like the issue did or might have something to do with the firmware upgrade (and then subsequent downgrade) and rebooting of the switches after all.

I can't pinpoint exactly what happened as there was nothing in the event logs on either my Hyper-V server, or the guest OS' running on it, but I shutdown the guest VMs, then restart the Hyper-V server, then restarted the guest VMs, and like magic everything is performing normally!

I guess the lesson here might be, if you upgrade your network switches firmware, reboot your servers afterwards.

[jayh]

reyahtbor wrote:

Ever since I updated our NetVanta 1534P switches to the R10.8.0 firmware (from R10.5.0), the network has slowed considerably.

If I run Wireshark on my PC and monitor the activity from me trying to use our ERP client app (which is an RDP remote app), I see it log a ton of TCP entries that state "TCP segment of a reassembled pdu" with a packet length of 91.  I assume something changed in the processing of packets with this update, but I couldn't find anything on the forum about this, at least not by searching for the Wireshark information.

The "TCP segment of a reassembled pdu" is Wireshark's way of letting you know that a file or message sent over TCP was broken down into packets that fit within the MTU of the transmission path.  If you have a 1500-byte MTU and you send a 5000 byte file you will see this on all but the last packet.  It basically means, "This frame is part of a larger message, more to follow".  In Wireshark you can turn off TCP reassembly and you won't see these.  This isn't an error in and of itself. 

However, it sounds like MTU may indeed be an issue.  I would try end-to-end pings with DF flag sent to look for a chokepoint.  You should be able to get 1472 bytes end-to-end on ethernet media (1500 less the 28-byte header is 1472).  The issue may be coincidental to the switch upgrade, if you roll it back does the trouble go away?  Check release notes to see if there is anything related to MTU that could be the trouble.  MPLS, Q-in-Q VLANS, tunnels and the like all add overhead and reduce the payload size.  Make sure that you don't have any firewalls blocking ICMP and thus breaking PMTUD.  You could also try increasing the MTU on the switch globally.

[jayh]

As another thought, check your interfaces for errors and/or collisions.  It's possible that something failed to negotiate duplex properly when you rebooted. 

What I do is

sh int | i errors

sh int | i collisions

You should see zero or near-zero everywhere. If you have errors and/or collisions clear counters and look again in a few minutes.

Most modern gear is good at duplex negotiation, but occasionally it fails.

Keep in mind that if one end of a link is hard-coded with speed and duplex by configuration, the other end will fail to negotiate and default to half-duplex.  In other words if you configure 10-full or 100-full on one side of a link you must also configure it on the other side.

[Anonymous · Accepted solution]

-

I went ahead and flagged this post as "Assumed Answered". If any of the responses on this thread assisted you, please mark them as Correct or Helpful as the case may be with the applicable buttons. This will make them visible and help other members of the community find solutions more easily. If you have any additional information on this that others may benefit from, please come back to this post to provide an update. If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

Thanks,

Noor