Out of Order packets starting when 908e 2nd Gen was replaced with 908e 3rd Gen

[Anonymous]

Hello,

I have a customer who's 908e 2nd Gen recently failed (DSP failure) and had to be replaced. We installed a 908e 3rd Gen in it's place and loaded the exact same config onto it. It connects via direct fiber (media converter) back to our network (SIP) and is a PRI handoff on the customer side. After replacement, we started seeing a moderate amount of Out of Order packets and the customer is reporting voice quality issues. We are not seeing any discards, just out of order (I know those are dropped too after reciept). QoS was previously configured and is still applied throughout the network.

The new 908e is running R10.9.0. I know that's the initial build. Is this a bug? Is there anything that needs to be modified in the config when upgrading from Gen2 to Gen3 to overcome this?

Thanks in advance.

Adtran908e#sh voice quality-stats

         Start                              Out of  Discard   Delay  

ID       Time      Length Codec  Order    Pkts*  Avg   Max

--------------------------------------------------------------------------------

10756    7:47 AM   33:43  G711u  152     0       83    90

10788    8:11 AM   0:27   G711u  3       0       57    60

10789    8:11 AM   0:06   G711u  4       0       51    60

10790    8:11 AM   0:05   G711u  0       0       50    50

10791    8:12 AM   0:07   G711u  0       0       50    50

10792    8:12 AM   1:24   G711u  0       0       50    50

10793    8:13 AM   1:43   G711u  13      0       90    90

10794    8:16 AM   0:27   G711u  4       0       90    90

10795    8:16 AM   1:38   G711u  2       0       50    50

10796    8:16 AM   4:07   G711u  7       0       59    60

10798    8:17 AM   1:33   G711u  1       0       50    50

10797    8:17 AM   0:04   G711u  1       0       80    90

10799    8:17 AM   1:31   G711u  1       0       50    50

10801    8:17 AM   2:26   G711u  7       0       90    90

10802    8:18 AM   1:55   G711u  9       0       90    90

10804    8:19 AM   0:04   G711u  0       0       50    50

10805    8:19 AM   1:06   G711u  3       0       90    90

10806    8:19 AM   0:56   G711u  1       0       50    50

10808    8:20 AM   0:44   G711u  1       0       90    90

10807    8:20 AM   0:20   G711u  0       0       50    50

[Anonymous · Accepted solution]

If the customer is experiencing call quality issues, then those stats are indicative of that problem.  Based upon the amount of delay reflected in the stats, I would guess the customer is hearing a lot of digitized sounding and echo-y voice.

I know it is not helpful when you are told to check everything on the network again, but I would check everything on the network again.  Inbound QOS is a little harder to verify but you can easily setup match statements to make sure you are receiving traffic with specific QOS tags (I guess EF for voice).  Make sure port speed/duplex is negotiated cleanly on both sides of any Ethernet links in use for the voice traffic.  Make sure T1 timing is set and any T1 interfaces in use for voice traffic are free and clear of errors.  Monitor all other inbound traffic to the IAD and on the circuit in general.  If the circuit is shared somehow, make sure no other devices are demanding bandwidth that you can't see through the IAD.

I just went through the exact some problem, thinking all of my QOS was fine, only to pull a PCAP off of the IAD and see that for some weird reason, my SIP gateway was tagging CS6 but I had the network setup to prioritize/respect EF.  The customer experienced voice quality issues whenever their bandwidth usage spiked, which didn't make sense until I realized the QOS wasn't actually correct.

I hope this helps.

[Anonymous]

Thank you for the response! We just got the problem corrected today and it ended up being a duplex mismatch (one side was auto and the other side was forced). Not sure why the Gen3 behaved different on the same config, but I'm glad it's resolved.

[ko6728]

Another thing:  Media conversion.  Auto-neg on a GB copper switchport requires an actual electrical connection to the other side of the link.  So, if there is fiber in between (through a media converter), many times GB neg will fail.  According to the GB protocol, the next failover mode for 1000 MB (full) is 100 MB (half.)  Many folks in the industry don't realize this.  If the media converter does actually work with the auto-neg, it will have to do it on both sides of the link.

On a good note, I have yet to experience this issue with Adtrans.  Cisco?  Yup, all day long.  Sometimes, if there is a media converter, you may not even get a duplex mismatch in the log.  One side will be full (if forced) and the other half.  Each side thinks the other is good.  Remember, half on one side and full on the other will still work - there will just be re-trans.  However, there is no half duplex on 1000 MB.  It's either full, or it drops to 100 half.  Only if the 100 half fails will the next failover mode be tried: 100 full.  I know, I know.  Doesn't sound right.   Luckily, Adtran boxes are flexible, and normally "fail" the 100 half, so you get the 100 full.  Depending on the the other side of the link, it could be a crap shoot if it is indeed 100 full.

Side note:  IEEE 802.3.  There are references to the 2012 version across many sites.  However, not every hardware manufacturer has product on the shelf today that include the 1000MB (half), not to mention the 100 MB (full) (and above 100 half) in the priority list.  Again, if there is no electrical connection (copper -> to <- copper), 1000 MB (full or half) will fail - unless your media converter can neg a link on each side.