3448 Dropping Packets or so says the ISP, I say it's all ISP issues?

[tdssupport]

I've got two Netvant 3448's setup for a company, one at the main office on Time Warner Fiber and one at a remote office setup on Time Warner Business Cable. Between the two sites I have a VPN setup and running. The entire setup has been up and running for 3 years with few problems. Over the past 9 months I have told the ISP the remote site on TW Business has been seeing higher levels of latency or ping variance as I use PRTG to basically ping the other side of the tunnel and the businesses website. This problem seems to come and go however now I am seeing massive packet loss, 20% and greater. the packet loss is greatest on any connection between the remote site and websites or services outside of the TW network such as yahoo.com. Pings across the VPN which stays on the TW network have been very good and in fact we are still running a VOIP system over that connection. Again to be clear, connections between main and remote office which both stay on the TW network and are in the VPN appear solid, traffic headed outside of the TW network drops packets primarily at the transfer point between TW and the backbone provider although I also see packet loss at hops just outside of our network.

The ISP, Time Warner, to date has refused to accept that the problems shown above are on their network. All calls to support have resulted in the same response, "we are seeing no dropped packets on our modem, this is entirely an internal problem."

Before I stick my neck out even further with the client and getting them involved with contacting their business account rep can you think of any reasons that this is not a TW problem. Can the Netvanta 3448 itself be starting to fail in some strange way? Other than going onsite, taking them offline and placing either a basic router in place for testing or plugging in a notebook for testing is there anything else you can think of doing to disprove it's the Netvanta? The game plan right now is to pull the 3448 and plug in a notebook using the static IP information and do the same basic tracert test. This will take the site down during business hours so whatever I do, do I need to make sure I get all the possible information I can get at that time. I do have an older, lower end Netvanta model (3200) that was at that site before but was pulled as the hardware couldn't support the amount of data we were pushing through the VPN which I can also put back in place I guess.

Thoughts?

[Anonymous]

Hi tdssupport:

I don't envy you in this situation.  All too often, the burden of proof seems to fall upon the equipment support vendor.  Let's see if the cause can be determined with sufficient clarity to be resolved (whether an equipment issue or ISP issue).  I'm not sure how far away the offices are, or tolerance for downtime.  If you have no spare 3448s, then perhaps you could save the configs, exchange the configs, then swap them.  See if the issue follows the 3448 in question or remains at the site.  There are many factors, but at least you could narrow down this way.

Have you checked your interfaces?  Duplex mis-matches, MTU and other aspect can result in collisions, errors and fragmentation.  show interface eth 0/1 (or whatever is your ISP-facing interface).  Look for input error and output errors, as well as collisions.  Discards are not necessarily problematic, as explained in the article .

The link between 3448 and ISP equipment/modem may be fine, but there could be a problem further up the ISP signal chain.  You might try to traceroute some hosts and see whether there is a significant jump in latency common to any certain hop.

RapidRoute (IP fast forwarding engine) should be enabled for all IP interfaces, generally speaking.  The command ip ffe can be issued for each IP interface.  The command was enabled by default on a somewhat-recent firmware release, so you may not see the command in your config after issuing. show running-config verbose reveals the state of that command either way.  IP FFE increases performance: 

show process cpu and show process queue can be used to see whether the 3448 is keeping up with load.  I'm not sure what speed connection you have from the ISP, but the 3448 may be limited by today's standards.  I wouldn't expect the 3448 to reliably handle sustained throughput over 40-50 Mbps.  The article about discards (above) includes useful info about this aspect.

Of course, you should be running the latest stable firmware revision (R10.9 as of this reply).  ADTRAN is constantly fixing bugs and enhancing their products and I have experienced a few cases where bringing an old unit up to date solves my problem.

It may be a good idea to eliminate features and sections of your configuration that aren't used, in order to streamline performance and simplify troubleshooting.

Best,

Chris

[Anonymous]

tdssupport:

Do you have further questions on this post?  If so, please do not hesitate to reply.  I will be happy to help in any way I can.

Levi

[tdssupport · Accepted solution]

No. The ISP agreed to give the client a 30 day money back guarantee on a higher bandwidth package and let it slip that their customers with higher level packages get a higher QOS. Since that time the problem has not come up again. Granted the ISP could have put this client on a higher QOS which they might take away once the 30 days are up but I think our monitoring will flag the heck out of that.