Multi-site VOIP Issues

[jterrence]

Hello Community,

This is my first time posting here and I'm looking for some ideas/feedback on how to address a problem we are having.  The current configuration existed before I arrived but the problems have never corrected themselves.

We currently operate 32 locations all running over an MPLS network with a cloud style managed firewall.  We have our own PBX system at the corporate office that operates our phone system across this network.  Each of our 32 locations has their own T1 connection/s that carry both phone and data.  We are using 3100 and 3400 series adtran routers at each location.  Our ISP who manages the firewall has a second Adtran router that they manage that provides our QOS and traffic control for the most part.

We are having issues with phone quality on a regular basis.  This quality is accompanied by a high level of dropped packets during those times of poor quality.  We have been trying to work with out provider to have the QOS modified but it seems as though they are using QOS profiles and we can't seem to eliminate the loss of packets no matter how the modify their end.

With 32 locations and each being a smaller office we would need to be able to setup something up that works at each site.  A hard limit on traffic by IP or port would be ideal.  Our internet traffic speed is never business critical since we aren't a point of sale type business.  I've looked into some of the options on limiting by port but didn't see that as an option on the 3100 series...am I correct?(Mostly 3120s)  The 3400 series though seemed like it was possible?(3448 currently in use)

I'm open to suggestions and or configuration ideas that don't involve dumping the whole system...buying 32 new routers or trying to change our provider.  Thanks for sharing your ideas.

[Anonymous · Accepted solution]

There are a couple of threads on here about this. I'll post them shortly. If you are seeing packet loss it's usually either the phones not tagging traffic correctly or your carrier not having something setup right somewhere along the line. With that big of a network there are many places for things to break down. The equipment you have should be fine to accomplish the QOS you need. What type of PBX are you using and have you checked it's config to make sure it's marking packets correctly?

Good threads:

https://supportforums.adtran.com/message/11573#11573

https://supportforums.adtran.com/message/9760#9760

Good documents to read and verify your configs are correct:

https://supportforums.adtran.com/docs/DOC-1649


https://supportforums.adtran.com/docs/DOC-3587

[jterrence]

Thanks for the quick reply.  I'll read up on those links after I post.

We are using the Panasonic TDE200.  We actually have 2 of them operating at 2 separate locations to divide the traffic.  The packet loss is on and off which has raised my suspicion that it could be related to times of increase network traffic at that location but I haven't pinpointed that.  Thanks

[Anonymous]

Hmm a quick read on that system and it does not look like it's got alot of option for QOS. The manual I found for it actually talks about using ToS which I've not had a carrier tell me they support. Most use either the DSCP value or UDP port ranges. If you've got a PBX vendor that can help maybe they can clarify the settings on your PBX and on the phones. The guide I saw also gave examples for KX-NT300 and 200 phones. If those are what you have I'd check the settings on them. I'd recommend doing a packet capture with wireshark or having your vendor do it and see if your VOIP traffic is getting marked at all and how it's tagged. You can then go back to your provider and see if/how they are matching traffic.

Here's the guide I was reading through. Wish I could help more but I've never worked on the Panasonics

http://www.voicesonic.com/panasonic/manuals/KX-TDE_100_200_600/KXTDE_IP_Networking_Guide_100-200-600...

[jayh]

My suggestion is that you schedule a call or meeting with your MPLS provider, your firewall provider, your phone system vendor, and anyone other than you who is responsible for network configuration. 

Broad overview:

VoIP has two distinctly different flows that need to be treated differently.

• SIP (or MGCP, etc.) signaling.  These are call setup and teardown messages, hold/resume, busy lamps, and other call control messages.  This traffic wants an "assured forwarding" class of service.  If the packet needs to be delayed for half a second it isn't a big deal but make sure it is delivered because it is used to set up a call. Don't worry about speed of delivery but if you have to make a choice delay rather than drop.
• RTP voice traffic.  This is the actual speech.  These packets are small but there are a lot of them and it is critical that they be delivered in order and at a steady pace.  If there is unavoidable congestion it is better to drop a packet than delay it.  A syllable of speech delivered half a second late is worse than not being delivered at all. It wants to be in a class called "expedited forwarding".  Put these packets first but if you have to make a choice, drop rather than delay.

QoS mechanisms can be based on a number of things and this is why the MPLS provider, phone system vendor, firewall vendor and you (or if not you the person responsible for router and switch configuration) need to be on the same page. Some things used for decision making:

• Layer 2 COS bits.  This is part of the Ethernet header, not concerned with IP addresses or ports.  Eight possibilities (3 bits) 0 through 7.  Switches typically use this for QoS decisions.  Limited granularity with only 8 possibilities.
• Layer 3 DSCP. This is part of the IP header, 64 possibilities (six bits).  This is much more granular and here you can specify the difference between expedited and assured forwarding, etc.
• Layer 3 manually specified address/port/protocol.  Here you identify traffic based on the specifics of the packet such as source/destination IP and port and protocol.  Typically done at the entrance edge if the connected device isn't capable of encoding DSCP.

Once you gain knowledge of the mechanism used by the phone system, the MPLS network, and the firewall(s), then it's a matter of ensuring that the QoS is classified correctly, marked appropriately for each device and/or network, and that the forwarding decisions honor the marking.  Also necessary is reclassifying and/or stripping QoS tags applied to traffic that your management policy decides are not to be given priority (such as online video games).

Then it's a matter of configuration and testing.  You shouldn't need to replace your Adtran gear, it's a matter of setting things up correctly and keeping them that way in the event of moves/adds/changes.  I'm assuming that the corporate office has higher bandwidth than a T1 which is only good for about 20 simultaneous G.711 calls assuming no other traffic.

[jterrence]

Thanks for looking.  We have our traffic tagged and the QoS appears to be picking up the calls and placing them in the appropriate level.  When we've investigated they've said that our traffic is overwhelming the QoS and that the voice calls are getting bumped.  We could setup a limit on our router at each location that limits bandwidth by port or IP but would that accomplish what we were looking for if our provider has a QoS set up that differs?  I've opened dialogue with them today to see what else they can do. 

[jayh]

jterrence wrote:

Thanks for looking.  We have our traffic tagged and the QoS appears to be picking up the calls and placing them in the appropriate level.  When we've investigated they've said that our traffic is overwhelming the QoS and that the voice calls are getting bumped.  We could setup a limit on our router at each location that limits bandwidth by port or IP but would that accomplish what we were looking for if our provider has a QoS set up that differs?  I've opened dialogue with them today to see what else they can do. 

Useful information.  Investigate the following:

• The actual total bandwidth you've purchased isn't adequate for all of your voice traffic alone.  You need a bigger pipe.
• Your MPLS provider has limited the size of the priority traffic to less than wire speed. This is common, and typically they will want you to pay more for higher priority bandwidth even if the total pipe remains the same.  This is because the MPLS cloud provider oversubscribes the non-QoS flows.  You need a bigger priority allotment.
• Other traffic is being inappropriately tagged as priority and squeezing out legitimate voice traffic.  You can guard against this at the entrance edge by re-marking default class to best-effort even if tagged as priority by its source.

Or a combination of two or more of the above....

[Anonymous]

I couldn't see whether the QoS/DSCP values had been confirmed in this thread.  Jayh's suggestions for typical SIP treatment across MPLS are fantastic.  If your voice equipment isn't tagging with DSCP values like that (or if the MPLS carrier wants other values), then you can leverage your ADTRAN routers to add or modify DSCP tags on voice traffic using a feature of Policy Based Routing (PBR).  Basically, you can match existing DSCP-tagged packets and/or match against an ACL in order to identify SIP and RTP packets.  Then you can apply new DSCP values prior to forwarding across MPLS links.

Something like this would match SIP/RTP at ingress of Eth 0/1 based on UDP ports in an ACL, then add (or modify) the DSCP value before forwarding:

!

ip local policy route-map VOIP

!

!

interface eth 0/1

  description LAN

  ip address  192.168.100.1  255.255.255.0

  ip policy route-map VOIP

  no shutdown

!

!

route-map VOIP permit 10

  match ip address tag-voip

  set ip dscp 46

!

!

ip access-list extended tag-voip

  remark tag SIP and RTP

  permit udp any  any  eq 5060

  permit udp any  any  range 10020 11000

!

Hopefully you don't need to mess with it, but it's good to have a tool for the job if you can't otherwise match up with the MPLS provider.  If you have to settle on one tag or class for SIP and RTP, I think 46/EF (expedited forwarding) is usually best.  Most SIP agents will re-transmit if no response is received within a short amount of time, but audio is unforgiving.

Also, is the degraded audio happening in one direction or another (toward endpoint or from endpoint), or both ways?  The reason I'm asking is because a PBX or server may tag DSCP or ToS however you configure it, but endpoints may need to be handled differently.  For example, automatically configuring DSCP values via LLDP-MED in your switches or manually assigning in each phone (hopefully not).  You'll need to make sure that traffic in each direction can be identified by the carrier properly so it can be prioritized.

Best,

CJ