cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
New Contributor III

TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

I have a couple of 908e units configured with data in on ETH 0/1, and PRI connections on NET 0/3 and 0/4 to legacy IVR equipment.  We have configured SIP service with a couple of carriers over the data connection.  This is working. 

However, we are getting a lot of these "threshold exceeded" errors.  Here's a sampling:

2013.09.25 12:45:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 12:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 12:46:25 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 12:46:30 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 12:46:43 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:01:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:01:41 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:01:54 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 13:02:07 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:16:09 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:16:13 T1.t1 0/4 DM 15 min threshold exceeded, ES 15 min threshold exceeded

2013.09.25 13:16:26 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:31:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:31:28 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:31:37 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 13:31:51 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:46:03 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:46:09 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:47:02 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:00:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:01:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:01:21 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:01:32 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:01:34 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:16:10 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:16:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:16:45 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:16:58 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:31:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:31:17 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:31:40 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:32:10 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:46:18 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:46:28 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:46:42 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:00:56 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:01:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:01:53 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:02:06 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:16:11 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:16:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:16:25 T1.t1 0/3 CSS 15 min threshold exceeded, DM 15 min threshold exceeded

2013.09.25 15:30:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:31:05 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:31:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:31:37 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:31:50 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:46:09 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:46:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:46:35 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:47:02 T1.t1 0/4 DM 15 min threshold exceeded

From my research, it seems that these could be caused by a clock source problem.  Our clock source is set to internal, since there is no T1 from a carrier involved.  Is this a correct assumption?

We are also experiencing lags in the audio between caller and recipient.  It can reach as much as a second or more, which results in the parties talking over each other.  I'm trying to determine if these two could be related.  I also have the carrier exploring this issue from their end.

Labels (3)
0 Kudos
Reply
1 Solution

Accepted Solutions
Highlighted
Honored Contributor
Honored Contributor

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

Double-check PBX5 and ensure that its primary clock is that coming from Adtran 2. If its T1s are swapped this would cause the problem you're seeing. Secondary clock on all devices should probably be internal.

Traffic isn't going to be a factor with regard to slips.

View solution in original post

0 Kudos
Reply
20 Replies
Highlighted
Honored Contributor
Honored Contributor

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

You definitely have a clocking issue.  CSS is controlled slip seconds which is likely the cause of the errored and degraded alerts.

The IVR is probably also set to internal, or to another T1 elsewhere.

Each T1 should have exactly one source of clock.  If the legacy IVR equipment is connected to a carrier or another source of T1 clock, then you probably want to have the IVR clock from that carrier and the TA908e clock from the IVR.

If the TA908e is the only TDM connection to the IVR, then you can really go either way, either set the IVR to clock from the TA908e or have the TA908e clock from the IVR, whichever is easier.

You don't want them both internal nor do you want each clocking from the other.

As far as the latency, fix the clock slips and see if it goes away.  There will be a small amount of latency inherent in an RTP-to-TDM conversion but rarely to that extent.  What do the MOS scores look like in VQM on the TA908e?

Highlighted
New Contributor III

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

Thanks, jayh.

We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.

I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?

0 Kudos
Reply
Highlighted
Honored Contributor
Honored Contributor

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

jonathanblack wrote:



We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.


You're on the right track.  The rule is that T-1 circuits are point-to-point with two ends.  Exactly one of those ends must source clock for the span.


I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?


Nothing significant in our experience and we push the TA900 series pretty hard.  Probably more impact rendering the flash on the page to view it than collecting the data but it's very well-behaved.

0 Kudos
Reply
Highlighted
New Contributor III

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

I turned on RTP monitoring (and the firewall) over the weekend.  Unfortunately, on two occasions one of my carriers started getting 504 errors (server timeout) from our end.  This wasn't immediate, but after running for several hours, with a couple of additional hours between the incidents.  After the second time, I turned it back off, and since then no further 504 errors.  I'm not 100% sure the problem was this, but I'm now reluctant to turn it back on.

The two Adtran 908e units are 1st generation.  Is it possible that RTP monitoring on the 1st gen units could cause an overload and this was fixed in 2nd gen?

0 Kudos
Reply
Highlighted
Honored Contributor
Honored Contributor

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

It could be a resource issue.  There was a software version that resulted in SIP resources not being released but that was fixed a while back.

The latest firmware for Gen. 1 is A4.11 .

Were you able to fix the timing slips and errors?

0 Kudos
Reply
Highlighted
New Contributor III

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

We're on A4.11.00.E for both.

Still working on changing the clock sources throughout.  Since these are live systems, I have to do it during maintenance windows, which aren't always the same for each system.  I actually have a total of three 908e's and 4 IVR servers all interconnected at some level.

0 Kudos
Reply
Highlighted
Valued Contributor
Valued Contributor

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

Jonathan,

I just thought I would check in with you and see if you still needed assistance.  I would make clearing the CSS the first priority.  This can affect voice quality and, in some cases, call control.  Once that is resolved we can work on the 504 responses.  We may need a long term debug capture in order to understand the reason for those failures.  Below are the common debug commands we use to determine the point of failure.

debug sip stack message

debug sip cldu

debug voice verbose

debug isdn L2-formatted

Our document Enabling Persistent Debug Logging can help you setup a debug capture that can run without closing down before the event occurs.

Thanks!

David

0 Kudos
Reply
Highlighted
New Contributor III

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

David,

Thanks for your response.  I'm still working on the clock source change.  We have a particular Dialogic board on which I'm having trouble setting the clock source.  (It's a D/480JCT-2T1, and I can't figure out how make it get clock source from one T1 versus the other.)  However, that's not an Adtran issue.

Regards,

Jonathan

0 Kudos
Reply
Highlighted
New Contributor III

Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"

Jump to solution

Ok, I've changed the clock source to what I believe is correct (only one source per span).  This seems to have cleared the Degraded Minutes, but I'm still getting Controlled Slip Seconds and Errored Seconds (1411 of each in the last 24 hours on one PRI).

0 Kudos
Reply