924e high CPU Utilization (RvSipProc0)

[answerphone]

We have a problem with our 924e 2nd gen unit giving high CPU utilization.

It seems RvSipProc0 is the problem.  Also PacketRouting is kind of high.  Here is my process list.  Any ideas? 

We are on R13-5-1-E

ABA 1 NY >show processes cpu

System load: 1sec:96.69%  1min:100.00%  5min:100.00%  Min: 0.00%  Max: 100.00%

Context switch load: 0.80%

                                      Invoked  Exec Time    Runtime    Load %%

Task Id    Task Name        PRI STA   (count)     (usec)     (usec)     (1sec)

1          Idle               0 W   365459653          7          0       0.00

3          PC Config          7 S    23654832        881       5174       0.52

4          PacketRouting     44 W   254970112          9     218607      21.86

5          Timer             46 W    53752362         13       4546       0.45

6          CallControlQue~   37 W     2850932         21         81       0.01

7          IsdnStackQueue    39 W    13011205         10        585       0.06

8          Thread Pool        4 R       28530        625          0       0.00

9          FrontPanel        43 W     5362616         74       1542       0.15

10         Driver Control     8 W         342        213          0       0.00

11         VOX               46 W   162802739         14      24954       2.50

12         CON0              46 W       18115          9         40       0.00

13         Flash Maintena~    4 W       23008        101          0       0.00

14         ICP Session        8 W       26461         32          0       0.00

15         RSTP              43 W     2726284        178        697       0.07

16         RSTP-BG           42 W          13          7          0       0.00

17         Port Manager       9 W     5003287         21         47       0.00

18         PCI Bridge        32 W     2502469          3          3       0.00

19         FramerBaseThre~   24 W    24171635          4          4       0.00

20         FramerBaseThre~   24 W    24171924          4          4       0.00

21         FramerBaseThre~   24 W    24192276          6          6       0.00

22         FramerBaseThre~   24 W    24376491         40         44       0.00

23         eth 0/3           46 W   114853809         32      11533       1.15

24         eth 0/4           46 W   108737338         41      11097       1.11

25         VoipDsp           46 W     1639272         38        253       0.03

26         VoipDspCapture     1 W           0   27202751          0       0.00

27         MLD Thread         6 W           6          5          0       0.00

28         RouteTableTick     6 R      425080         14          0       0.00

29         RouteTableTick     6 R      426868         29          0       0.00

30         IGMPTick           6 R      262380         34          0       0.00

31         IGMP-Receiver      6 W           0   27168657          0       0.00

32         IP Events         27 W      427385         61         61       0.01

33         tcptimer          25 W      640308          5          0       0.00

34         tcpinp            25 W      155943         24          0       0.00

35         tcpout            25 W      355056         12          0       0.00

36         DnsClient         19 W      133374         36         36       0.00

37         DnsProxy          19 W       72640         46          0       0.00

38         DnsTable          19 W       52076          4          0       0.00

39         PhoneManagerQu~   41 W    16014218         21       1519       0.15

40         SIP_Stack         38 W     7014736         85      25835       2.58

41         MgcpActiveQueue   39 W           0         76          0       0.00

42         WWW               22 W      337214         56          0       0.00

43         VoipDsp           46 W     1512938         16        194       0.02

44         eth 0/1           46 W   200596033         27      18067       1.81

45         eth 0/2           46 W    35879244         19       2183       0.22

46         VPN Module        46 W           1          1          0       0.00

47         SnmpThread         6 R    12075562          7          0       0.00

48         IKE                6 R       14889        177          0       0.00

49         IPSecKeyGen        4 W           0      91729          0       0.00

50         SCEP               6 W           1        182          0       0.00

51         MediaConnectio~   39 W      806825        628       1085       0.11

52         FTPServer List~    5 W           1          4          0       0.00

53         SMTP Client       19 W           0        130          0       0.00

54         SNTP Client       22 W           3         20          0       0.00

55         CLIInjectQ         6 W           0      21881          0       0.00

57         OSPF               6 W           0      20332          0       0.00

58         RipOut             6 R      253114          4          0       0.00

59         RipIn              6 W           2         43          0       0.00

60         MTEX_MEIF_Star~   40 W           0         90          0       0.00

61         MTEX_HIF_Task     40 W        2006         51          0       0.00

62         MTEX_CC_Task      40 W        3699         54          0       0.00

63         MTEX_L3IF_Star~   40 W        3825         12          0       0.00

64         MTEX_L2IF_Star~   40 W        1907         14          0       0.00

65         UDP Relay         22 W           1         81          0       0.00

66         Mcc1              46 W       58274         32          0       0.00

67         ntpd              22 W      307846        104        104       0.01

69         AUTOLINKQ          4 R        2988         62          0       0.00

70         HttpClientQ        6 W           0        163          0       0.00

71         DHCPv6            34 W           0        452          0       0.00

72         RvSipProc0        39 W    45837606        596     620098      62.01

73         UDP In            42 W    22639991        113      18458       1.85

74         PacketCapture      4 R      510391         39          0       0.00

75         DHCP Server       34 W           0        166          0       0.00

76         Flow Meter Log~   20 W      322524         39          0       0.00

77         OSPFv3             6 W           0    1718079          0       0.00

78         TWAMP-Control      6 W           0    1703492          0       0.00

79         TWAMP-Test        19 W           1         42          0       0.00

The audio us choppy for SIP users and the admin interface is basically inaccessible.  Thanks for any help.

[jayh · Accepted solution]

I suspect that the device may be under some kind of attack. Do you have a SIP access-list in place limiting SIP traffic to known endpoints?

[answerphone]

Thank you for your reply.  I am not sure if we have a SIP access-list in place.  Could you possibly point me in the right direction to get this added?  I wouldn't mind locking this unit down tight.

[apenichet]

I have seen this many times too.

1. Could be SIP Attacks. You can type in "debug sip stack messages" and log the output and if you are getting hit you will see tons of SIP Invites from different source ip's. In that case you will need to add an access list that only permits your SIP Server.

2. Could be DNS Attacks: Make sure domain-proxy is enabled. Its enabled by default the command to remove is no domain-proxy.

Good luck..

[jayh]

ip access-list standard sip-allow-list

permit [subnet and wildcard mask of local IP phones]

permit host [IP address of external SIP provider]

permit [any other IP with which you communicate SIP]

sip access-class ip "sip-allow-list" in

[jayh]

Almost certainly not DNS, see task ID 36 through 38 in original message.

[answerphone]

Thanks for all the help with troubleshooting.  I have the tools I need now to keep an eye on this unit now.

[answerphone]

Greetings. 

We are still having issues with this Adtran with what is apparently attacks from the outside world.  Maybe I can ask for some help in diagnosis and debugging.

Our ISP is asking us to forward connection logs from the unit so they can mitigate any possible attacks.  Do you know how I can pull the logs off this adtran?

One final issue I have is when I issue the "debug sip stack messages" command, it scrolls and scrolls tons of messages to the point where I can't enter in the stop command.  When I try to run "undebug all" it is still scrolling nonstop the sip connection rejects and I can't get it to register the stop command.  Is there a quicker way to get the debugging to stop?  Closing the console to COM1 port doesn't seem to help.

Thank you.

[jayh]

It sounds as if the box has been compromised, your SIP access list isn't working, or both.

Try the following:

Don't enter any debugs right away.

Issue the command "no events" to stop most of the noise.

Enter the command:

show ip policy-sessions

which should show all open sessions on the box. If this is overwhelming, try

show ip policy-sessions | i 5060

which will show only the SIP sessions but without some of the information. Look for any unrecognized IP addresses.

As far as killing debugging, the console port by default is 9600 baud. A large debug will spew for a long time. If you log in to the box by SSH instead of console, closing the SSH session will kill the debug immediately. You can kill a buggy SSH session with <enter> <~> <.>  if needed. That's enter - tilde - dot. You can also set the console speed to something higher like 115200.

Debugging SIP stack messages can be noisy on the 9600 baud console even on a working box under no attack  if it has a lot of phones with busy lamp fields, shared appearances, and the like.