Help troubleshoot NV3200 problem at remote site...

[Anonymous]

I have a Netvanta 3200 that is ~7 hours away from me, and the site is reporting problems. 

I'm able to login to the 3200 via telnet, but "show process cpu" is giving me 100% usage.

System load: 1sec:99.66%  1min:100.00%  5min:100.00%  Min: 0.00%  Max: 100.00%
Context switch load: 0.34%

Full output:

                                      Invoked  Exec Time    Runtime    Load %%
Task Id    Task Name        PRI STA   (count)     (usec)     (usec)     (1sec)
1          Idle               0 W      795667       1001          0       0.00
3          PC Config          7 S      463882       1345      56222       5.62
4          PacketRouting     38 W      575872         22     651806      65.18
5          Timer             39 W     1053662         42       8735       0.87
6          Timer-00          10 W     1132233         11        787       0.08
7          Nm01               5 W           0   14019541          0       0.00
8          Clock              9 W       12904         74        113       0.01
9          FrontPanel        37 W       60211        366       3956       0.40
10         con0              39 W         352         28          0       0.00
11         ICP Session        8 W         531         34          0       0.00
12         Thread Pool        4 R        1503        162          0       0.00
13         RouteTableTick     6 R        1345        237          0       0.00
14         OSPF               6 R        1036        425          0       0.00
15         IGMPTick           6 R         857        155          0       0.00
16         IGMP-Receiver      6 W           0   13553631          0       0.00
17         IP Events         24 W        5033        108        108       0.01
18         tcptimer          22 W        3915         51        351       0.04
19         tcpinp            22 W        2636        587       4196       0.42
20         tcpout            22 W        6251        217       4104       0.41
21         WWW               19 W        1209        180          0       0.00
22         DnsProxy          16 W        1282        116          0       0.00
23         DnsTable          16 W         880         15          0       0.00
24         SnmpThread         6 R       38017         37          0       0.00
25         Port Manager       9 W       50171        128       1664       0.17
26         eth 0/1           39 W      151171       4161     220758      22.08
27         FramerBaseThre~   21 W      129714        127       1766       0.18
28         FramerBaseThre~   21 W      128210         62        807       0.08
42         FTPServer List~    5 W           1        306          0       0.00
43         SMTP Client       16 W           0        415          0       0.00
44         SNTP Client       19 W         214         69          0       0.00
47         RipOut             6 R         819         21          0       0.00
48         RipIn              6 W           3        214          0       0.00

Looks like Eth 0/1 is showing high usage, no?

eth 0/1 is UP, line protocol is UP
  Hardware address is 00:A0:C8:4A:6E:6A
  Running 802.1Q Encapsulation
  100Mb/s, negotiated full-duplex, configured full-duplex
  Last clearing of counters never
  5 minute input rate 65184920 bits/sec, 82003 packets/sec
  5 minute output rate 691960 bits/sec, 1557 packets/sec
    Queueing method: fifo
    Output queue: 0/256/0 (size/max total/drops)
    Interface Shaper: NOT ENABLED
    31308420 packets input, 3201993017 bytes
    73025 unicasts, 31235395 broadcasts, 0 multicasts input
    0 unknown protocol, 0 symbol errors, 0 discards
    0 input errors, 0 runts, 0 giants
    0 no buffer, 0 overruns, 0 internal receive errors
    0 alignment errors, 0 crc errors
    1416816 packets output, 87654194 bytes
    0 unicasts, 0 broadcasts, 0 multicasts output
    0 output errors, 0 deferred, 0 discards
    0 single, 0 multiple, 0 late collisions
    0 excessive collisions, 0 underruns
    0 internal transmit errors, 0 carrier sense errors
    0 resets, 142246 throttles

So yeah, something is abusing the Ethernet interface.  (Note that this router was rebooted maybe 10-15 minutes ago.)  At this point, I suspect there may be a bridge loop at the site - I have a managed switch onsite, but I can't access it because the abuse of the interface is causing me to lose my access to the switch management!

Is there anything I can do to try to mitigate this with the Adtran, so I can perhaps get to the switch management and then find the problem port and shut it off?

[Anonymous · Accepted solution]

:

Thank you for asking this question in the support community.  As you mentioned, the unit is being over-utilized and can't process traffic fast enough.  You can see this in the output you included.  First, in the show process cpu command the one second system load is "99.66%" used.  Furthermore, PacketRouting is "65.18%" and eth 0/1 is "22.08%." 

Then when you look at the interface Ethernet 0/1, the main thing to notice is the number of throttles. Throttles indicate the number of times an AOS device is unable to handle the incoming data rate and, as a result, proactively drops packets to get it to a rate that it can handle.  (Please, see this post for further information on throttles and possible causes:  https://supportforums.adtran.com/message/2058#2058)

Unfortunately, you are limited on how to troubleshoot this remotely, but one thing I would recommend is try to determine if a single host on the network is causing the problem.  If you issue the show ip policy-sessions and show ip policy-stats commands you can see all the sessions going through the firewall.  Often, if a single machine has a virus or is transmitting an abnormal amount of traffic, you will see a lot of policy-sessions in the firewall.  Then you can create an access-control list (ACL) to block this IP address, which might provide you a window to get to the rest of the LAN.

I hope this makes sense, but please don't hesitate to reply to this post with any additional information or questions.  I will be happy to help in any way I can.

Levi

[Anonymous]

:

I went ahead and flagged this post as “Assumed Answered.”  If any of the responses on this thread assisted you, please mark them as either Correct or Helpful answers with the applicable buttons.  This will make them visible and help other members of the community find solutions more easily.  If you still need assistance, I would be more than happy to continue working with you on this - just let me know in a reply.

Levi

[Anonymous]

I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and select another in its place with the applicable buttons.  If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

Thanks,

Noor