Problem with SNMP Polling

[ccmh]

Two of our NetVanta 3448 Routers have suddenly started going to nearly 100% CPU Utilization when we enable SNMP polling to communicate with a network monitoring software we use.  It works on the other routers, one of which is also a NetVanta 3448.  Settings all appear to be identical.  The setup worked fine for 2 years............. 

Any thoughts would be appreciated................

Thanks!

[Anonymous]

:

Thank you for asking this question in the support community.  Without seeing the configuration or the output from the show process cpu command, it would appear that you may be receiving an SNMP denial of service attack.  Here are a few recommendations:

• Disable SNMP if you are not using it (no snmp agent or no ip snmp agent (depending on firmware version))
• Configure an SNMP access-group and apply it to the public facing Internet connection to block SNMP traffic from reaching the unit's processor:

          ip access-list extended BLOCK-SNMP

            deny udp any  any eq snmp

            permit ip any  any

          !

          interface eth 0/2

           ip access-group BLOCK-SNMP in

• Upgrade the firmware to R10.7.0 and allow the ADTRAN's firewall to block the attack

I hope that makes sense, but please do not hesitate to reply to this post with any additional questions or information.  I will be happy to help in any way I can.

Levi

[Anonymous]

:

Do you have any further questions on this post? 

Levi

[ccmh]

We updated to R10.7.2 but this was not helpful. We had to shut down Solarwinds Orion Network Performance Monitor to resolve the problem. Solarwinds claims this is a bug in Adtrans's AOS but we cannot confirm why this worked for 2 years without this issue.

[Anonymous]

:

Did you try using the ACL and access-group to see if the problem was caused by a SNMP DoS attack?

Levi

[ccmh]

We know the Solarwinds software is causing the problem as when we shut down the software (Orion Network Performance Monitor) the problem with he Netvanta 3448 CPU usage stops. There is no external attack. Unfortunatly the purpose of the offending software is to monitor the network appliances for the same problems it seems to be causing...
Currently we have hidden the Netvantas in our network from the software. I am not very happy with Solarwinds at this time as they are pointing at Adtran's AOS as the problem claiming there is a "known bug".

[Anonymous]

:

I'm sorry for the trouble this is causing you, but if you have any further questions on the ADTRAN unit, please do not hesitate to reply to this post.

Levi

[Anonymous]

:

I marked this post as "assumed answered," but please do not hesitate to reply if you have further questions.

Levi

[chazh]

I have a customer with nearly the exact same problem. They are running Solarwinds 10.6.1, and also collecting cache flow data. Exactly every four hours they say the "new" routers, 3430's running r10.5.1 code nearly stop routing. We can not log into them, and pings are not returned.

They turned off SNMP monitoring and cache flow collection, and there are no problems anymore. However, they want to get SNMP and cach flow data. SolarWinds to my CSR that it is a problem with the Adtran version of OS. They did not offer a solution. We have opened a ticket with Adtran, and called several times, and sent about half dozen emails, and we have not received information back.

Has anyone found a solution to this yet? Is there an AOS that specifically addresses this issue?

[Anonymous]

:

I sent you a message requesting the ADTRAN Technical Support ticket information, so we can assist you with troubleshooting for your customer.

Levi

[Anonymous]

:

Thank you for providing the ticket information in the private message.  It appears the contact information ADTRAN had on file for the ADTRAN ticket caused problems with Technical Support reaching you.  Now that Support has that corrected, please continue to work this through the open ticket with ADTRAN Technical Support.  When the issue is resolved, this thread can be updated with the solution.

Levi

[chazh]

Thanks much, and Tyler did finally call me on a day I was in.

Regards,

Chuck Hauge

[billingslyd]

I am currently experiencing the same problem. Our network consists of 700+ Adtran 1224 and 1335 (POE and non POE) We recently installed SolarWinds as our Network Monitoring application.  After reviewing this thread we have disabled Adtran polling via Solarwinds. We had approx. 18 switches lock up over a period of 4 days before disabling in Solarwinds We thought this had corrected the issue then  have had 4-8 more lock up over the past 3 days. Wondering if the snmp walk or snmp like DOS attack left its impression on the Adtran in tying up memory resources or CPU. And this just took a few days to set it over the edge and cause it to lockup. We are running the blow FW versions on the majority of these switches and see some bugs in the software release notes where an snmp walk can cause a system lockup. I'm wondering how If this theory makes sense to anyone and what cpu or memory would be a good place to look for compare free memory heap cpu etc to give me a warning that a switch is about to lockup. Seems like a memory error moreso then a cpu but I cannot get in to 1 after it locks up so cannot be sure,   Any assistance is appreciated.

"NV1335A-17-08-02-00-E.biz"

"NONVOL:/NV1335A-17-01-01-00-E.biz"

a random site switch stats

#show memory heap
Memory Heap:
  HeapFree:   78584816
  HeapSize:   96136176

Block Managers:
  Mgr         Size        Used        Free    Max-Used    Overhead
           (bytes)    (blocks)    (blocks)    (blocks)     (bytes)
  0              0           1           3           4         128
  1             32       23291        1789       25080      802560
  2             96        7834         966        8800      281600
  3            224        1438          11        1449       46368
  4            480        1383         933        2316       74112
  5            992         215           2         217        6944
  6           2016         103           0         103        3296
  7           4064          94           6         100        3200
  8           8160          42           3          45        1440
  9          16352          33           5          38        1216
  10         32736          11           0          11         352
  11         65504          12           0          12         384
  12        131040           2           0           2          64
  13        262112           2           0           2          64
  14        524256           0           0           0           0
  15       1048544           0           0           0           0
  16       2097120           0           0           0           0
  17       4194272           0           0           0           0
  18       8388576           0           0           0           0
  19      16777184           0           0           0           0

  Total Overhead (bytes):    1221728
  Total Used (bytes):        6101056
  Total Free (bytes):         732896

#show processes cpu

System load: 1sec:7.38%  1min:7.43%  5min:7.40%  Min: 0.00%  Max: 100.00%

Context switch load: 0.17%

                                      Invoked  Exec Time    Runtime    Load %%

Task Id    Task Name        PRI STA   (count)     (usec)     (usec)     (1sec)

1          Idle               0 W    70846543       2003     924498      92.45

2          Thread Pool        2 W         793        289          0       0.00

3          PC Config          5 S    12847152       1022      49432       4.94

4          PacketRouting     36 W     5212274         15       3010       0.30

5          Timer-00           8 W    49871152          5       1189       0.12

6          Nm01               3 W           0     186296          0       0.00

7          Clock              7 W      300829         29         49       0.00

8          FrontPanel        35 W     1982627        422       8559       0.86

9          con0              37 W         483         11          0       0.00

10         CF Manager         7 W      189961         24         31       0.00

11         PCI Bridge        25 W      988712          4         67       0.01

12         Switch            37 W     4544858         13        646       0.06

13         Stacking           7 W      101948         19         19       0.00

14         SwitchQ           13 W     9356001         16       1021       0.10

15         RSTP              35 W     1224035         13       2714       0.27

16         RouteTableTick     4 W      165421         89        124       0.01

17         OSPF               4 W      233047         32        544       0.05

18         IGMPTick           4 W      100844         35         35       0.00

19         IGMP-Receiver      4 W           0    2485173          0       0.00

[kylem]

I am also having this problem on a 924e first gen. It does not appear to be an SNMP attack based on the diagnosis procedures found here: https://supportforums.adtran.com/docs/DOC-6376#CLI

Here is my config:

! ADTRAN, Inc. OS version A4.11.00.E

! Boot ROM version 14.04.00

! Platform: Total Access 924e (1st Gen), part number 4240924L1

! Serial number LBADTN0730AF192

!

!

hostname "BMB-530BB207"

enable password encrypted 141250aff3047ff9916b87537a57e90a8b95

!

clock timezone -5-Eastern-Time

!

ip subnet-zero

ip classless

ip routing

!

!

!

!

no auto-config

!

event-history on

no logging forwarding

no logging email

!

service password-encryption

!

username "i123" password encrypted "292f6b4a7bd8e8efd48be17ad71b507967cd"

!

!

ip firewall

no ip firewall alg msn

no ip firewall alg mszone

no ip firewall alg h323

!

!      

!

!

!

no dot11ap access-point-control

!

!

!

!

!

!

!

!

!

!

qos map WAN-EDGE 10

  match dscp 26 46

  priority unlimited

!

!

!

!

interface eth 0/1

  description Uplink to Radio

  ip address  152.160.44.142  255.255.255.252

  ip ffe

  ip access-group BlockSNMP in

  media-gateway ip primary

  traffic-shape rate 3000000

  qos-policy out WAN-EDGE

  no awcp

  no shutdown

!

!

interface eth 0/2

  description LAN Handoff

  ip address  152.160.47.25  255.255.255.248

  ip ffe

  no shutdown

!

!

!

!

interface t1 0/1

  shutdown

!

interface t1 0/2

  shutdown

!

interface t1 0/3

  shutdown

!

interface t1 0/4

  shutdown

!

!

interface fxs 0/1

  rx-gain +0.0

  tx-gain +0.0

  no shutdown

!

interface fxs 0/2

  rx-gain +0.0

  tx-gain +0.0

  no shutdown

!

interface fxs 0/3

  rx-gain +0.0

  tx-gain +0.0

  no shutdown

!

interface fxs 0/4

  shutdown

!

interface fxs 0/5

  shutdown

!

interface fxs 0/6

  shutdown

!

interface fxs 0/7

  shutdown

!

interface fxs 0/8

  shutdown

!

interface fxs 0/9

  shutdown

!

interface fxs 0/10

  shutdown

!

interface fxs 0/11

  shutdown

!

interface fxs 0/12

  shutdown

!

interface fxs 0/13

  shutdown

!

interface fxs 0/14

  shutdown

!

interface fxs 0/15

  shutdown

!

interface fxs 0/16

  shutdown

!

interface fxs 0/17

  shutdown

!

interface fxs 0/18

  shutdown

!

interface fxs 0/19

  shutdown

!

interface fxs 0/20

  shutdown

!

interface fxs 0/21

  shutdown

!

interface fxs 0/22

  shutdown

!

interface fxs 0/23

  shutdown

!

interface fxs 0/24

  shutdown

!

!

interface fxo 0/0

  no shutdown

!

!

!

!

!

!

!

!

ip access-list standard ACCESS-IN

  permit 216.234.96.0 0.0.1.255 log

  permit host 66.103.225.122 log

  deny   any log

  permit host 66.103.225.123 log

  permit host 216.234.103.118 log

!

ip access-list standard SIP-IN

  permit 216.234.105.72 0.0.0.7

  permit 66.103.225.88 0.0.0.7

  deny   any log

!

!

ip access-list extended BlockSNMP

  permit udp 216.234.96.0 0.0.1.255  host 152.160.44.142 eq snmp  

  deny   udp any  host 152.160.44.142 eq snmp  

  permit ip any  any   

!

!

!

!

ip route 0.0.0.0 0.0.0.0 152.160.44.141

!

no ip tftp server

no ip tftp server overwrite

ip http server

no ip http secure-server

ip snmp agent

no ip ftp server

no ip scp server

no ip sntp server

!

ip http access-class ACCESS-IN in

!

snmp-server community public RO

!

!

!

!

ip sip

ip sip udp 5060

no ip sip tcp

!

!

!

voice feature-mode local

voice transfer-mode local

voice forward-mode local

!

!

!

!

!

!

!

voice dial-plan 1 local NXX-XXXX

voice dial-plan 2 long-distance NXX-NXX-XXXX

!

!

!

!

voice class-of-service GLOBAL

  call-privilege all

!

voice codec-list GLOBAL

  default

  codec g711ulaw

!

!

!

voice trunk T01 type sip

  description "123-SIP"

  match dnis "NXX-XXXX" substitute "248-NXX-XXXX"

  sip-server primary 216.234.105.74

  registrar primary 216.234.105.74

  codec-group GLOBAL

!

!      

voice grouped-trunk SIP

  description "Outbound Calls"

  trunk T01

  accept $ cost 0

!

!

voice user 01

  connect fxs 0/1

  password encrypted "3f3b802c6899a4539120d2dbd1042cd80022"

  no call-waiting

  caller-id-override external-number 2487240441

  did "2487240441"

  no special-ring-cadences

  forward-disconnect delay 1000

  sip-authentication password encrypted "2226c76989b7cc49da5ab592196a551b8c23"

  codec-group GLOBAL

!

!

voice user 02

  connect fxs 0/2

  password encrypted "1410d98cc24448d33d16eaa509eb5aef8ed3"

  no call-waiting

  caller-id-override external-number 2487240442

  did "2487240442"

  no special-ring-cadences

  forward-disconnect delay 1000

  sip-authentication password encrypted "191d6d2607a0b620443811b254d8f6fb7ecb"

  codec-group GLOBAL

!

!

voice user 03

  connect fxs 0/3

  password encrypted "1e1aa742c2891bd23ae4dfcabae276c7347e"

  no call-waiting

  caller-id-override external-number 2487240443

  did "2487240443"

  no special-ring-cadences

  forward-disconnect delay 1000

  sip-authentication password encrypted "23276f427bfc89780d7b16d61c9b89691d63"

  codec-group GLOBAL

!

!

!

!

!

!

!

!

!

ip sip access-class SIP-IN in

!

!

!

!

!

!

!

!

!

!

!

ip sip qos dscp 46

!

!

!

!

ip rtp quality-monitoring

ip rtp quality-monitoring udp

ip rtp quality-monitoring sip

ip rtp quality-monitoring history max-streams 500

!

line con 0

  login

!      

line telnet 0 4

  login local-userlist

  no shutdown

  access-class ACCESS-IN in

line ssh 0 4

  login local-userlist

  no shutdown

  access-class ACCESS-IN in

!

sntp server 216.234.97.3

!

!

!

!

end

[billingslyd]

The key here is to know the firmware version on your system.Then look at the release notes of the newer version to see when the SNMP leak issue was resolved. I believe it was fixed in 2012 maybe 2013. Release notes will confirm.

[kylem]

It is running ADTRAN, Inc. OS version A4.11.00.E. It is the latest version for this device.