This document discusses ways to recognize and protect an AOS unit against SNMP Denial of Service (DoS) attacks. If unprotected, an AOS unit's processor may become overworked. This situation could negatively affect normal processing of legitimate network traffic during an SNMP attack. Similarly, management to the unit may be unresponsive as well. This document discuss the use of SNMP access-classes, access-groups, and the AOS firewall to effectively prevent this issue.
This document contains the following sections:
Hardware and Software Requirements
Diagnosing a SNMP Denial of Service Attack
Protecting Against SNMP-based Attacks Using the AOS Web Interface
Protecting Against SNMP-based Attacks Using the CLI
- Using an SNMP Access-Class
- Using the AOS Firewall
- Using Access-groups to Block Illegitimate SNMP Traffic
Hardware and Software Requirements
SNMP-based denial of service (DoS) attacks are possible on any AOS unit that supports SNMP as according to the AOS Feature Matrix - Product Feature Matrix. If SNMP is not-supported on the product or is disabled completely through use of the no snmp agent command, these DoS attacks are not a concern.
Note: In earlier versions of firmware, SNMP was disabled using the no ip snmp agent command.
There are many different types of SNMP-based attacks. Only the more prominent DoS attacks will be covered in this document.
Note: Before discussing these attacks, it is important to understand SNMP community strings. A SNMP community string is a text-based string that is used as a simulated "group identity" to help the unit understand what permissions the client possesses. By default, most units come with a Public read-only community string of "Public" so that a user can read data from the unit from any network connected to the internet. This means an attacker would also have access without the proper filtering. Though they could not write data to the unit, they can still tie up router resources with illegitimate requests.
All of the types of SNMP-based attacks covered here are "SNMP floods" which are a type of DoS attack. These attacks are meant to cause interruption of service to the unit or units. There are several different types of SNMP floods, but they are all predicated on spamming large amounts of traffic at an SNMP agent running on a unit to process until the unit runs out of resources to do so. When an AOS unit receives an SNMP packet destined for one of the IP addresses it owns, it has to process the packet completely and decide whether to respond, or to drop the packet and throw an error. These processes take precious CPU cycles from important operation in the unit. SNMP is also a UDP based protocol which means packets are smaller and can be generated quicker without having to establish any sort of session with the target.
A common type of SNMP flood is a "bad request" flood. In this case, an attacker will generate constant SNMP requests with little or no actual SNMP data inside of them. These packets wont be able to pull information from the unit to which they are directed, but the unit will have to process all of them assuming they are not blocked. Most of the time they will use the default community string of "public", but if the unit has a different community string that is not secured, it could possibly be discovered and compromised in the same manner. Though the unit receiving the requests will not actually be sending back data, the unit may become overwhelmed trying to respond to all of the request, even if just sending back an error.
Another common type of SNMP flood is the SNMP-walk Flood. An SNMP Walk is a request sent to a unit for all of the main SNMP information such as device health, interface types, hostname, etc. The response includes a large amount of information. To perform this, an attacked may have a unit or a group of units all requesting SNMP-walk's repeatedly from an unsecured public community string from the victim. These requests will be received as legitimate if the victim unit is insecure and therefor will cause the unit to repeatedly respond with the walk data which use a large amount of CPU cycles to process.
Even if there is not a public community string that an attacker can figure out, the unit is not protected from illegitimate SNMP traffic. Assuming an attacker can generate messages fast enough, a unit can still experience problems because it has to look at every packet to see whether it includes the correct community string or not. When the bad packets are fully examined by the unit, they are denied but CPU cycles are wasted up until then deciding whether the traffic is legitimate before that.
This is just an overview of the type of SNMP-based DoS attacks a unit can be susceptible too. Though not all are discussed here, following the steps below to secure a unit's SNMP engine should protect the unit from any others.
Diagnosing a SNMP Denial of Service Attack
SNMP based DoS attacks present many symptoms that are ambiguous and could be attributed to many different factors. While these symptoms are important to help understand the problem, it is important that they aren't strictly associated with a unit suffering from an SNMP-based DoS attack only. There are many other DoS attacks that can cause the same symptoms.
Symptoms
When an SNMP-based DoS attack is taking place on an AOS unit, you will notice one or more of the following in most cases:
- Any management traffic to the unit will be unresponsive and slow.
- This would include SSH, Telnet, HTTP and HTTPS, and management functions the unit provides like DHCP, DNS, etc.
- The WAN link's bandwidth on the unit may be saturated.
- During an SNMP flood (repeated bad SNMP requests sent at a unit), the traffic flood may become so great that it would completely saturate the bandwidth on the link. This is especially prevalent on low bandwidth links like T1 and DSL links.
- The unit's CPU utilization may be 80% plus, even up to 100%.
- The firewall policy-sessions in the unit (if the firewall is enabled) will show an abnormal amount of SNMP sessions referenced by sessions destined for an IP address owned by the QOS unit and a destination port of UDP 161.
Verifying Symptoms
Unresponsive Management: This symptom is normally obvious to a user attempting to log into the unit. If using the Web Interface, it will be very slow to load and switching pages may take a very elongated period of time. If accessing the unit by command line, when commands are typed, they will be very delayed. An administrator could also attempt to contact the unit using PING to verify connectivity and latency. In the case of an SNMP DoS attack, PINGs will most likely be very latent and may even timeout.
Note: If your unit is experiencing this symptom, it is highly recommended to use the CLI to troubleshoot and configure security measures to alleviate the problem. The Web Interface uses a considerable amount more data than SSH, Telnet, or the console, so they will be relatively more responsive.
Wan Link Bandwidth Saturation: To check if the bandwidth is saturated on the WAN link of the unit, you can either use a third party program that monitors bandwidth like netflow, or you can use the Web Interface or CLI to check the current bandwidth reading from the interface (this is explained below).
To check the WAN bandwidth in the Web Interface, navigate to the "data" menu on the left hand side and click on "IP Interfaces". Click on the corresponding WAN interface link. Once here, you can view the stats of the interface at the bottom of the page as shown below:
Examine the "30 sec. Rate bps (pkts/s)" section in both the output statistics and input statistics (note: This may be up to 5 min. rate depending on unit configuration). If this rate is much higher than normal and approaching the WAN links actual bandwidth provided by the service provider, this may be evidence of a Denial of Service attack of which could be SNMP related.
To check the interface bandwidth stats via the CLI, use the show interface <type> <slot/port> command as shown below (applicable stats in red):
#show int eth 0/1
eth 0/1 is UP, line protocol is UP
Hardware address is 00:A0:C8:00:E1:78
Internet address is 69.73.82.183, netmask is 255.255.254.0 (via DHCP)
IP MTU is 1500 bytes
BW is 100000 Kbit
100Mb/s, negotiated full-duplex, configured full-duplex
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: never
30 second input rate 9424 bits/sec, 14 packets/sec
30 second output rate 6168 bits/sec, 1 packets/sec
Queueing method
Configured Queueing Method: fifo
Effective Queueing Method: weighted fair
Output queue: 0/130/684/64/1505 (size/highest/max total/threshold/drops)
Conversations 0/11/128 (active/max active/max total)
Available Bandwidth 5250 kilobits/sec
Interface Shaper: 7000/43750/43750 (rate/budget/max budget)
875 bytes added to budget every 1 ms
packet stats: 85568490/0/1505/133684 (packets sent/waiting/dropped/delayed)
111906628 packets input, 2519085629 bytes
97692525 unicasts, 14214103 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
85569995 packets output, 3410531807 bytes
85502336 unicasts, 67659 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
If the reported bps is higher than normal or approaching the WAN link's bandwidth provider by the service provider, this may be a sign that a Denial of Service attack is taking place.
High CPU Utilization: An AOS unit's CPU can be high (above 80%) for many different reasons, many of them legitimate. This document will only cover how the CPU utilization statistics will look during a SNMP Denial of Service attack.
SNMP is a management protocol and is therefor a part of the "PC Config" CPU process. When viewing the CPU utilization, if the 5 minute utilization is high and PCconfig seems to be taking a large percentage of the available processor, this may indicate an SNMP based DoS. To view these stats in the AOS web interface, navigate to the "System" menu, and then click on "System Summary". Once there, click the link called "CPU Utilization". You should see something similar to the below:
Note: Depending on your AOS unit, the processes in this output may be different. However, "PC Config" will be present in every AOS unit.
If the PCconfig process is above 80%, this is a sign that there is a problem with the management thread that could be related to SNMP (It is important to note that even if "PC config" is not higher than 80%, this does not mean there is not an SNMP problem occurring).
To check the CPU Utilization in the CLI, use the show process cpu command:
#show process cpu
System load: 1sec:5.98% 1min:5.44% 5min:4.86% Min: 0.00% Max: 100.00%
Context switch load: 0.13%
Invoked Exec Time Runtime Load %%
Task Id Task Name PRI STA (count) (usec) (usec) (1sec)
1 Idle 0 W 2021768843 496 938859 3.45
3 PC Config 7 S 215792028 1050 34519 91.76
4 PacketRouting 38 W 207333661 8 4296 0.43
5 Timer 39 W 1970341470 19 5302 0.53
6 Thread Pool 4 W 4660 122 0 0.00
7 Timer-00 10 W 1978692752 1 1011 0.10
Note: Output is truncated.
As you can see from the above, the PC config process is 91.76% which indicates the management processes in the unit are using a large amount of the CPU to process tasks. This may be a sign an SNMP DoS attack is taking place.
Firewall Policy Sessions
This step should be used last and and combines with the earlier verification of the unit's symptoms to confirm a SNMP based attack. Check the unit's policy sessions for destination port UDP 161. If there are a large number of sessions, including sessions from IP addresses other than the dedicated SNMP server, this is a sign an SNMP attack is occurring.
To check for this in the web interface, navigate to the "Data" menu and click on "Security Zones" under the "Firewall" Header. On the bottom of the page, you should see the unit's different security zones at the bottom along with a session count represented as a number with an embedded link. Here, you can click this number next to each security zone and you will get a list of current sessions in the unit as shown below:
To check the policy sessions from the CLI, use the show ip policy-session command as shown below (note: show ipv6 policy-sessions should be used as well if IPv6 is enabled and rout-able on the unit):
#show ip policy-session
Src Vrf (if not default), Src policy class:
Protocol (TTL) [in crypto map] -> [out crypto map] Dest VRF, Dest policy-class
Src IP Address Src Port Dest IP Address Dst Port NAT IP Address NAT Port
--------------- -------- --------------- -------- ----------------- --------
Policy class "Public":
tcp (299) -> Private
105.10.100.4 50217 17.172.232.201 443 d 69.73.83.187 1038
tcp (600) -> self
104.10.100.5 44359 108.10.100.1 8922
tcp (586) -> Private
105.10.100.5 44136 23.73.150.218 80 d 69.73.83.187 44136
tcp (572) -> Private
105.10.100.5 43900 63.234.248.223 1935 d 69.73.83.187 43900
tcp (586) -> Private
105.10.100.5 44349 64.210.100.89 80 d 69.73.83.187 44349
tcp (593) -> Private
105.10.100.5 44350 68.71.220.62 443 d 69.73.83.187 44350
tcp (579) -> Private
105.10.100.5 42462 72.165.61.185 27017 d 69.73.83.187 42462
udp (579) -> Private
100.10.100.5 43129 204.93.75.136 161
udp (572) -> self
100.10.100.5 43613 204.93.75.136 161
udp (572) -> self
100.10.100.5 44338 204.93.75.136 161
udp (586) -> self
100.10.100.5 44339 204.93.75.136 161
udp (572) -> self
100.10.100.5 44341 204.93.75.136 161
udp (579) -> self
100.10.100.5 43492 208.92.211.86 161
As you can see from the above, there are a large number of SNMP sessions referenced by the destination UDP port of 161.
Final Verification
Assuming all of or the majority of the above symptoms are present, you can also use SNMP debugs to see if bad requests or repeated requests from unauthorized IP addresses are coming.
To view the current SNMP activity in the unit's CLI, use the debug snmp packet command shown below:
#debug snmp packet
Received a SNMP packet to be processed.
SNMP community packet filter validation failed.
SNMP V1 RX: GET Request PDU from 172.22.70.116:60398 (community=public)
VRF = -DEFAULT-
request id=0, error status=0, error index=0
max repetitions=0, non repetitions=0
SNMP receive failed: Unable to decode packet.
Received a SNMP packet to be processed.
SNMP community packet filter validation failed.
SNMP V1 RX: GET Request PDU from 172.22.70.116:60398 (community=public)
VRF = -DEFAULT-
request id=0, error status=0, error index=0
max repetitions=0, non repetitions=0
SNMP receive failed: Unable to decode packet
Received a SNMP packet to be processed.
SNMP community packet filter validation failed.
SNMP V1 RX: GET Request PDU from 172.22.70.116:60398 (community=public)
VRF = -DEFAULT-
request id=0, error status=0, error index=0
max repetitions=0, non repetitions=0
SNMP receive failed: Unable to decode packet.
Another simple way to verify it is SNMP causing the issue with the unit is to turn it off and see if the unit resumes normal operation. This is not a solution, but a temporary workaround until your unit is secured properly and protected from further SNMP attacks. Assuming SNMP is causing the symptoms you have noticed, turning it off will normally restore responsiveness to management traffic, CPU utilization will be back to normal, and the ill effects of the over-utilized router will cease.
To turn SNMP off in the web interface, navigate to "System" on the left hand menu and click on "SNMP". At the top of the first page, un-check the box that says "SNMP Server" as shown below:
To disable SNMP in the CLI, use the no snmp server global configuration command shown below (Note: Older versions of firmware may only accept the deprecated no ip snmp server command):
(config)#no snmp server
If the AOS unit returns to normal, you have confirmed SNMP to be the problem. Proceed to the next sections to protect the unit from further SNMP attacks.
Protecting Against SNMP-based Attacks Using the AOS Web Interface
During an SNMP attack, the Web Interface may be unresponsive. If this occurs, it may be more efficient to use the CLI. Do this using the section above Protecting Against SNMP-based Attacks Using the CLI. If using SNMP over IPv6, you must use the CLI.
- Using an SNMP Access-Class
To create an SNMP access-class, you must first create an ACL to specify what traffic to "permit" and "deny". Navigate to "Data" on the left hand menu and click the "Firewall/ACLs" link. On the presented page, click the button at the bottom that says "Configure ACLs". On the next page, add a name for the access-list and click "standard" for the type (access-classes can only use standard ACLs). This is shown below:
Click "Add New ACL" to create the ACL. On the next page, traffic selectors will need to be added. Use this to select the hosts that are permitted to access the SNMP engine of the AOS unit. Click "Add New Traffic Selector". You will see the below:
Select the "permit" option and put in the IP address or hostname of the allowed server. Once done, press the "apply" button to add it to the ACL. Note that here you should always add permit options for each IP address or range that is permitted access. The implicit deny will block all unauthorized traffic. Continue creating rules until your ACL has permitted all hosts.
Once the ACL is finished, navigate to "System" on the left hand menu and click "SNMP". Click the "Community Strings" page. Add the community string you are protecting in the proper field along with any other options and then select the access-list you created using the drop-down menu next to "access-list". This is shown below:
- Using the AOS Firewall
To block SNMP properly using the AOS firewall, you must create a firewall rule to allow permitted IP addresses using SNMP and block all other SNMP traffic. If you already have a general rule allowing SNMP traffic through the firewall, create the new rule and then delete the previous one. To do this, go to "Data" on the left hand menu and select "Security Zones" under the "Firewall" section.
Once inside, click on the security zone corresponding to your public interface (or whatever interface the illegitimate traffic is entering the unit). You should see your current rules in that security zone:
Click the "Add Policy to Zone <name>" button. On the next page, select a new policy type of "allow" and then click "continue" at the bottom. Fill the next page out by selecting a source address out for the permitted IP address for your SNMP server along with a port of UDP 161 as shown below:
Note that the "mask" field is 255.255.255.255 to specify one single address. Once this has been selected, click "apply". The resulting page will show your new rule. If you need to add more permitted IP addresses to the rule, click on it and add traffic selectors to specify the other addresses. Remember that because of the implicit deny at the bottom of the Security Zone, all other SNMP traffic will be blocked unless you have another policy allowing it.
For more information on the AOS Firewall, see Configuring the Firewall (IPv4) in AOS
Protecting Against SNMP-based Attacks Using the CLI
As stated above, during an SNMP-based attack, the CLI will be more responsive than the web interface so it is recommended to proceed using the CLI. If you still wish to configure these steps using the Web Interface, please see the section Protecting Against SNMP-based Attacks Using the AOS Web Interface. Note: The solution using access-groups can only be configured using the CLI. IPv6 options are also only available in the CLI.
An SNMP access-class is an Access Control List that is applied directly to an SNMP community string to control what IP addresses can contact the SNMP server running on the AOS unit. To configure an access-class, you must first create a standard ACL (If you are not familiar with ACLs, you can read more about them using https://supportforums.adtran.com/docs/DOC-1643). A standard ACL only defines a source address to either permit or deny using the "permit" or "deny" statements. It is recommended that all hosts that are allowed SNMP access be added as permit statements and let the implicit "deny" at the end of the access-list block all other SNMP traffic. The creation of a standard ACL is shown below:
(config)#ip access-list standard ProtectSNMP
(config-std-nacl)#permit 10.10.10.1
(config-std-nacl)#permit 10.10.10.2
(config-std-nacl)#exit
In the above example, the permitted IP addresses correspond to the the two SNMP servers on the network that are allowed to contact the AOS unit. Now this can be applied as an access-class to the unit's SNMP community string, which will be "example" for the below example:
(config)#snmp-server <community string> example ip access-class ProtectSNMP
Now the SNMP community is protected from IP addresses other than the two specified. Note: If running SNMP over IPv6, a separate IPv6 access-class must be applied. For more usage of the above command, see the document https://supportforums.adtran.com/docs/DOC-1655.
SNMP can be blocked in the Firewall using a policy-class statement as well. If you are not familiar with the AOS Firewall, please read Configuring the Firewall (IPv4) in AOS before continuing.
At the end of a policy-class, there is an implicit deny just like ACLs. Due to this, SNMP needs to be allowed through the firewall with a specific rule if it is being used. However, this rule should be specific to the hosts permitted to access the SNMP engine inside the AOS unit. An example of an insecure way to allow SNMP through the firewall is shown below:
ip access-list extended SNMP
permit udp any any eq 161
!
ip policy-class Public
allow list SNMP
The above configuration is insecure because SNMP traffic from all source addresses is allowed. It is recommended that the ACL only permit the exact hosts that need access to SNMP. This is shown below:
(config)#ip access-list extended SNMP
(config-ext-nacl)#permit udp host 10.10.10.1 any eq 161
(config-ext-nacl)#permit udp host 10.10.10.2 any eq 161
(config-ext-nacl)#exit
(config)#ip policy-class Public
(config-policy-class)#allow list SNMP
The above example only allows SNMP from the specified hosts through the firewall. Any other SNMP traffic will be denied by the implicit deny in the policy-class. Make sure no other rule you have configured allows SNMP before it would hit the implicit deny. Note: If using SNMP over IPv6, you must separately configure an IPv6 ACL and the IPv6 firewall in the same manner. For more information on IPv6 operation, see https://supportforums.adtran.com/docs/DOC-1639.
Access-groups are very similar to access-classes except instead of being tied to an AOS service, they are tied to a traffic flow direction on a single interface. To create an access-group, you create an ACL to permit or deny traffic based on "permit" and "deny" statements, and then it is applied to an interface in either the "in" or "out" direction. This is sometimes referred to as "static packet filtering". It is static because it does not keep track of sessions. All packets are checked when they come in or out the interface based on the direction the access-group is applied. An example of an access-group used to block SNMP on a public WAN connection is shown below:
ip access-list extended BlockSNMP
permit udp host 10.10.10.1 any eq 161
permit udp host 10.10.10.2 any eq 161
deny udp any any eq 161
permit ip any any
!
interface eth 0/1
ip access-group BlockSNMP in
In the above example, the ACL permit's SNMP if it is from a source IP of one of our permitted servers. The third rule in the ACL then blocks all other SNMP traffic. The last rule is added so that the implicit deny at the end of the ACL does not block all other traffic. Finally we apply it to the eth 0/1 to incoming packets. If using SNMP for IPv6, you must use IPv6 ACLs and the ipv6 access-group <name> in command. For more information on access-groups, please see https://supportforums.adtran.com/docs/DOC-5994.
Note: If you configure an access-group, you must enable IP FFE or the AOS unit may experience a hit in performance. IP FFE is enabled by default in versions R10.4.0 and above. For more information about IP FFE, please see https://supportforums.adtran.com/docs/DOC-5062.
After adding any of the above configurations, it is important to verify they are working properly and they have properly blocked the illegitimate traffic. These configurations can only be verified using the CLI. However, you can still check that the symptoms have cleared up using the web interface by reworking through the section Recognizing a SNMP Denial of Service Attack.
.
Though the biggest sign that the configurations were properly applied is the absence of the symptoms from the attack, it is important to verify that the configurations are working to block the traffic properly, as well as making sure they are not blocking any traffic that should be permitted to the unit.
Before verifying the specific configurations, check back through the list of symptoms to make sure they have dispersed. Use the show process cpu command to make sure the processor is back to normal as well as using show ip policy-sessions to verify that the policy-sessions that were illegitimate are now gone. Also check show interface <type> <slot/port> to check that the bandwidth on the interface has returned to normal levels. Navigate to the section Recognizing a SNMP Denial of Service Attack for more information about verifying the existence (or lack there-of) of the symptoms.
To verify any of the suggested configuration to block illegitimate SNMP are working properly, use the show ip access-list command to check that you see matches on the ACL that is used by either the access-class, the Firewall, or the access-group:
#show ip access-list
Standard IP access list BlockSNMP
permit host 10.10.10.1 log (10 matches)
permit host 10.10.10.2 log (10 matches)
The implicit deny at the bottom of the access-list does not show matched for denied traffic, so to further verify, it is recommended that you attempt to access the SNMP engine on the ADTRAN using an address not permitted in your access-list to verify it is properly denied.
SNMP-based attacks are not only prevalent form the outside public internet. A computer inside an organizations private network can become compromised and be the initiator of an SNMP-based attack for an attacker outside the organization. Because of this, it is important to secure every interface that could be susceptible to these attacks. Before deploying a router with SNMP enabled, ensure that your SNMP community strings do not include the defaults of "public" or "private". Also, ensure that any or all of the methods of securing the unit from SNMP in the above sections using access-classes, the AOS firewall, or access-groups are applied to the unit before it is deployed to prevent any of the symptoms described in this document. For more information on security best-practices recommended by ADTRAN, please see https://supportforums.adtran.com/docs/DOC-5288.
For more information on SNMP, see the document https://supportforums.adtran.com/docs/DOC-1655
For ADTRAN recommended security best practices, please see the document https://supportforums.adtran.com/docs/DOC-5288
