Company and business operations commonly require the use of equipment that resides on a Private LAN to be accessible from the outside world. Utilizing port forwarding, these devices can be accessed using their specific protocol and ports numbers by configuring the router to forward or "open" these ports to the device. This document gives an overview of port forwarding and common synonyms for the operation as well as demonstrating how to properly troubleshoot port forwards that are not functioning correctly. This document should be consulted after a port forward has been configured.
If you need help configuring port forwarding using AOS, please first see Configuring Port Forwarding in AOS or the video [video] Configuring a Port Forward in AOS (NetVanta).
Sections Included in this Document
Hardware and Software Requirements
- Troubleshooting Using the AOS Web interface
- Troubleshooting Using the CLI
- What to Do if You Cannot Easily Test the Application
If the Port Forward is Still Not Operational
Hardware and Software Requirements
Port Forwards and the troubleshooting options associated with them are only available in AOS units that support the AOS Firewall operation. Please consult the AOS Feature Matrix - Product Feature Matrix to see if your unit supports this operation.
Prior to troubleshooting a port forward or "open ports" that are not working properly, it is important to understand what port forwarding is and how it operates inside an AOS unit. The following sections cover this operation in detail. It is recommended you only proceed past this section if you completely understand the terms "Destination Network Address Translation" and "open ports". If so, proceed to the section Troubleshooting Port Forwards.
Commonly, enterprise applications in the modern network have become "hosted", which means that another company provides the servers and resources for an application that a private business uses for their personal needs. A common application a company might pay someone to "host" for them is a video streaming service. All the video application and control signaling comes from servers at the hosted company site. An internal server on the customer's LAN receives the forwarded connection from the hosted service and distributes it out to LAN clients.
In a case like the above, the company hosting the service may ask a customer to "open ports" in their internet router so that the special connections are properly allowed through the firewall. Using the phrase "open ports" is actually playing off an analogy to describe TCP/IP Layer 4 operation where a firewall has different numbered ports that, if opened, traffic can enter properly. In short, ports are actually a number inside the layer 4 header of a packet that identifies a logical connection. This is what endpoints on a connection use to distinguish one connection from another. For example, if a PC initiates a connection to a server using a source port of 1500, when the server responds, the PC will expect the traffic to come back on port 1500.
In the case of a "hosted" service, the specific connections from the hosting company will carry a certain port number so that the customer can identify the service inside the firewall. However, this traffic is not actually just allowed through the firewall as the opening ports" analogy suggests. In order to fully understand, consider the following scenario:
In the above example, the hosted service has an IP address of 65.65.65.65 which is a publicly routable IP address over the internet. The customer themselves has an internet connection, but the internal server that the hosted service must contact is behind that firewall and has a private IP address. In this case, any ports the hosted service uses need to be "opened" in the firewall. However, if the traffic is simply allowed through the firewall which opening a port seems to suggest, the connection will not work properly. This is because the hosted service must have a destination IP address for the internal server that it can use to contact it. 10.10.10.1 is a private address specified by RFC 1918 and therefor can not be routed across the internet. In this case, the hosted service must use the public IP address the customer has, 1.1.1.1. So when the hosted service initiates a connection, it will point the traffic to 1.1.1.1. How does it make it to the server?
This is where port forwarding takes place. By setting up the port forwards in the AOS Firewall, the customer is telling the firewall that if it receives a connection on the specified ports, the firewall will forward the traffic to the internal server's private IP address (this is where the term "port forward" comes from). The firewall does this by modifying the destination address in the packet to be the internal server's IP address instead of the firewall's public address through something called Network Address Translation (NAT). This is shown in the picture below:
As you can see, a packet is sent from the hosted service with a destination IP address of 1.1.1.1. When the firewall receives the traffic, it will recognize the port numbers labeling the service and, according to it's firewall rules, forward the traffic on those ports to the internal server by changing the destination IP address to 10.10.10.1. When that internal traffic responds, the same thing happens in the opposite order using the AOS firewall. When the firewall receives the response traffic, it will translate the address back to the public address.
Regarding the above information, it is very important to understand that this is what is meant if a company requires traffic to be "allowed through the firewall". Just allowing it will not function properly. If you observe the pictures from above, you can see if the firewall was just set to allow the traffic through, the destination address would still be 1.1.1.1. This means the firewall itself would receive it. Due to the fact that it most likely will not support the service, the traffic is dropped. This is why port forwarding is required.
Several of the terms for port forwarding are covered in the above section, but below is a more comprehensive list of other names for this operation:
- Open ports: Discussed above, traffic on these ports needs to be forwarded to a private IP in the network.
- Destination NAT: The destination part of this term means it is the destination IP address that is being translated, meaning the traffic is forwarded.
- Port Mapping: Comes from the idea of mapping different ports to different services and different destination units inside the internal network.
- Port Redirection: Traffic is being "redirected" when it is forwarded.
Though these are the most common names for this application, there are many other names for it. Many companies use a different proprietary name for the same operation. Be sure to ask the application company to describe how the traffic works so that you can make sure it is a port forward operation.
Before starting to troubleshoot, it is important to know exactly what to look for and what pitfalls there commonly are. A port forward is the equivalent of a "NAT destination" in the CLI (it is still referred to as a "port forward" in the Web Interface). It is treated just like any other firewall rule in that it matches certain traffic and then performs an operation on it (in this case translating the destination IP address). When troubleshooting, it is important to look to make sure the traffic is actually matching the specific port forward/NAT destination rule and it is properly being forwarded. This is done by looking at the router's policy-sessions which show a session for each stream of traffic entering the firewall (this includes a bi-directional stream as one stream). In the session, a source and destination IP address of the original packet should be seen, and then the NAT destination address (the private address the traffic is being forwarded to) should be seen as well. Seeing the correct representation of this that is shown in the following sections confirms the firewall rule is working properly.
- Common Pitfalls
Firewall rules are assessed on traffic in a top-down manner. If traffic doesn't match the first firewall rule, it is checked against the next and so on until an action is performed on the traffic. When a port forward (or any other firewall rule) is added to a Security Zone/Policy-class, that rule is placed at the bottom of all the existing rules. If a rule above the port forward matches the traffic that should be forwarded, the port forward rule will never take action on the traffic. This means it is important to have the port forward rule higher up than any other rules that could incorrectly match the traffic and nullify the port forward rule. Similarly, it is important to have all rules in the firewall tailored to only match the exact traffic that needs the associated operation performed on it. You can read more about this using Configuring the Firewall (IPv4) in AOS.
Similar to the above problem, a firewall rule that has no associated access-list (traffic selectors if using the Web Interface) actually matches all traffic. Though it is difficult to make this mistake in the Web Interface, this can happen by mistake using the CLI. If a user has logged into the via CLI and configured firewall policies, it is possible this mistake has occurred. Be aware that misspelling an associated access-list in a firewall policy-class in the CLI is the equivalent to this. For example, if the policy states that traffic matching the ACL named "lan_traffci" is allowed, but the correct ACL is named "lan_traffic", this typo would cause all traffic to match that rule and take the associated action nullifying all rules below it. The following sections discuss how to account for this in the Web Interface and the CLI.
A port forward requires interaction of the party the traffic is being forwarded to. In many cases, if this unit is not configured properly, it may not respond to the forwarded traffic which may cause the associated function or application to fail. This will be discussed further in the section Verifying the Local Unit is Operating Properly.
Finally, the application itself must be working properly for the port forward to function correctly. For instance, if the traffic is never making it to the firewall, it can not be forwarded. Or, if something is misconfiguration with the initiator of the application, the application will not work thought he port forward is configured correctly. This will be discussed in more detail in the following sections.
Before continuing, consider using the CLI to troubleshoot port forwards if you are somewhat familiar with CLI operation. Troubleshooting port forwards is more easily and efficiently done in the CLI>
As stated in the above section, the sessions inside the router need to be examined while the application to be port forwarded is trying initiate traffic so that you can see if it is being forwarded correctly. to view the current policy-sessions in a unit, proceed to the section "Security Zones" which is under "Data" on the left-hand menu in the AOS Web Interface. On this page, you should see the section that allows you to edit security zones at the bottom as shown below:
You can only view the sessions from one Security Zone at a time. Find the security zone your port forward traffic should enter (depending on the interface - normally your public connection) and click on the link under the column "Active Sessions" for that zone. In the above picture, it is "Public". You should see sessions inside the router like the below:
As you can see above, there are three columns. These are all attributes of a packet that is entering the Security Zone from outside the unit (like traffic coming in the public internet connection). The columns correspond to the protocol in the packet, the source and destination IP address and corresponding port of the original packet, and the NAT IP address the destination IP address is changed to if applicable (Source NAT is also possible here, but are not covered in the scope of this document).
Once here, initiate the application and then press the "Refresh" button to see if you see the traffic taking your associated action specified by the port forward rule. As an example, in the below unit we have an application that may come from any IP address and that resides on TCP destination port 6000. This needs to be forwarded to an internal server at IP address 10.10.10.1. Below shows an example of the sessions while this is working properly:
As you can see above, the last set of sessions in the unit shows traffic with a destination address of 10.19.233.1 (the IP on the incoming interface) on port TCP 6000. Under the NAT Address column, you can see that it is being forwarded to 10.10.10.1 properly. Note: It may be better to set the "rows per page" higher so that you can easily find the associated traffic).
If you do not see the proper session information for your application, take the following actions. Be aware that if you see the associated traffic, but it is just being allowed (is not showing up with a destination NAT IP address, it is most likely matching another firewall rule):
- Make sure the Security Zone is properly attached to the corresponding interface: On the Security Zones page, at the top you can select which zone is connected to which IP interface.
- Check to make sure rules above your port forward rule do not match the associated traffic: You can do this by clicking on the associated Security Zone and clicking on individual rules. At the bottom of each rule page, you should see traffic selectors which show what traffic the rule is matching.
- Make sure there are no empty access-lists referenced by firewall rules: Similar to the above, while checking each rule above your port forward, make sure traffic selectors are present in each rule. If a rule has no traffic selectors, all traffic will be matched.
- Attempt to move the rule up in the Security Zone: If you can't find a mistake, this attempts to take the "human error" out of the equation.
- If you see the traffic forwarded properly but the application is still not working: Navigate to the section If the Port Forward is Still Not Operational.
- If you do not see the session for the corresponding traffic at all: Please navigate to the section Further Steps.
If you need help logging into the AOS CLI, please see the document Accessing the Command Line Interface in AOS.
Once logged into the unit, in enable mode use the following command to look at the policy-sessions in the unit show ip policy-sessions <policy-class NAME> (Note: You should run this command directly after initiating the application to make sure you can see the traffic while it is hitting the unit:
#show ip policy-sessions Public
Src Vrf (if not default), Src policy class:
Protocol (TTL) [in crypto map] -> [out crypto map] Dest VRF, Dest policy-class
Src IP Address Src Port Dest IP Address Dst Port NAT IP Address NAT Port
--------------- -------- --------------- -------- ----------------- --------
Policy class "Public":
icmp (60) [VPN 10] -> self
10.10.100.1 1 10.19.233.1 1
udp (327) -> self
10.19.233.2 123 10.19.233.1 123
udp (383) -> self
10.19.233.4 123 10.19.233.1 123
udp (299) -> [VPN 10] Public
10.19.233.5 123 10.10.100.1 123
udp (313) -> self
10.19.233.5 123 10.19.233.1 123
udp (404) -> self
10.19.233.6 123 10.19.233.1 123
udp (285) -> [VPN 10] Public
10.19.233.9 123 10.10.100.1 123
udp (278) -> self
10.19.233.9 123 10.19.233.1 123
udp (565) -> [VPN 10] Public
10.19.233.13 123 10.10.100.1 123
udp (383) -> self
10.19.233.13 123 10.19.233.1 123
udp (579) -> [VPN 10] Public
10.19.233.14 123 10.10.100.1 123
udp (404) -> self
10.19.233.14 123 10.19.233.1 123
udp (334) -> self
10.19.233.16 123 10.19.233.1 123
tcp (600) -> self
172.22.68.48 7791 10.19.233.1 23
tcp (20) -> default
172.22.68.48 46328 10.19.233.1 6000 d 10.10.10.1 6000
As you can see in the above output, there are columns that show the original packet's (hitting the interface this policy-class is assigned to) source and destination IP address and port, as well as the protocol of the traffic. The final column shows the NAT IP address and port if the traffic was translated.
Using the output from above, consider an application that uses protocol TCP port 6000. On the unit, this traffic should be accepted coming from any IP address if it is destined for the unit's outside IP address of 10.19.233.1. All of this traffic should be forwarded to the internal application server residing at 10.10.10.1. The configuration for this would look like the below:
ip access-list extended PortForwardVideo
permit tcp any host 10.19.233.1 eq 6000
!
ip policy-class Public
nat destination list PortForwardVideo address 10.10.10.1
So using the output from the "show ip policy-sessions" command above, you can see the following session when traffic is initiated by the application:
tcp (20) -> default
172.22.68.48 46328 10.19.233.1 6000 d 10.10.10.1 6000
This shows that the application is being received because the destination IP address and port are 10.19.233.1 and 6000 respectively. Also, you can see in the NAT column, it shows "d" for destination and an address of 10.10.10.1 on port 6000. This shows the router's port forward is functioning correctly. If your policy-sessions output is very large, you can also use the "include" command:
#show ip policy-sessions | include 6000
172.22.68.48 46328 10.19.233.1 6000 d 10.10.10.1 6000
If you do not see the proper session information for your application, take the following actions. Be aware that if you see the associated traffic, but it is just being allowed (is not showing up with a destination NAT IP address, it is most likely matching another firewall rule):
- Make sure the correct Policy-class is properly attached to the corresponding correct interface: You can add a Policy-class to an interface by entering into that interface's configuration mode and using the ip access-policy <name> command.
- Check to make sure rules above your port forward rule do not match the associated traffic: Check all access-lists referenced by rules above the port forward rule in your Policy-class to make sure they don't match the associated traffic.
- Make sure there are no empty access-lists referenced by firewall rules: Similar to the above, while checking each rule above your port forward, make sure the ACL names are all spelled correctly and exist.
- Look at the policy-stats (compliments all above steps):
The command show ip policy-stats <Policy-class NAME> shows every rule in the specified policy-class and how many packets have initiated traffic using the associated rule. Though you can not exactly see the traffic that is being matched, if there are a large amount of hits on one rule and none on any of the rules below it, this may mean the ACL for that rule is incorrectly matching traffic:
#show ip policy-stats Public
Policy-class "Public":
15 current sessions (13500 max)
Discards/Allows/NAT: 0/119918/77
Entry 1 - nat destination list RDP interface vlan 1 overload
3373651 initiator bytes, 96183240 responder bytes, 75 hits
Entry 2 - allow list Local
24263648 initiator bytes, 23846761 responder bytes, 119918 hits
Entry 3 - nat destination list VideoStream address 10.10.10.1
0 initiator bytes, 0 responder bytes, 0 hits
As you can see above, even though the traffic was initiated properly, the rule "Entry 3" which is the associated port forward, was never matched. It looks as though the rule above it is incorrectly matching everything. The ACL in that rule is most likely either misspelled, matches all traffic, or does not exist.
- If you see the traffic forwarded properly but the application is still not working: Navigate to the section If the Port Forward is Still Not Operational.
- If you do not see the session for the corresponding traffic at all: Please navigate to the section Further Steps.
Many times when verifying a port forward, the company with the hosted service or the outside party in general that needs the traffic to be forwarded may not be able to properly simulate the traffic for testing purposes. In this case, there are a few things that can be done to simulate the connection. Note that if your firewall rule is only matching traffic from the applications specific IP address, these methods will not work.
If the associated protocol for the application is TCP, you can test using the Windows command prompt (or any other application that utilizes telnet and allows you to change the destination port). When you initiate the telnet connection on that port (it will most likely fail because nothing will respond), you can quickly check the policy-sessions using one of the described methods. Because this traffic will carry the destination port of the application you are attempting to use, it should match your rule and take the associated destination NAT action assuming the rule is set up correctly. This operation is shown below:
If using UDP, you can attempt to use the same test using a program that utilizes UDP and also allows you to change the destination port. A common application is TFTP. If you TFTP from a unit on the specific port and quickly observe the policy-sessions, you should be able to see the UDP traffic being forwarded to your private NAT address.
Port scanners like Nmap and sites that test "open ports" are useful in this situation, provided they are used in the proper context. Scanners and port "testers" actually send traffic to the target IP on the associated port and wait to see a response. If they receive no response, the port is deemed "closed". If they receive a response, the port is deemed "open". When a port is forwarded, the traffic will be sent to the corresponding unit that owns the destination NAT address. If this unit does not respond, the scanner will still never see a response and deem the port "closed". While this can indicate that the port is not being forwarded correctly, it is important to check the policy-sessions while using the scanner to make sure you see the session coming in and being forwarded properly using the steps from above. If you do see the session being forwarded correctly, but the port still shows as "closed", it is possible there is a problem with the local unit that the traffic is forwarded to and it will need to be troubleshot. It is important to not take the scanner as a definitive answer without observing the policy-sessions.
If the Port Forward is Still Not Operational
This section should only be used if you have verified that you are seeing the traffic correctly forwarded in the policy-session, or if you do not see the traffic at all. If the port forward (or the corresponding application) is still not operational after troubleshooting using the above sections, use the next steps to verify operation outside of the AOS unit.
If you see that the policy-sessions show the traffic being properly forwarded to the internal unit, but the application is still not working, it is important to verify that the local internal unit is working properly.
First, check that the unit has the correct default gateway. In most cases this will be the private IP address of the firewall. Outside traffic that is port forwarded will still carry the source IP address of the entity that initiated the connection. This address will be outside the internal unit's subnet, meaning it must send all return traffic to its default gateway to be routed. If this is incorrect, the traffic may not make it back to the firewall to be router back to the application source.
Make sure that the local unit is properly set up to receive and respond to the application. If the application is not running properly or the unit has not been configured to receive it, it will not function though the port forward works properly. Use the windows DOS command netstat -a on the local unit to make sure it is listening on the associated destination port for the application:
If you look in the above picture, under the column "Local Address" you will see <ip address>:<port number>. The port number here shows the port this connection is listening on. The "Proto" field shows the associated protocol to go with it.
Finally, check to make sure the local unit does not have an operating system firewall like "Windows Firewall" turned on, or make sure that it has a rule to allow the associated traffic.
If you have verified these steps, proceed to the next section.
If traffic is being forwarded properly according to the router, but the application is still not functioning, taking a packet capture on the local internal unit with a program like "Wireshark" is the next step to take. In some cases, the local unit may be listening on the associated port, but may not respond for some other reason. A capture of the packets can be consulted. Once this packet capture has been taken to verify receipt and response of the application packets, this can be provided to your IT staff or the hosted application support team to verify correct operation. Steps to take a packet capture are not covered in the scope of this document.
If none of the above troubleshooting steps work and you have correctly verified the ADTRAN unit is properly forwarding traffic, it is important to consult with the technical support staff of the application service you are using. There may be other ports that need to be open, or they may not have the correct information for your site. They will provide troubleshooting steps of their own to help resolve the issue.
If during the testing, you notice that the traffic does not seem to be hitting the firewall that should forward the port, you can verify the traffic is actually making it to the firewall by taking a packet capture in front of the unit (physically) using a port mirror or a hub and PC combo. You can also take a packet capture in the unit (be aware this will not catch anything immediately dropped by the firewall so make sure you have properly troubleshoot the firewall before hand) to make sure the traffic is making it there using the document Configuring Packet Capture in AOS.
For more information on Firewall operations, navigate to Configuring the Firewall (IPv4) in AOS
For help properly configuring a port forward, navigate to Configuring Port Forwarding in AOS
For help configuring and properly using access control lists (ACLs), navigate to Configuring IP Access Control Lists (ACLs) in AOS
