When installing multiple switches in a network, switched-network design practices and switch configuration can make a major difference to a Network's security, efficiency, and general health. This document explains consideration you should make for your network before you deploy layer 2 and layer 3 switches, basic principles you should follow when planning a network, as well as important features to enable on your switches and general best practices. While this document applies to any switch network and can be implemented after a switch network is deployed, it is recommended to use these best practices in the planning of a network design and configuration. Following these guidelines will help ensure that your network functions to the best of its abilities.
Sections Included in this Document
Types of Switches Referenced in this Document
Switch Network Design Configurations
- Physical Setup Recommendations and Documentation
- Management
- General Security Best Practices
- SNTP Configuration
- Logging
- Quality of Serivice
- VLAN Configuration
Types of Switches Referenced in this Document
Before using this document, it is important to understand the different switches that are being referenced and the different configurations that are being referenced. These are shown below. To discern which type of switch you are deploying based on its NetVanta Series and features available, use the :
Layer 2 Switch: Switches that do not have any Layer 3 capabilities aside from Management. This also includes switches that are layer 3 capable, but have been deployed as a layer 2 switch (This means that no end-units are using the switch as a gateway and routing is disabled).
Layer 3 Lite Switch: Switches that have limited Layer 3 capabilities not including routing protocols and VRRP. This includes and NetVanta 1200 series unit on R10.8.0 or higher NetVanta 1530 series units.
Layer 3 Switch: Switches that have full routing capabilities including routing protocols and VRRP. This includes the NetVanta 1540 Series and the NetVanta 1600 Series.
Switch Network Design Configurations
In the same manner that a network administrator should view and troubleshoot a network using a top-down method like the OSI model, a successful network design should follow the same concept. By properly grouping units that perform the same functions and creating a simple network hierarchy no matter the amount of units within it, a network can be set up to remain healthy, efficient, and will be much more easily to troubleshoot when problems occur.
ADTRAN recommends that a network be set up with three major layers: The core layer, the distribution layer, and the access layer. These three represent units that all have similar functions and work together to achieve the same goals while providing the necessary separation and hierarchy that help troubleshoot the network anytime issues arise. The flow chart representing this network design concept is shown below:
Starting from the bottom of the diagram, the Access layer is the most high density layer in terms of ports and units. This encompasses the ports that the users actually connect to. It provides functions like PoE, port security, VLAN assignment, and LLDP. This layer normally is completely made up of Layer 2 switches as very limited to no routing is done in this layer. As shown above, the layer is separated into "access switch clusters" which represent 1 or more access layer switches that work together to provide redundancy to the higher distribution layer and also generally will share the same VLAN configuration to simplify configuration and provide natural VLAN separation. There is not a limit to the number of switches per cluster, nor is there a limit or recommendation to the number of access switch clusters per distribution cluster (which will be discussed shortly below). The placement and density are up to the network designer, but should be built with the physical places the switches need to go, the VLAN configuration, and network simplicity in mind. For example, using the above design diagram, a company may have 3 buildings within their company which each have two access switch cluster. Each of these clusters house two separate departments with different VLANs and end user needs. In this case, all the other access switch clusters don't need the VLANs for separate departments configured and they don't need to have features configured that they don't individually need. If a problem occurs in one of the clusters, it is separated completely from other clusters keeping the rest of the network stable.
The middle layer of the network design is called the "distribution layer". This is the layer that controls the inter-domain routing in the network. The units in this layer serve as the default gateways for all the end-users in the network and route traffic to other sections of the network without having to send traffic to the core layer. This usually will provide traffic with general ACL filtering and CoS. Generally, this layer will consist of Layer 3 Switches like NetVanta 1500s and NetVanta 1600s that will run routing and redundancy protocols to properly route internal traffic throughout the network and provide backup routing in case of failure. Each cluster will generally consist of 2 L3 switches. Each access switch cluster will have at least one link to each switch in the distribution cluster to provide redundancy and the two switches should run VRRP to give units in the access layer a redundant gateway (This will be a layer 2 link, not routed). Each cluster will also connect to the next cluster in line (As shown in the diagram) which will represent a fully meshed connection between the 2 switches in each cluster. These will be routed Layer 3 links, hence the need to separate VLANs in the access layer as discussed above. These concepts will be further explained throughout this document.
The final top layer is the "Core Layer". This layer houses all of the WAN and MPLS routers that connect to the internet and other outside networks. Generally this layer is only made up of complete layer 3 routers. Each distribution cluster will have a set of connections (from each switch in the cluster) back to the core layer for redundancy and routing of traffic that needs to exit the internal network. The core layer, being made up of pure routers, will not be discussed in great detail in this document.
It is important to note before continuing, that these concepts should be a general guide in network setup and not perceived as the only way to set up a network design. There are many more acceptable configurations: for example Layer 3 links between Access layer clusters, increased numbers of switches in distribution clusters, and even combined core and distribution layers. The network examples above and below should be a guide for how switches in a network design should be provisioned to simplify network setup and configuration, to provide redundancy with no single point of failure, and to keep a recognizable and logical network hierarchy with an administrator making the decision to expand on the concepts how they see fit.
Below is an example of basic redundancy between the access and distribution layers:
In the above physical configuration, the two NetVanta 1544's represent a single distribution switch cluster. The two NetVanta 1238's represent a single access switch cluster (of the many that could be connection to this single distribution switch cluster). As you can see, the switches are fully meshed to each other. These will generally be layer 2 links (there are certainly acceptable configurations where these connections are routed, but this document covers the simplest configurations) meaning that they will all be trunked with the same native VLAN and the NetVanta 1238's will not make any routing decisions, and one of the two 1544's will be the 1238's end user default gateway. Spanning-tree will naturally shutdown one of the 4 links shown above, but it will be available as a backup if any part of the setup fails. For example, if the one of the link between the left most 1238 and 1544 fails, all the clients on the 1238 will still have access to the distribution layer through the backup switch.
On top of this, if VRRP is used, the end users will be protected if the master 1544 becomes unavailable for some reason (like a hardware failure) as the backup switch will take over for the master and continue routing packets. It is recommended to start with the above design example for each access switch cluster to distribution switch cluster when creating the network design, and then extra fail-over can be added if needed (as discussed before). For example, if desired, a link could be connected between the 1544's so that if a link failed, there would be more direct access between the two units. A link could also be connected between the 1238s in the access layer for the same reason. This should serve as only a basis for network building, but at its simplest form should be adequate to provide a healthy network with no obvious single point of failure.
Following the recommendation that the routing in the network be handled in the distribution layer, the distribution switch clusters will generally have layer 3 routed links between them. This does mean that no VLANs will cross the path between them (besides the transit VLAN they route on), but it offers several advantages. First, networks are easily segmented and one section of the network can be focused on during an issue rather than seeing an issue expand to all parts of the site. Also, the network will be more secure as the links between distribution switch clusters can filter traffic easily for what should be permitted to connect with other sections of the network and what shouldn't. This will also help conserve bandwidth on the network because broadcast and multicast traffic will not cross every link in the infrastructure. The connections between distribution switch clusters will generally look like the below:
The two NetVanta 1544s on the left represent a distribution switch cluster, and the two on the right represent a neighbor distribution switch cluster. These are fully meshed for redundancy with layer 3 links: this just means they use a transit VLAN to route over (something a client wont connect on). All of the ports above will be access ports (since they only carry this transit VLAN) for this VLAN, and all routes to each other will use a next hop IP address in the same transit VLAN. As you can see, no tags will cross these links (hence a layer 3 link designation) except for the transit VLAN tag. Spanning-tree can again be used to naturally shut down a link and use it as a backup for if a link fails. The meshing of the links protects against a switch in each cluster failing as well.
Though the core layer is not covered here, it is important to note that it should also follow the "mesh" design shown above. A link from each switch in a distribution cluster should go back to the core. Though the core layer will normally include pure Layer 3 units, a high density distribution layer (many switch clusters) may require a higher port density in the core layer. In this case, a NetVanta 1638 or similar could be used to aggregate all the links, and make basic routing decisions at the edge of the core. This is, however, outside the scope of this document.
It is also important to remember that too much redundancy can cause more problems than it solves. For example, say you want to protect an access switch cluster from a link to the distribution layer failing. In this case, you have a back up link to the distribution switch cluster. Commonly, it may seem more backup links just means a safer configuration. However, this creates complexity, costs more in terms of units and available ports, and creates a more complex network to troubleshoot in times of failure. Instead, it is important to identify possible failure points in your network and judge the chance of each of these points failing, and what redundancy is needed. It normally is very important to have a backup link for a critical application. However, is it worth it to have a second backup link? What are the chances that both the primary and secondary link fails? These are the questions an administrator should ask themselves when creating a network design.
This section touches on provisioning recommendations that should apply to all switches, not any specific type.
While configuration of any unit is vital to network health (and a switch is no exception), physical network setup, documentation, and planning can be extremely vital as well. Consider a company that buys 30 switches. They put them all in the same room in no orderly fashion at all, and just plug random cables into each. This most likely will work as well as a well organized switch closet assuming everything is plugged in and configured correctly. However, when a problem emerges, they may have to find the physical port a user is plugged into, or which switch the user is on, etc. This makes things extremely difficult and can waste valuable seconds while users have lost service of some kind.
There are several recommendations to overcome these issues. First and foremost is a network diagram. This has a considerable upside despite the time involved. For any network administrator, being able to look at documentation explaining how a network is set up can be vital when there is a problem. Similarly, if there are any changes in network staff, they can be easily filled in on network design and processes with hands on visual output they can use and refer back to. Finally, this can be provided to ADTRAN Support any time they are contacted to help troubleshoot network issues. this can save ADTRAN Support, and in turn a network administrator, a considerable amount of troubleshooting time. ADTRAN offers Vizio Stencils of their units front and back panels for use in creating a network design showing all connections. These can be found at https://www.adtran.com/web/page/portal/Adtran/wp_visiostencils.
Next, it is important to stack switches in a rack in an orderly fashion so that they can easily accessed from the front or back if needed. Make sure space is left in between each for cooling reasons. Second, it is recommended that every port, or at least series of ports and switches are labeled with a clear label that gives a name of a switch or port and what type of client(s) are on it. For example, a label on a switch might read "Switch 1_Floor 1_East_Phones". A port label (attached to the cord plugged into it) might read "office5_phone&PC". This will keep things very organized and make it easier to physically troubleshoot if any issues come up later. Any time spent in setting this up will surely be saved later if troubleshooting is needed.
Inside the switch itself, these things can be done as well so that an administrator who logs into a switch can also get an overview of who is plugged into what switch and what port. It is recommended that the hostname of the switch correspond with the switches label and clients just as the physical label would so that the switch is easily identified if logged into:
(config)#hostname Switch 1_Floor 1_East_Phones
Labels should also be applied to each port in the configuration so that they can easily be identified using the description <string> command:
(config)#interface sw 0/1
(config-sw 0/1)#description office5_phone&PC
IP addressing should follow a simple scheme as well, at least in the management domain. Normally if the switches are numbered, these will be the numbers that fill the fourth octet of the management subnet. For example, if management is in the 10.10.10.0 /24 subnet, switch 1 might be 10.10.10.1 and switch 2 might be 10.10.10.2 and so on making it easy to know a unit's management address.
On any ADTRAN switch deployed, there will be at least one VLAN that is active with an IP address that the switch can be managed at. When provisioning switches, ADTRAN recommends that there be a separate, non-user-traffic VLAN that is used explicitly for management. There are several reasons for this. First, ADTRAN switches process management traffic with the lowest priority when compared to network traffic and processes related to network health. For instance, if a unit's CPU becomes over-utilized to a certain degree and processes have to be ignored for a slight period of time, management processes such as console, telnet, ICMP traffic destined to the unit, etc. will be ignored before critical network traffic. Similarly, each particular VLAN in the unit can use a certain percentage of the unit's CPU resources at any given time before processes like management are dropped (so that these processes don't affect critical processes in other VLANs). This means that if you set up management in the same VLAN that your higher bandwidth traffic resides, if this VLAN starts to push its resource allocation, management processes may be dropped or delayed for a short period of time even if the overall CPU is not.
Taking this into consideration, a separate VLAN for management will allow the unit to allocate CPU resources directly for management. These resources will still not affect critical processes in other VLANs, but it will keep the management processes from being ignored in bursts of over-usage in a particular VLAN.
Furthermore, using a separate management VLAN makes management security much easier because you do not have to block traffic to certain IPs. Instead, you can block normal network traffic from accessing the whole management VLAN instead of individual IPs of each unit deployed. This also saves IP address space in other allocated subnets.
For Layer 3 and Layer 3 Lite switches, this will not have a major impact (you can choose whether to have a VLAN IP interface for your non-management VLANs or not), but in layer 2 switches this creates on issue. These switches only have one management VLAN meaning it would have to be the one that is enabled. The implications of this are mild, but consider if a unit must be locally accessed, but there are no ports that are assigned to the particular VLAN management uses? In this case, the switch could not be accessed without routing. In this case, two things can be done: 1. Make a console cable available for each switch deployed so that they can be accessed without IP considerations. 2. Leave one port on each switch as an access port to the management VLAN that can be plugged into by anyone needing to troubleshoot the switch locally. A console cable is the preferred method.
Some switches, like the NetVanta 1638, have dedicated management ports that are not on the same routing plane as any VLAN interfaces configured in the unit. This should be used for the management subnet whenever possible as it will help avoid the concerns stated above.
As with any AOS unit, security is very important during deployment. Access to the unit should be protected and controlled, passwords should be encrypted in the configuration, and all services on the unit should be properly restricted and protected from attackers. This document will not cover specific recommendations. ADTRAN recommends you follow all the recommendations in when deploying AOS switches.
In a healthy network, time-stamps are very important. Anytime network problems occur and a network administrator starts to troubleshoot the problems, time-stamps on event messages and debugs can make a huge difference to fixing the issues at hand. Also, time synchronization among units ensures that these units will be working together in harmony in a network (For example, Ncommand MSP, ADTRAN's AOS management system, keys off time and can report incorrect information if time is not properly kept).
It is recommended that all switches in the network be set up to use an SNTP server. It is further recommended that the main core router or switch in each section of the network be configured as a SNTP/NTP time server and that the switches below that unit in the network design use it as a SNTP server. This is preferred over pointing all units at a publicly controlled network time server as these units wont be using internet bandwidth to contact it. Depending on the amount of units in your switch network design, it can be re-sectioned so that each section uses the main switch/router for SNTP in that section and each main unit uses the overall core switch/router as their SNTP server.
Logging can become an afterthought during setup and normal operation of any network until a problem happens. Once an issue on a network has occurred and must be troubleshot, logs of the event messages in each unit can be vital to discovering what caused the problem.
ADTRAN recommends having a dedicated Syslog server for every deployed network. All units should be pointed to this server and set up with the desired granularity so that messages will be kept and stored. In the event of a problem on the network, the server will have already gathered a lot of relevant data. The server can be set with a retention time so that old logs are eventually flushed to save room on disk space. For more information, please see . There are many free syslogging services on the web that can be used.
If syslogging is not possible, ADTRAN recommends setting up Email Logging. This will allow a unit to see emails when certain events occur, like a port going down or an exception report being created. Be aware that the more granularity configured, the more emails that will arrive which is why syslog is a recommended option. For more information on Email Logging see .
General Quality of Service setting should always be configured at the network edge and especially where there are any "bottle necks", or places where congestion could occur because more traffic could be transmitted than the link could actually carry. An example would be an internet link that provides speeds of 10 Megabits/second, but is run on a 100M Ethernet port.
AOS switches currently do not support layer 3 QoS settings including QoS maps and shaping. However, they do utilize Class of Service (CoS) and CoS to DSCP markings and queuing systems to ensure priority traffic is transmitted properly on switch links. In addition to this, it is important to establish proper markings and "trust boundaries" (where the switch allows traffic to pass with its own marking set, or where the switch will overwrite it) on the access switches at the network edge. To see all configurations options associated with Class of Service, please see Configuring Ethernet Switch QoS and CoS in AOS.
When deciding how many VLANs or subnets will be in a network design, separation of different types of traffic is extremely important. Commonly, when designing small networks, it seems logical to put all units into one VLAN and subnet to simplify configuration and physical setup. Although this is true during setup, it can be extremely detrimental to a network if it encounters a problem that must be troubleshot. One important consideration is broadcast and multicast traffic. In a normal switch LAN, most of these packets, if not all, will be sent to all other units in the subnet/VLAN. This means if a phone is plugged in that uses multicast paging, all your data units in that same VLAN would receive that traffic, wasting your network bandwidth and resources.
Consider a network with 20 PCs and 20 phones. In this case, the network administrator sets up all of the units on the same VLAN for simplicity. Unfortunately, one of the PCs becomes infected with a virus that causes it to flood the network with broadcast traffic eating up the bandwidth for all the units in the VLAN. IN this scenario, not only are the PCs affected, but the phone users start experiencing dropped calls and call quality issues. If this network had been set up with the phones and PCs on separate VLANs, the PC virus would not have negatively affected the phones as the broadcast traffic would not cross between VLANs.
Considering these factors, ADTRAN recommends that before designing the VLAN configuration, group all units running the same services into separate VLANs. For instance, a network might have the following VLANs:
- VLAN 1: PCs
- VLAN 2: Phones
- VLAN 3: File Storage Servers
- VLAN 4: Application Servers
- VLAN 5: Wireless
- VLAN 6: Management
This will keep different types of traffic and applications separate from each other and ensure that problems with certain applications will have minimal impact on other critical applications in the network.
Note: Commonly, network designers will decide to group all servers together into one VLAN. This is not necessarily recommended against, but should come with a caveat. Provided servers do not require to communicate with one another, these should be deployed with their corresponding switchport they are connected to in "protection" mode. This is achieved with the command below:
(config)#interface sw 0/1
(config-sw 0/1)#switchport protection
Switchport protection is a state a port is set in to protect it from certain types of traffic; in this case, it is protected from all other protected ports. This means that any ports using switchport protection can not transmit to one another, but can transmit to anything else. This will allow a user to provision devices like servers in the same VLAN, but still keep the individual application traffic from each separate, protecting each device from issues other devices in that VLAN may have.
VLAN configurations on individual ports have important security and connectivity considerations. In basic setup, all ports connected to end devices should be access ports and all links between switches that will carry multiple VLANs should be trunk ports. However, as networks get more complex, so do VLAN configuration concerns. In many instances, a company may deploy a phone and a PC to each user in each office. The PC will be in a data VLAN and the phone in a voice VLAN. To conserve ports on switches, many phones have the ability to pass a network connection on to another device. In this case, one switchport on a switch would have a phone in one VLAN, with a PC plugged in behind it in a separate VLAN. This port is considered to carry multiple VLANs (which is true), so many users will configure it as a trunk port. However, this has several negative effects.
In normal switch operation, if broadcast traffic enters a switch in a VLAN, the frames are broadcast out all access ports for that VLAN, and all trunk ports that carry that VLAN. If all end ports are configured as trunks, all VLAN broadcast traffic (or any traffic that is broadcasted because ARP or MAC address entries don't exist) will be sent out that port, eating bandwidth on the port and possibly causing problems on the phone receiving and having to drop the traffic. In this case, it is recommended to use LLDP-MED, or the Voice VLAN feature.
Most enterprise phones can use LLDP-MED to negotiate a VLAN with the switch. With this, the switch can be configured with a normal access VLAN for the PC, and then designate a voice VLAN for the phone (which it will learn through LLDP messages). The advantage is that the port will be limited to the two VLANs that are configured on the link, removing the added security concerns. This is shown below:
(config)#inteface sw 0/1
(config-sw 0/1)#switchport mode access
(config-sw 0/1)#switchport access vlan 1
(config-sw 0/1)#switchport voice vlan 2
The above configuration will allow the phone to negotiate VLAN 2 and the data unit to use VLAN 1 without adding extra unneeded concerns. If the phones are not capable of LLDP and must have their VLANs hard-set in the phone configuration, this command still works as it allows the voice VLAN tag on the port regardless of LLDP-MED negotiation. If a trunk port must be configured for a scenario such as this, we recommend that the VLANs the port can carry be limited to the VLANs it actually configured using the switchport trunk allowed VLANs <input> command:
(config)#inteface sw 0/1
(config-sw 0/1)#switchport mode trunk
(config-sw 0/1)#switchporttrunk allowed vlan 1,2
As indicated above, a Layer 2 switch in the scope of this document is a switch with no routing capabilities, like a 1st generation NetVanta 1234, or a Layer 3 switch that is in "layer 2 mode" like a NetVanta 1534 with routing disabled and with no units using the switch as their default gateway. This is an important differentiation to make in when deploying switches as Layer 2 only switches (units not in the routing paths). Commonly, a network administrator could assume that having no units pointed at a switch's IP address as a default gateway would mean the switch is not routing. While this is somewhat true, Layer 3 switches can still route multicast traffic in this mode if IP route-cache express is on. If it is, all Layer 3 multicast traffic will be routed between any VLANs configured on the unit. Considering all of this, if intending to deploy a switch as a Layer 2 switch, ADTRAN recommends only enabling one VLAN (the management VLAN) on the switch and disabling routing using the global no ip routing command.
Rapid Spanning-Tree Protocol (RSTP) is automatically enabled on all AOS switches and does not need any special configurations to keep a layer 2 switched network loop free. With RSTP, convergence times in case of a failure will be within a matter of seconds without any other configuration as well. ADTRAN does not recommend turning off spanning-tree in your network, nor using a different protocol than RSTP as it is the most efficient protocol that AOS supports. However, there are many different considerations that should be made when deploying units to make spanning-tree, and your units running it the most efficient they can be, and to ensure the network has the least amount of single points of failure.
Before deploying a network, you must design it. Following the recommendations made in the section Switch Network Design Configurations a, it is recommended that each access layer switch has at least 2 paths back to the distribution layer so that one link failure does not cause a whole switch worth or more of units to lose network connectivity. Loops in a network can sometimes scare a Network Administrator because they can definitely be detrimental to network operation - without spanning-tree. Loops are actually a very healthy and important part of a functioning network as they offer a bit of insurance to any issues that could occur with a link in the network. Loops are therefor encouraged in a network provided you are using RSTP and they should not be feared when setting up a network design.
An important addendum to the above recommendations: Do not overdo your redundant links. Having extra links to provide redundancy is good, but adding too much can create an unneeded complexity to your network. For example, a network administrator might go overboard when thinking of redundancy and fully mesh the 12 switches in the company's network. Is this fully necessary? Only one link will be active using Spanning Tree, so these become wasted links unless several different links fail at one time. If added redundancy is needed, ADTRAN would instead recommend turning single links into port channels. If each access layer switch has a link back to its distribution layer switch and its sister access layer switch, these could be tripled to be a 3 link port channel to each switch. If one link fails, both connections are still up and you have more bandwidth available.
Though Spanning Tree in AOS does not require configuration to keep a network loop free, all switches must perform spanning tree calculations to keep this protocol working properly. The more efficient the RSTP configurations are, the better the switches will perform.
First off, the root switch of any layer 2 network is very important. The whole spanning "tree" is created from this switch's location in the logical network and it will perform more calculations than the individual switches in the rest of the network. There are two main considerations to this: making sure the root switch can handle the processor load and making sure the root switch is in a central part of the network. The root switch (or switches based on your design) should always reside in the distribution layer and should not be elected so that each switch is taking the least amount of switch links to the network core. Though RSTP will help you do this by default, think about picking a "root" branch of an actual tree assuming that the "core" exists at the very top. You would want to pick the most center branch at the top of the tree below the core. If you pick a lower branch near one side, some switches needlessly have to take a longer path to reach the root switch.
To ensure that a switch is the root switch as long as it is functioning normally, enter the global command spanning-tree priority 0. Spanning-tree will always elect this switch the root (make sure there is only one switch in your network set in this manner).
It is also recommended that the network has a alternate root switch as well so that any administrators will know which switch has become the root if the primary root fails. To achieve this, on the desired alternate root switch, enter the global configuration command spanning-tree priority 4096. All other switches should have their spanning-tree priorities set to default.
Individual port configurations are very important to STP processing and calculation as well. STP knows of a network topology change when a link changes state in certain ways. When this happens, the network has to perform calculations to converge to a new "tree". Each time this happens, precious CPU cycles are used to calculate all the different STP equations and processes all of the messages. To help STP out with this, a network administrator can tell STP which ports it needs to pay attention to and which to ignore changes on. This is done through a configuration option called "Edgeport mode". When a port is in Edgeport mode, this means that STP should consider this port as connected to an end user piece of equipment that does not participate in STP nor can cause a loop. When this port changes state, STP will ignore the change and will not re-converge. In a very busy network, this can save 1000s of convergence instances. Furthermore, to protect from a user accidentally plugging a switch into an edgeport and causing a loop, edgeport will still listen for STP messages and convert back to a normal switchport if a BPDU is received.
It is recommended that all ports connected to end user equipment in a network are in Edgeport mode. This can be achieved using the interface level spanning-tree edgeport command on each port.
Another concern can be deploying non-AOS switches in the same network with AOS switches. If these switches do not run RSTP/STP, or run their own proprietary protocol to prevent layer 2 loops, it is recommended these are segmented from AOS switches. All links connecting the two "segments" should be configured with the interface level spanning-tree bpdufilter enable command. This will segment the two STP domains and they can be managed separately. The only concern is that loops created by redundant links between the two segments must be manually controlled.
More information and detailed configurations options for STP can be found using the document Configuring Spanning Tree in AOS.
Besides the normal general security concerns that should be considered when deploying all switches, Layer 2 have some specific concerns that are unique to them. A big one is management. As stated above, ADTRAN recommends using a separately configured management VLAN that can not be accessed by normal network users. In layer 2 switches, because only one VLAN IP interface will be active, this should ensure local users to the switch do not have a route directly to the management VLAN without first being routed through filtering mechanisms in a network router.
However, if a port is left free for local IP management (assigned to the VLAN management is used for) or there is not a separate management VLAN, ADTRAN recommends securing the services running on the switch (HTTP, Telnet, SSH, etc.) with access-classes to keep unwanted users from attempting to manage the switch. More information on this recommendation can be found in the document Security Best Practices for AOS Products.
Provisioning any switch is an important process, but because Layer 3 switches are directly in the routing path of user network traffic, provisioning these switches properly can be the difference between a having a healthy network or a disaster waiting to happen.
An important consideration when choosing what clients a Layer 3 Switch will handle is the number of hosts a unit's route-cache can support. The majority of the NetVanta Switch line uses hardware switching to achieve line speed rates routing between subnets. However, this is based on a hardware cache that has a finite size. If more hosts are utilizing this switch as their default gateway than the hardware cache has room for, extra hosts will be handled by the switch processor which will route at much lower speeds. Taking this into consideration, use the to make decisions regarding which switch units will use as a default gateway.
The Layer 3 Lite switches are able to route using line-rate speeds provided they are not exceeding their hardware route-cache limit, but the NetVanta 1540 and 1600 series switches should be deployed where possible in critical routing areas. This is because the full layer 3 switches possess the capability to run Layer 3 redundancy, routing protocols, and hardware ACLs. These switches also have more layer 3 resources to control these and other features. In smaller networks that do not require these features, the Layer 3 Lite switches will function in the same capacity provided they are servicing the correct amount of units.
Any type of redundancy is important in a healthy network, however, Layer 3 redundancy is perhaps the most important. If a normal Layer 2 switch link fails, perhaps that switch's clients lose network connectivity. If a Layer 3 switch or main Layer 3 link fails, every single unit pointed at that switch has now lost network connectivity. This could be all of the network depending on the routing design. Layer 3 redundancy is a very simple process and requires very few setup steps, but can make a world of difference during some type of network failure.
In AOS switches, layer 3 Redundancy is provided by a protocol called Virtual Router Redundancy Protocol (VRRP). In VRRP operation, there is a defined group of routers or layer 3 switches that share a virtual IP address used by clients on the network. Of this group, one unit is elected the master and will assume forwarding operation for the network by responding to ARPs for the virtual IP address with a virtual VRRP mac address. If this unit fails, an election takes place again and the next unit becomes the master and assumes all forwarding. In this case, if there are link failures or total unit failures, traffic will still be able to flow as normal over a backup path. This section does not cover in depth detail about VRRP. For more information, please see the document Configuring VRRP in AOS.
ADTRAN recommends that every Layer 3 switch in the network, if possible, be backed up by at least one VRRP group member. In this case, if any of these units experience a failure of some type, network downtime will be kept to a minimum while the network converges to the new VRRP master. To achieve this, each main connection from the lower access switches to a main routing distribution switch should be duplicated to at least one of the backup switches so that there is no link that represents a single point of failure in the network. Consider the below illustration used from prior access to distribution layer redundancy examples:
In the above diagram, the NV 1544 on the top left is the VRRP master and all of the inter-switch links are in the same VLAN. If the NV 1544 VRRP master becomes unavailable, the the second NV 1544 will respond to ARPs for the VRRP virtual IP address and will assume forwarding for all the clients on the NV 1238s. This minimizes the downtime caused by the switch issue which gives a network administrator more time to find and troubleshoot the issue without the pressure of a service affecting situation.
Be aware that the above network design is just one example of a redundant design with minimal downtime. As you add more switches, make sure you carefully plan before making decisions such as using more than 1 redundant link. In this case, it is recommended to use port channels so that there is still a single virtual redundant link and the network complexity is not increased.
Layer 3 switches do not possess a full Layer 3 stack like a router and therefor do not possess all of the layer 3 functions that a router may possess like Firewall functions or rate limiting. However, it is still important to secure the switch itself and manage the routing performed by the switch so that VLAN to VLAN traffic is regulated.
In a Layer 3 switch, normally more than one VLAN will be configured as there may be many VLANs of clients using the switch as a default gateway. This means that in each VLAN, all services on the switch will be running and available to anyone in those VLANs. Considering this, it is recommended to secure all services using access-classes so that only certain users may access the switch's management services and only from the management VLAN.
Traffic filtering should also be configured when using multiple VLANs. By default, all VLANs that are configured in a Layer 3 switch will be able to route traffic to each other through the switch. This may not be the intended operation. Using Hardware ACLs, a network administrator can provision the Layer 3 so that only VLANs that require inter-VLAN communication can route between each other and other traffic is blocked. This keeps the network from wasting resources routing traffic that is not needed, and also can help keep different VLANs secure form problems that could arise in neighboring VLANs. Along with this, Hardware ACLs can be used to block certain clients from accessing equipment- like a user from accessing a web server- that should be isolated. ADTRAN recommends using hardware ACLs to filter traffic in the network that fits the above criteria. More information on this feature can be found in the document Configuring Hardware ACLs in AOS.
There is a common misconception with Layer 3 switches (Lite or full layer 3) that these units are basically routers with a switch in them. This is actual not the case. In fact, they technically do not route at all - they Layer 3 switch which is the equivalent of having a CAM table for IP addresses instead of MAC addresses (allowing Layer 3 to operate separately). The main differences do not affect most configurations; layer 3 switch or route, as long as the traffic is sent where it needs to go and follows a routing table it is a moot point. However, there are some differences that are important to point out before setting up a configuration and when planning for possible future expansion.
While a layer 3 switch can "route" all traffic between subnets without trouble, there are a few things they cannot do because of the lack of the full layer 3 stack a router has. The two most important of these are Layer 3 load balancing and route-maps or source routing. Load sharing is an application that allows a unit to either swap between links for certain connections, or actual alternate packets between them to utilize more than one piece of bandwidth. This is not supported on the Layer 3 switches. The larger connotation of this is seen when using routing protocols. OSPF especially allows for equal cost path load-balancing in normal operation. However, this is not available using the Layer 3 switches because of the layer 3 stack limitation. It is important before hand to manually load balance traffic over different links by setting up routes for different subnets over differ different connections. For example, if using two links and you wanted to load balance traffic to the destinations 10.10.10.0 /24 and 10.10.11.0 /24 you could manually load balance them by routing 10.10.10.0 /24 traffic over one link and 10.10.11.0 /24 over the other.
Fully layer 3 switches, like the NetVanta 1544 and NetVanta 1638, possess the ability to run routing protocols like RIP, OSPF, and even BGP. These work the same way as a router running routing protocols, but they take the routes from the protocols and push them into the hardware routing tables to be switched just like a normal static route in a layer 3 switch. This means that they have the same limitations as above with reference to load-balancing, etc. One further recommendation is to make sure to use summarization whenever possible. The switches do not possess a full layer 3 stack, and therefor are not made to hold 10,000 prefixes like a core router (depending on the model). Though they will still function in this state, preserving the processor for other layer 3 functions will be better later.
This document should be used a basis for designing a network of switches and, in turn, provisioning them, but it is not a end-all source for network information. It should be used as a guide to help a network administrator set up a network to be healthy and successful and most of all, easy to troubleshooting during an issue. The recommendations in this document may take a considerable amount of planning, time, and testing. However, this time will be easily recovered because of the ease of management, configuration, and troubleshooting when these steps are followed properly.
For information on security best practices, please see Security Best Practices for AOS Products.
For information on configuring ACL filtering in switches, please see Configuring Hardware ACLs in AOS
For information on setting up Spanning-tree protocol, please see Configuring Spanning Tree in AOS
For information regarding CoS and QoS, please see Configuring Ethernet Switch QoS and CoS in AOS
For information regarding syslogging, please see Configuring Syslog Logging in AOS
For information regarding email logging, please see Configuring Email Logging in AOS
For information on VRRP (Layer 3 Redundancy), please see Configuring VRRP in AOS.
