cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
New Contributor

Lose network access to admin when connected to customer network

Jump to solution

Hi all,

We're scratching our head.  Customer added a new building where we've installed four Netvanta 1550P switches.  We pre-configured and deployed, and all are working like a champ except for admin via LAN (http, ssh, etc).

When these four switches are disconnected from the rest of the customer's network, all admin works exactly as expected.  However, when we connect to the rest of customer's network, LAN admin simply stops working.  Note that all OTHER network devices in this new building remain accessible via http, as do all other non-Adtran devices in their "old" buildings.   Connection to "old" building is via fiber with yet another Adtran 1550P at far end.  We cannot access that switch either when rest of network is connected. Firmware on all switches is 12.3.4.  Serial admin functions fine at all times.  We have addresses assigned to both VLANs, no joy accessing either with rest of network connected. Customer has a third-party company managing their core network, we haven't gone too far down that road as all other devices appear to be normally accessible - only the Netvanta switches are negatively affected.  We have a fairly straight-forward config - all switches use essentially the same config, with different IP addresses:

hostname "BldgC Data cabinet B"

enable password XXXX

!

clock timezone -5-Eastern-Time

!

ip subnet-zero

ip classless

ip default-gateway 172.16.0.254

ip routing

name-server 75.75.75.75 8.8.8.8

!

ip route-cache express

!

no auto-config

!

event-history on

no logging forwarding

no logging email

!

no service password-encryption

!

username "XXXX" password "YYYY"

!

!

no dot11ap access-point-control

no dos-protection

no desktop-auditing dhcp

no network-forensics ip dhcp

!

vlan 1

  name "Default"

!

vlan 18

  name "Public Network"

!

interface gigabit-switchport 0/1

  no shutdown

  switchport access vlan 18

!

interface gigabit-switchport 0/2

  no shutdown

  switchport access vlan 18

!

interface gigabit-switchport 0/3

  no shutdown

!

interface gigabit-switchport 0/4

  no shutdown

!

interface gigabit-switchport 0/5

  no shutdown

!

interface gigabit-switchport 0/6

  no shutdown

!

interface gigabit-switchport 0/7

  no shutdown

!

interface gigabit-switchport 0/8

  no shutdown

!

interface gigabit-switchport 0/9

  no shutdown

!

interface gigabit-switchport 0/10

  no shutdown

!

interface gigabit-switchport 0/11

  no shutdown

!

interface gigabit-switchport 0/12

  no shutdown

!

interface gigabit-switchport 0/13

  no shutdown

!

interface gigabit-switchport 0/14

  no shutdown

!

interface gigabit-switchport 0/15

  no shutdown

!

interface gigabit-switchport 0/16

  no shutdown

!

interface gigabit-switchport 0/17

  no shutdown

!

interface gigabit-switchport 0/18

  no shutdown

!

interface gigabit-switchport 0/19

  no shutdown

!

interface gigabit-switchport 0/20

  no shutdown

!

interface gigabit-switchport 0/21

  no shutdown

!

interface gigabit-switchport 0/22

  no shutdown

!

interface gigabit-switchport 0/23

  no shutdown

  switchport mode trunk

!

interface gigabit-switchport 0/24

  no shutdown

  switchport mode trunk

!

interface xgigabit-switchport 0/1

  no shutdown

  switchport mode trunk

  speed auto

!

interface xgigabit-switchport 0/2

  no shutdown

  switchport mode trunk

  speed auto

!

interface xgigabit-switchport 0/3

  no shutdown

  switchport mode trunk

  speed auto

!

interface xgigabit-switchport 0/4

  no shutdown

  switchport mode trunk

  speed auto

!

interface vlan 1

  ip address  172.16.0.142  255.255.0.0

  no awcp

  no shutdown

!

interface vlan 18

  ip address  172.18.0.62  255.255.0.0

  no awcp

  no shutdown

!

no tftp server

no tftp server overwrite

http server

http secure-server

no snmp agent

no ip ftp server

no ip scp server

no ip sntp server

!

snmp-server engine local 8000029803000000000000

!

line con 0

  no login

!

line telnet 0 4

  login

  password XXXX

  no shutdown

line ssh 0 4

  login local-userlist

  no shutdown

!

sntp server us.pool.ntp.org

!

end

I'd be thrilled if anyone can point out our boneheaded mistake.  Thanks

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
Anonymous
Not applicable

Re: Lose network access to admin when connected to customer network

Jump to solution

This sounds like an issue where the CPU is handling network traffic and not responding to management interfaces ( as it is designed to do network traffic first priority).

switch# sh proc cpu will show what processes that are running.  And switch# sh proc cpu hist will show the CPU load over time.

This will sometimes give you an idea of the type of network traffic.   It could be as simple as a very high level of Multicast or Broadcast traffic.

Also the VLAN interfaces do not have -

ip route-cache express

This should not matter if they are not routing any traffic though.

To troubleshoot further, please provide this info to support.  

View solution in original post

0 Kudos
8 Replies
Highlighted
Anonymous
Not applicable

Re: Lose network access to admin when connected to customer network

Jump to solution

This sounds like an issue where the CPU is handling network traffic and not responding to management interfaces ( as it is designed to do network traffic first priority).

switch# sh proc cpu will show what processes that are running.  And switch# sh proc cpu hist will show the CPU load over time.

This will sometimes give you an idea of the type of network traffic.   It could be as simple as a very high level of Multicast or Broadcast traffic.

Also the VLAN interfaces do not have -

ip route-cache express

This should not matter if they are not routing any traffic though.

To troubleshoot further, please provide this info to support.  

View solution in original post

0 Kudos
Highlighted
New Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

While network traffic is possible, when we deployed there was virtually no traffic, hence the switch CPUs should have had ample time to respond.  One of these switches has only three or four devices connected - so I'm thinking that there is something else at play here.  I've turned off the ip route-cache express with no joy.

Cheers!

Highlighted
Honored Contributor
Honored Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

Is it your intent that this switch act as a layer 3 router between VLAN 1 and VLAN 18? If not, disable IP routing or remove the IP address from the VLAN interface you don't want to route.

From what IP address or VLAN are you trying to access the devices?

Highlighted
New Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

I've disabled IP routing and greatly dumbed down the config for diagnostics.  Current version below.

In testing we have:

  • Added another Adtran 1531P with very basic config - same phenomenon occurs:  When stand-alone all admin works normally.  Once connected to customer's network, IP admin stops working. We tried to mimic their network in our shop - all works perfectly normally.
  • Via serial connection, realized that we cannot ping other devices from the 1550.  We CAN ping our own address.  The 1550 does not reply to a ping.
  • Tried changing IP address to something way out of the normal ranges (this is a /16 network - lots of open territory), no joy/difference
  • All connected devices continue to work normally
  • Other web-enabled devices are fully and normally accessible.  It is ONLY the Netvanta switches that stop responding to IP services.

At this point, I'm fairly certain that it is something about the customer's network - but I"m at a loss to what may be wrong.  I'd welcome suggestions on a valid test.  We can insert a network monitor somewhere and capture packets for wireshark analysis.

Current config:

!

!

! ADTRAN, Inc. OS version R12.3.4

! Boot ROM version BVS1.0

! Platform: NetVanta 1550-24P, part number 17101524PF1

! Serial number XXXX

!

!

hostname "Switch"

enable password YYYY

!

!

!

ip subnet-zero

ip classless

no ip routing

!

!

no ip route-cache express

!

no auto-config

!

event-history on

no logging forwarding

no logging email

!

no service password-encryption

!

username "xxxx" password "yyyy"

!

!

!

!

!

!

no dot11ap access-point-control

no dos-protection

no desktop-auditing dhcp

no network-forensics ip dhcp

!

vlan 1

  name "Default"

!

interface gigabit-switchport 0/1

  no shutdown

!

interface gigabit-switchport 0/2

  no shutdown

!

interface gigabit-switchport 0/3

  no shutdown

!

interface gigabit-switchport 0/4

  no shutdown

!

interface gigabit-switchport 0/5

  no shutdown

!

interface gigabit-switchport 0/6

  no shutdown

!

interface gigabit-switchport 0/7

  no shutdown

!

interface gigabit-switchport 0/8

  no shutdown

!

interface gigabit-switchport 0/9

  no shutdown

!

interface gigabit-switchport 0/10

  no shutdown

!

interface gigabit-switchport 0/11

  no shutdown

!

interface gigabit-switchport 0/12

  no shutdown

!

interface gigabit-switchport 0/13

  no shutdown

!

interface gigabit-switchport 0/14

  no shutdown

!

interface gigabit-switchport 0/15

  no shutdown

!

interface gigabit-switchport 0/16

  no shutdown

!

interface gigabit-switchport 0/17

  no shutdown

!

interface gigabit-switchport 0/18

  no shutdown

!

interface gigabit-switchport 0/19

  no shutdown

!

interface gigabit-switchport 0/20

  no shutdown

!

interface gigabit-switchport 0/21

  no shutdown

!

interface gigabit-switchport 0/22

  no shutdown

!

interface gigabit-switchport 0/23

  no shutdown

!

interface gigabit-switchport 0/24

  no shutdown

!

!

interface xgigabit-switchport 0/1

  no shutdown

  switchport mode access

  speed auto

!

interface xgigabit-switchport 0/2

  no shutdown

  switchport mode access

  speed auto

!

interface xgigabit-switchport 0/3

  no shutdown

  switchport mode access

  speed auto

!

interface xgigabit-switchport 0/4

  no shutdown

  switchport mode access

  speed auto

!

!

!

interface vlan 1

  ip address  172.16.0.143  255.255.0.0

  no awcp

  no shutdown

!

!

!

!

!

no tftp server

no tftp server overwrite

http server

http secure-server

no snmp agent

no ip ftp server

no ip scp server

no ip sntp server

!

!

!

!

!

snmp-server engine local 8000029803000000000000

!

!

!

!

line con 0

  no login

!

line telnet 0 4

  login

  password XXXX

  no shutdown

line ssh 0 4

  login local-userlist

  no shutdown

!

!

!

!

end

Highlighted
Honored Contributor
Honored Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

Is this a flat /16 network? You don't have a default route in your latest configuration. Connected management host on same subnet? Connected management host getting the proper /16 mask from customer's DHCP server? Firewall or similar doing proxy-arp perhaps? When you try to ping the unresponsive Adtran device, does the correct MAC show up in the management host's ARP table?

Highlighted
New Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

Indeed, a flat /16 network.  Latest config was one of many, in hopes of finding a silver bullet - tried without default route, as goal is truly a layer two operation with management.  Host PC is on same subnet, and physically connected to the switch that we're doing most diagnostics on.  PC IP address is assigned via DHCP, and getting a /16 address in same subnet.  We had tried assigning static address to PC for testing, no difference.  When I try to ping unit, correct MAC shows in PC's arp table.

Highlighted
Honored Contributor
Honored Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

That's a real stumper. The only thing I can think of is that something on the network, perhaps a firewall, is doing proxy-arp, has a duplicate IP to the switch, or is tearing down TCP connections with a RST, but that wouldn't explain loss of ping.

If you can work in a maintenance window, you can isolate sections of the customer network until you find the culprit.

Highlighted
New Contributor

Re: Lose network access to admin when connected to customer network

Jump to solution

Just to close this out, the culprit was overwhelming multicast and broadcast traffic.  The good folks at support provided fabulous assistance in identifying the issues as well as some possible resolutions.  We still need to deal with the culprits, but at least we can now appropriately manage the switches as we work thru the process.