Netvanta 7100- Multi-ISP setup will work for weeks or days then locks up secondary ISP connection

[bradh]

I have a Gen1 Netvanta 7100.
A5.03.00.E Firmware

I have had a multi-isp setup working for months.  We upgraded the DSL to a higher tier, which required a change in the gateway for "WanInt" ISP2.  Ever since making these changes, the setup will work for weeks, or days, then WanInt completely stops working.  Reboots of modem and router do not fix the issue. reloading a backup of the config made after changing the gateway has gotten it working again twice, but it has failed within days or hours afterword. PPP2 shows up with the correct IP address and Gateway, but cannot ping it or access the internet from Vlan 1.  Any insight or assistance would be appreciated.  Configs, and some debug and IP policy information included below:  Internet connections Voice- Eth 0/1 Internet Data Eth 0/22

vlan 1

  name "Default"

!

vlan 20

  name "VoIP20"

!

vlan 50

  name "VoiceInt"

!

vlan 100

  name "WanInt"

!

interface eth 0/1

  description WAN

  spanning-tree edgeport

  no shutdown

  switchport access vlan 50

!

!

interface eth 0/2

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1-49,51-4094

!

!

interface eth 0/3

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1-49,51-4094

......

!

!

interface eth 0/21

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1-49,51-4094

!

!

interface eth 0/22

  spanning-tree edgeport

  no shutdown

  switchport access vlan 100

!

!

interface eth 0/23

  description Engenius10.20.0.14

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1-49,51-4094

!

!

interface eth 0/24

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1-49,51-4094

!

interface vlan 1

  ip address  10.20.0.1  255.255.255.0

  ip policy route-map WanInt

  access-policy Private

  no shutdown

!

interface vlan 20

  description VoIP20

  ip address  10.20.20.1  255.255.255.0

  access-policy Private

  media-gateway ip primary

  no shutdown

!

interface vlan 50

  description WanInt

  no ip address

  no shutdown

!

interface vlan 100

  description WanInt

  no ip address

  no awcp

  no shutdown

!

interface ppp 1

  ip address negotiated   (This is static address 173.187.aaa.bbb- addresses are reserved)

  access-policy Public

  crypto map VPN

  media-gateway ip primary

  no fair-queue

  ppp pap sent-username xxxxx password xxxxxx

  no shutdown

  cross-connect 1 vlan 50 ppp 1

!

interface ppp 2

  description WanInt

  ip address negotiated no-default    (This is static address 216.97.jjj.kkk- addresses are reserved)

  access-policy WanInt

  no fair-queue

  ppp pap sent-username xxxxx password xxxxx

  no shutdown

  cross-connect 2 vlan 100 ppp 2

!

!

router rip

  version 2

!

!

!

route-map WanInt permit 10

  match ip address WanInt

  set ip next-hop 75.91.xxx.yyy   (this is the Static gateway negotiated through PPP1 and PPP2- addresses are reserved)

!

!

!

!

ip access-list standard wizard-ics

  remark NAT list wizard-ics

  permit any log

!

!

ip access-list extended alarmline

  permit tcp any  host 173.187.aaa.bbb eq 7700   log

  permit udp any  host 173.187.aaa.bbb eq 7700    log

!

ip access-list extended Internet

  permit ip 0.0.0.0 255.255.255.0  any   

!

ip access-list extended Remote

  remark Remote Access WanInt

  permit tcp any  host 216.97.jjj.kkk eq www   log

  permit tcp any  host 216.97.jjj.kkk eq smtp   log

  permit tcp any  host 216.97.jjj.kkk eq domain   log

  permit tcp any  host 216.97.jjj.kkk eq https   log

  permit tcp any  host 216.97.jjj.kkk eq 987   log

  permit tcp any  host 216.97.jjj.kkk eq 1723   log

  permit udp any  host 216.97.jjj.kkk eq domain    log

  permit udp any  host 216.97.jjj.kkk eq 987    log

  permit udp any  host 216.97.jjj.kkk eq 1723    log

!

ip access-list extended self

  remark Traffic to NetVanta

  permit ip any  any     log

!

ip access-list extended vpn-10-vpn-selectors1

  permit ip 10.20.0.0 0.0.255.255  10.20.0.0 0.0.255.255   

!

ip access-list extended WanInt

  deny   ip 10.20.0.0 0.0.0.255  10.20.0.0 0.0.255.255     log

  permit ip 10.20.0.0 0.0.0.255  any     log

!

ip access-list extended web-acl-10

  remark Admin Access

  permit tcp any  any eq ssh   log

!

ip access-list extended web-acl-11

  remark Internal Allow 1

  permit ip 10.20.0.0 0.0.255.255  10.20.0.0 0.0.255.255   

!

ip access-list extended web-acl-12

  remark Admin Access

  permit tcp any  any eq https   log

  permit tcp any  any eq ssh   log

  permit tcp any  host 173.187.aaa.bbb eq telnet   log

!

!

ip access-list extended web-acl-5

  remark SIP Trunk

  permit udp host 64.94.mmm.nnn  any eq 5060  

!

ip access-list extended web-acl-7

  remark Internet NAT

  permit ip 10.20.0.0 0.0.0.255  any     log

!

ip access-list extended web-acl-9

  remark Remote Access

  permit tcp any  host 173.187.aaa.bbb eq www   log

  permit tcp any  host 173.187.aaa.bbb eq smtp   log

  permit tcp any  host 173.187.aaa.bbb eq domain   log

  permit tcp any  host 173.187.aaa.bbb eq https   log

  permit tcp any  host 173.187.aaa.bbb eq 987   log

  permit tcp any  host 173.187.aaa.bbb eq 1723   log

  permit udp any  host 173.187.aaa.bbb eq domain    log

  permit udp any  host 173.187.aaa.bbb eq 987    log

  permit udp any  host 173.187.aaa.bbb eq 1723    log

!

!

ip policy-class Private

  allow list self self

  allow list web-acl-11

  nat source list web-acl-7 interface ppp 2 overload

  nat source list wizard-ics interface ppp 1 overload

  allow list vpn-10-vpn-selectors1

!

ip policy-class Public

  allow list web-acl-5

  allow list web-acl-12 self

  allow list vpn-10-vpn-selectors1 stateless

  nat destination list web-acl-9 address 10.20.0.254

  nat destination list alarmline address 10.20.0.190

!

ip policy-class Publicc

  ! Implicit discard

!

no ip policy-class WanInt rpf-check

ip policy-class WanInt

  allow list web-acl-10 self

  nat destination list Remote address 10.20.0.254

!

Debug- appears that traffic from Vlan1 is not being matched or otherwise is still trying to flow out ppp1:

2014.02.04 12:21:26 FIREWALL   nat source -> 216.97.jjj.kkk, flags = 0x00000002, 0x00000000, timeout = 60

2014.02.04 12:21:26 FIREWALL   Selector1: Dir=Private, int=vlan 1, Protocol=17  cookie-> ppp 1

2014.02.04 12:21:26 FIREWALL     SrcIp: 10.20.0.254, DstIp: 68.12.16.25

2014.02.04 12:21:26 FIREWALL     SrcPort: 49434, DstPort: 53

2014.02.04 12:21:26 FIREWALL   Selector2: Dir=Public, int=ppp 1, Protocol=17 

2014.02.04 12:21:26 FIREWALL     SrcIp: 68.12.16.25, DstIp: 216.97.jjj.kkk

2014.02.04 12:21:26 FIREWALL     SrcPort: 53, DstPort: 1072

2014.02.04 12:21:26 FIREWALL Adding new associations to DB

2014.02.04 12:21:26 FIREWALL   Assoc Index = 15652, Count (total, policy-class) = 82, 70

2014.02.04 12:21:26 FIREWALL   nat source -> 216.97.jjj.kkk, flags = 0x00000002, 0x00000000, timeout = 60

2014.02.04 12:21:26 FIREWALL   Selector1: Dir=Private, int=vlan 1, Protocol=17  cookie-> ppp 1

2014.02.04 12:21:26 FIREWALL     SrcIp: 10.20.0.254, DstIp: 166.102.165.13

2014.02.04 12:21:26 FIREWALL     SrcPort: 50466, DstPort: 53

2014.02.04 12:21:26 FIREWALL   Selector2: Dir=Public, int=ppp 1, Protocol=17 

2014.02.04 12:21:26 FIREWALL     SrcIp: 166.102.165.13, DstIp: 216.97.jjj.kkk

2014.02.04 12:21:26 FIREWALL     SrcPort: 53, DstPort: 1073

2014.02.04 12:21:26 FIREWALL Adding new associations to DB

2014.02.04 12:21:26 FIREWALL   Assoc Index = 15653, Count (total, policy-class) = 83, 9

2014.02.04 12:21:26 FIREWALL   allow, flags = 0x00000000, 0x00000000, timeout = 20

From the Private policy sessions it looks lik they are trying to go out correctly:

Private Policy-class sessions

UDP(17) 10.20.0.254 / 49851 68.12.16.30 / 53 216.97.165.25 / 15625
UDP(17) 10.20.0.254 / 49856 68.12.16.30 / 53 216.97.165.25 / 15561
UDP(17) 10.20.0.254 / 49904 68.12.16.30 / 53 216.97.165.25 / 15631
UDP(17) 10.20.0.254 / 49951 68.12.16.30 / 53 216.97.165.25 / 15646
UDP(17) 10.20.0.254 / 49976 68.12.16.30 / 53 216.97.165.25 / 15599
UDP(17) 10.20.0.254 / 50071 68.12.16.30 / 53 216.97.165.25 / 15569
UDP(17) 10.20.0.254 / 50200 68.12.16.30 / 53 216.97.165.25 / 15585
UDP(17) 10.20.0.254 / 50366 68.12.16.30 / 53 216.97.165.25 / 15555
UDP(17) 10.20.0.254 / 50406 68.12.16.30 / 53 216.97.165.25 / 15562
UDP(17) 10.20.0.254 / 50493 68.12.16.30 / 53 216.97.165.25 / 15595

Please let me know if additional information is needed, and thank you for any assistance.

BradH

[bradh]

Update:

So after further troubleshooting, it appears that the Carrier gateway being identical on both DsL PPP interfaces is the issue.  I am able to successfully use one or the other by adjusting the configuration, but I can't use pbr with both links up, successfully, for long periods of time.  It seems to work briefly, depending on which interface comes up first, but once one of the dsl interfaces resets itself for any reason, it breaks.   I assume this is because the route-map is unable to determine the correct ppp interface to route through, because the gateway is the same for both, and is defaulting to ppp1.  I am working with the carrier to have my connection moved to a different ip scheme and gateway, though they are not terribly optimistic, as this is a rural location.

[Anonymous · Accepted solution]

:

Thank you for asking this question in the support community.  First, let me say that if the PPP gateway is the same for both interfaces, then this application most likely will not work.  However, I do have a few suggestions for you with this application. 

Even though I will provide you with some recommendations for the policy-based routing (PBR) portion of the configuration, please understand that PBR is not supported on the NetVanta 7100 as outlined in ADTRAN's Feature Matrix.

• In the route-map, I recommend you change the set ip next hop <address> command to set interface ppp <number>, this will allow the physical address to change, without having to be manually changed in the configuration. 
• I recommend you disable LLDP on both of the ISP interfaces with the command no lldp send-and-receive.
• I think you should add the destination policy-class to the end of the Private source NAT statements.  Here is an example configuration:

ip policy-class Private

  allow list self self

  allow list web-acl-11

  nat source list web-acl-7 interface ppp 2 overload policy WanInt

  nat source list wizard-ics interface ppp 1 overload policy Public

With that said, this application will more efficiently for you if the ISP is able to separate your PPP interfaces into two different subnets.  I hope that makes sense, but please do not hesitate to reply to this post with any additional questions or information.  I will be happy to help in any way I can.

Levi