VyOS: MPLS over DMVPN Between VyOS and Cisco Part 1

How-To: VyOS and Cisco

Monday 7th April 2025

I would argue that if you can set up an MPLS over DMVPN design from scratch, you are pretty much an expert in routing and switching. The concept involves following topics:

• MPLS = Multi Protocol Label Switching

• VRF = Virtual Routing and Forwarding

• GRE tunneling

• NHRP = Next-Hop Router Protocol

• Complex BGP configuration

• May include IGPs like OSPF and EIGRP

• ZBF = Zone-Based Firewall

• IPSec

For a complete design it can also include:

• IPv6 support

• TLS/SSL certificates

• IPSec NAT traversal

This concept used to be a somewhat Cisco Proprietary, but it builds on open standards so it should not be impossible to implement on other vendors. In fact, I have just proven that it can be performed in a mixed vendor environment.

Therefore I’m going to demonstrate how to configure a VyOS hub and a Cisco IOS-XE spoke to perform MPLSoDMVPN.

This will be covered in two parts:

• Part 1: Setting up the DMVPN tunnel interface and configure dynamic routing inside the tunnel.

• Part 2: Implement MPLS, VRFs and BGP VPNv6 address family. I’m going IPv6 only (at least IPv6 mostly).

Why you would still use MPLSoDMVPN today

Since this is such a complex setup, why bother when there are SDWAN solutions? SDWANs still uses a lot of the same underlying concepts, but automate the configuration and hides the complexity away from the operator. SDWANs could potentially minimize the complexity of day-to-day operations, but comes with a cost: They are usually subscription based and vendor proprietary. Going for open source solutions could potentially reduce the TCO = Total Cost of Ownership.

Note: Some might argue that SDWAN reduces the TCO because you don’t need to hire experts. In reality however, it’s debatable if the cost just moves from the pockets of your employees, to the pocket of the vendor.

One alternative is to run Zero Tier. It replaces the need for a GRE tunnel with a cloud interface that interconnects your routers, with a very simple management GUI. Many open source platform supports installation of the Zero Tier client, but Cisco does not. I have no problem going all open source, but I still have a few Cisco routers that are useful, as long as I can make them interoperable with other vendors.

Note: This blog is in fact inspired by Level Zero Networking, who setup a MPLSVPN network over Zero Tier:

• Part 1: https://lev-0.com/2024/01/08/dynamic-multipoint-vpn-with-zerotier-and-vyos/

• Part 2: https://lev-0.com/2024/01/09/dynamic-multipoint-vpn-with-zerotier-and-vyos-part-2-mpls/

The benefit of MPLS over DMVPN setup is that it doesn’t require any third party. Therefore, because of high TCO and vendor lock-in with SDWAN, and lack of support between Cisco and Zero Tier, the good old-fashioned MPLSoDMVPN design is still valid.

Design

Before you begin configuring it, it’s mandatory to have drawn it first.

DMVPN Underlay Design

Explanation:

• The GRE tunnel is running IPv4 only. The reason for that:

  • Cisco only supports LDPv4, while VyOS supports LDPv6. In addition;

  • VyOS only supports NHRPv4, while Cisco supports NHRPv6.

• This is only an underlay network so as a workaround you can tunnel IPv6 VRF traffic on top of it (therefore the design becomes IPv6 mostly).

• iBGP will be used to advertise the loopback / dummy interfaces. I tried first with OSPF using point-to-multipoint network type, but I had issues with flapping neighbors. BGP scales better anyway and works best when multiple vendors are involved.

  • On Cisco they are called Loopback interfaces

  • On VyOS they are called dummy interfaces

• The 2nd spoke router is just for show. I might include it in the future.

Internet Edge

With Internet Edge, I refer to the outside interfaces of the hub and spoke routers. In my lab, I have a very simple topology for internet edge; they are actually directly connected on the same subnet:

Explanation:

• Both routers have direct connection to each other. Normally, you would have to implement a static default route if this was over the internet. In this case, there is no need for routing configuration, on the internet edge at least.

• Cisco routers supports something called FVRF = Front-Door VRF. This is used for an easy way to perform DIA = Direct Internet Access. It makes traffic not destined to any internal networks take the local default route, instead of using the one learnt via the tunnel interface.

• The VyOS router does not seem to have support for FVRF configuration and therefore uses the global routing table.

• Hub routers are usually behind a firewall. A spoke router however, might be directly exposed to the Internet. In that case, ZBF = Zone-Based Firewall is recommended. However, that configuration is so complex that it deserves it’s own post.

• IPSec with IKEv2 encryption is necessary for FVRF support.

That’s enough drawings to get started. More design details will be covered in part 2.

DMVPN Configuration

Internet Edge

Let’s start with the “Internet Facing” interface configuration

VyOS Hub Router

set interfaces ethernet eth1 vif 3000 address '2001:DB8:1234::3/64'
set interfaces ethernet eth1 vif 3000 address 'fe80::3/64'
set interfaces ethernet eth1 vif 3000 address '10.0.1.3/26'
set interfaces ethernet eth1 vif 3000 description 'VLAN 3000 External'
set interfaces ethernet eth1 vif 3000 ipv6 address no-default-link-local

That’s it. Since the spoke device in this case is directly connected, no static route is needed in this lab setup.

Cisco IOS-XE Spoke Router

! FVRF configuration:
!
vrf definition EXTERNAL
 rd 65001:1
 !
 address-family ipv4
  route-target export 65001:1
 exit-address-family
 !
 address-family ipv6
  route-target export 65001:1
 exit-address-family
!
! Interface configuration:
!
interface GigabitEthernet0/0/1.3000
 description FW-EXTERNAL
 encapsulation dot1Q 3000
 vrf forwarding EXTERNAL
 ip address 10.0.1.2 255.255.255.192
 ipv6 address FE80::2 link-local
 ipv6 address 2001:DB8:1234::2/64
!

There’s a little more to it on the spoke router because of the Front-Door VRF and ZBF.

GRE tunnel configuration

VyOS Hub Router

set interfaces tunnel tun3 address '192.168.2.1/32'
set interfaces tunnel tun3 enable-multicast
set interfaces tunnel tun3 encapsulation 'gre'
set interfaces tunnel tun3 ip adjust-mss '1360'
set interfaces tunnel tun3 ipv6 adjust-mss '1320'
set interfaces tunnel tun3 mtu '1400'
set interfaces tunnel tun3 parameters ip key '3'
set interfaces tunnel tun3 source-interface 'eth1.3000'

Explanation:

• When enabling NHRP on the tunnel interface, the IP address has to be a /32 prefix. The same limitation is not on Cisco routers.

• the parameters ip key'3' is the tunnel identifier and has to match on the remote end. I just chose 3 at random.

• tunnel PMTUD is turned on by default on VyOS. You only have an option to turn it off:

vyos@LBS-RO1# set interfaces tunnel tun3 parameters ip 
Possible completions:
...
   no-pmtu-discovery    Disable path MTU discovery

Cisco IOS-XE Spoke Router

interface Tunnel3
 ip address 192.168.2.3 255.255.254.0
 ip mtu 1400
 ip tcp adjust-mss 1360
 ipv6 mtu 1400
 ipv6 tcp adjust-mss 1340
 tunnel source GigabitEthernet0/0/1.3000
 tunnel mode gre multipoint
 tunnel path-mtu-discovery
 tunnel key 3
 tunnel vrf EXTERNAL

Explanation:

• tunnel vrf EXTERNAL tells the router to tunnel traffic from the global routing table over VRF EXTERNAL. I have not found a VyOS equivalent for this command.

• On Cisco you have to manually enable tunnel PMTUD.

Choice of MTU

Using Ciscos MTU calculator (unfortunately decommissioned), this is how I came up with the values for the MTU and TCP MSS:

Firewalls and Path MTU Discovery

Most applications today supports Path MTU Discovery, and it can also be configured on the tunnel interfaces. However the protocol relies on ICMP to figure out what the maximum MTU is. If a firewall along the path to the destination is not configured to permit ICMP code 3 “packet to big” and “destination unreachable”, it will fail and cause a black hole. And it’s a pain trying to contact a technician in a different organization to fix their configuration.

Sources:

• Wikipedia on PMTUD

• VyOS interface configuration guide

As Internet wide PMTU discovery rarely works, we sometimes need to clamp our TCP MSS value to a specific value. - VyOS Documentation

Therefore I think that manually setting MTU and MSS is a good practice.

To turn on PMTUD globally:

Cisco IOS-XE:

ip tcp path-mtu-discovery

In VyOS I haven’t found an equivalent setting. I’m not sure if PMTUD for internal router traffic is on by default or not. But it’s on by default for tunnel interfaces at least.

NHRP = Next-Hop Router Protocol

GRE Multipoint encapsulation + NHRP protocol = DMVPN. It is configured under the tunnel interface.

VyOS Hub Router

set protocols nhrp tunnel tun3 authentication 'secret'
set protocols nhrp tunnel tun3 holdtime '60'
set protocols nhrp tunnel tun3 multicast 'dynamic'
set protocols nhrp tunnel tun3 network-id '3'
set protocols nhrp tunnel tun3 redirect
set protocols nhrp tunnel tun3 registration-no-unique

Explanation:

• multicast dynamic indicates that this is a hub router

• The redirect command enables direct spoke-to-spoke communication. The nhrp shortcut command is also required on the spoke.

• registration-no-unique allows NAT Traversal

Cisco IOS-XE Spoke Router

interface Tunnel3 
 ip nhrp authentication secret
 ip nhrp network-id 3
 ip nhrp holdtime 60
 ip nhrp nhs 192.168.2.1 nbma 10.0.1.3
 ip nhrp redirect
 ip nhrp shortcut
 ip nhrp registration no-unique

Explanation:

• The Next-Hop Server (nhs) is set to the IPv4 nbma of the outside interface of the hub. This is supposed to work with the IPv6 address as well at least between two Cisco Routers, but the tunnel connection won’t establish. That means the NAT Traversal is imminent.

• ip nhrp shortcut and ip nrhp registration no-unique is turned on by default on newer IOS-XE versions and might not be listed under running configuration, unless you type “show running-config all”.

• Cisco recommends having ip nhrp redirect on all DMVPN routers.

When using this feature [nhrp shortcut], we recommend configuring the ip nhrp redirect command on all the DMVPN nodes. This configuration would be useful in the event the data traffic takes a spoke-to-spoke-hub-spoke path.

- Cisco Documentation

IPSec

This was the hardest part. IPSec configuration in general is a pain. Having to translate IPSec syntax from VyOS to Cisco in addition, is even more complicated. This was a good time to ask Braves AI “Leo” for help.

After a little back and forth (for instance I had to tell Leo that the configuration he gave me was for IKEv1, not IKEv2 as I requested) we came close to something that would work. After a little manual adjustments on the Cisco side, I came up with this configuration…

VyOS Hub Router

IPSec Profiles:

set vpn ipsec esp-group DMVPN-ESP lifetime '3600'
set vpn ipsec esp-group DMVPN-ESP mode 'transport'
set vpn ipsec esp-group DMVPN-ESP pfs 'dh-group2'
set vpn ipsec esp-group DMVPN-ESP proposal 1 encryption 'aes256'
set vpn ipsec esp-group DMVPN-ESP proposal 1 hash 'sha1'
set vpn ipsec ike-group DMVPN-IKE key-exchange 'ikev2'
set vpn ipsec ike-group DMVPN-IKE lifetime '28800'
set vpn ipsec ike-group DMVPN-IKE proposal 1 dh-group '2'
set vpn ipsec ike-group DMVPN-IKE proposal 1 encryption 'aes256'
set vpn ipsec ike-group DMVPN-IKE proposal 1 hash 'sha1'
set vpn ipsec interface 'eth1.3000'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile DMVPN esp-group 'DMVPN-ESP'
set vpn ipsec profile DMVPN ike-group 'DMVPN-IKE'

Apply on the Tunnel Interface:

set vpn ipsec profile DMVPN bind tunnel 'tun3'

Explanation:

• I’m using pre-shared keys for now. I’m planning to implement TLS authentication in the future.

Cisco IOS-XE Spoke Router

! IKEv2 Configuration:
!
crypto ikev2 proposal IKEV2-PROPOSAL 
 encryption aes-cbc-256
 integrity sha1
 group 2
crypto ikev2 policy IKEV2-POLICY 
 match fvrf any
 proposal IKEV2-PROPOSAL
crypto ikev2 keyring DMVPN-KEYRING
 peer DMVPN-PEER
  address 0.0.0.0 0.0.0.0
  pre-shared-key secret
 !
!
crypto ikev2 profile DMVPN-IKE
 match fvrf any
 match identity remote address 0.0.0.0 
 authentication remote pre-share
 authentication local pre-share
 keyring local DMVPN-KEYRING
 dpd 30 5 on-demand
 set ikev2-profile DMVPN-IKE
!
! IPSec Configuration:
!
crypto ipsec transform-set DMVPN-ESP esp-aes 256 esp-sha-hmac 
 mode transport
!
crypto ipsec profile DMVPN
 set transform-set DMVPN-ESP 
 set ikev2-profile DMVPN-IKE
!
! Apply IPSec Profile on the tunnel interface:
!
interface Tunnel3
 tunnel protection ipsec profile DMVPN

Explanation:

• match fvrf any could also have been match fvrf EXTERNAL

DMVPN Verification

After that the tunnel interface should come up and you are able to ping the tunnel interfaces:

SAUNA-RO1#ping 192.168.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

On the Cisco router you can perform this command to verify DMVPN operation:

SAUNA-RO1#show dmvpn 
...
Interface: Tunnel3, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.0.1.3            192.168.2.1    UP 00:00:09     S

On VyOS:

vyos@LBS-RO1:~$ show ip nhrp cache
Iface    Type     Protocol     NBMA      Claimed NBMA  Flags  Identity
tun3     local    192.168.2.1  10.0.1.3  10.0.1.3             -
tun3     dynamic  192.168.2.3  10.0.1.2  10.0.1.2      T      10.0.1.2

Configure BGP for DMVPN peering.

The last thing before wrapping up part 1 is to configure BGP neighbor relationship between the tunnel IP addresses, as shown in the drawing in the beginning.

VyOS Hub Router

Configure dummy interface to be used by MPLSVPN later:

set interfaces dummy dum0 address '192.168.0.1/32'
set interfaces dummy dum0 address 'FE80::c0a8:1/64'
set interfaces dummy dum0 description 'BGP-RouterID'
set interfaces dummy dum0 ipv6 address no-default-link-local

BGP Configuration:

set protocols bgp system-as '65001'
set protocols bgp parameters router-id '192.168.0.1'
set protocols bgp parameters log-neighbor-changes
set protocols bgp address-family ipv4-unicast network 192.168.0.1/32
set protocols bgp listen range 192.168.2.0/23 peer-group 'PG-DMVPN'
set protocols bgp peer-group PG-DMVPN address-family ipv4-unicast route-reflector-client
set protocols bgp peer-group PG-DMVPN passive
set protocols bgp peer-group PG-DMVPN remote-as '65001'
set protocols bgp peer-group PG-DMVPN password 'secret'

Explanation:

• The Router ID does not have to match the dummy interface, but it’s recommended to use as Router ID since it’s always up.

• The DMVPN hub is configured to be a route reflector and does not initiate TCP sessions to other spokes.

• Optional: Since BGP neighbors are established dynamically, a password to authenticate the neighbor establishment is recommended. Although, a hacker have to first crack the IPSec encryption, so it may be overkill.

Cisco IOS-XE Spoke Router

Loopback Configuration:

interface Loopback0
 description BGP-RouterID
 ip address 192.168.0.3 255.255.255.255
 ipv6 address FE80::C0A8:3 link-local
!

Note: The link-local addresses will make sense in part 2.

BGP Configuration:

router bgp 65001
 bgp router-id 192.168.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.2.1 remote-as 65001
 neighbor 192.168.2.1 transport connection-mode active
 neighbor 192.168.2.1 password secret
 !
 address-family ipv4
  network 192.168.0.3 mask 255.255.255.255
  neighbor 192.168.2.1 activate
 exit-address-family

Explanation:

• The spoke router is the one initiating the TCP session with the Hub

• no bgp default ipv4-unicast prevents new neighbors from automatically become enabled in the ipv4-unicast address family.

BGP Verification

Now it’s time to verify that the loopback/dummy interfaces are reachable through the tunnel interface

The commands and the outputs are almost identical for both vendors. Therefore, I only include outputs from the VyOS hub router:

vyos@LBS-RO1:~$ show ip route 192.168.2.3
Routing entry for 192.168.2.3/32
  Known via "nhrp", distance 10, metric 0, best
  Last update 02:06:14 ago
  * directly connected, tun3, weight 1

vyos@LBS-RO1:~$ show bgp ipv4 192.168.0.3
BGP routing table entry for 192.168.0.3/32, version 73
Paths: (1 available, best #1, table default)
  Advertised to non peer-group peers:
  192.168.2.3
  Local, (Received from a RR-client)
    192.168.2.3 from 192.168.2.3 (192.168.0.3)
      Origin IGP, metric 0, localpref 100, valid, internal, best (First path received)
      Last update: Thu Mar  6 20:00:03 2025

vyos@LBS-RO1:~$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=255 time=0.664 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=255 time=0.783 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=255 time=0.952 ms
^C
--- 192.168.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2041ms
rtt min/avg/max/mdev = 0.664/0.799/0.952/0.118 ms

Now the underlay is ready. In Part 2 I will be covering:

• MPLS configuraiton

• VRF configuration

• BGP VPNv6 configuration

Special Mentions

I want to thank Leo (Brave AI) for helping me out with the IPSec configuration. AI tools can be very useful when used properly, but they are not at the point where they replace me yet.