VyOS: MPLS over DMVPN Between VyOS and Cisco Part 2

How to: VyOS and Cisco

Monday April 14th 2025

This part is covering MPLS, VRF and advanced BGP configuration.

Recap of what was configured in part 1:

• GRE Multipoint tunnel interface with NHRP

• Securing the tunnel with IPSec and IKEv2, using shared secrets

• Loopback / Dummy interfaces that will be used for router ID

• iBGP peering between hub and spoke routers

Note: the 2nd spoke is just for show.

So basically the underlying infrastructure is operational. Now it’s time to transport different VRFs on top of the DMVPN tunnel.

Oh by the way; Leo told me that this part wasn’t possible:

This time it seems like I won’t get any help out of Leo.

Design

Overlay Design

The main benefit of implementing MPLSVPN over DMVPN is that it reduces the amount of tunnel interfaces needed. Without MPLS, you have to create one tunnel interface per VRF, which get old pretty fast. With MPLS enabled, you can transport multiple VRFs between sites using a single GRE tunnel interface.

The only drawback of tunneling MPLS over DMVPN is that it takes up 4 extra bytes of MTU.

Explanation:

• MPLSVPN is an overlay network and it does not matter what the underlying network is; if it’s ethernet, GRE or some third party solution like Zero Tier, the configuration is the same.

• Only prefixes for IPv6 will be included. For IPv4 prefixes you would use the VPNv4 Address-family.

To further explain the details behind how MPLSVPN works, examine this drawing:

Explanation:

• This is an example of two VRFs and how they separate their routing tables.

• The RD = Route Distinguisher is locally significant and does not have to match on all routers. It works like an ID number for a MPLSVPN routing table of a specific VRF.

  • Note: RD is only required when configuring MPLSVPN

• The routers export their own routes towards a RT = Route-Target, and import routes from others routers with the same RT.

  • Note: Import and Export RT values doesn’t have to be the same but it makes sense if you want all DMVPN to be able to speak directly to each other.

• Route targets can also be used to leak routes between VRFs. That is useful for configuring DIA = Direct Internet Access (not covered in this article).

But you probably know all and just want to know how it’s configured. This article is not about teaching all the concepts about MPLSVPN anyway.

Configuration

VRF Configuration

VRF Table

I’m going to configure the 3 VRFs shown in the overlay design:

VyOS Hub Router:

Cisco IOS-XE Spoke Router:

Note: The reason why the names and RDs are different between hub and spoke are of historical reasons. The important thing is that the route targets matches.

VyOS Hub Router

I’m only showing configuration for one VRF:

set vrf name NMS table '110'
set vrf name NMS protocols bgp parameters router-id '10.1.255.1'
set vrf name NMS protocols bgp system-as '65001'

set vrf name NMS protocols bgp address-family ipv6-unicast aggregate-address 2001:DB8:1234:A000::/60 summary-only
set vrf name NMS protocols bgp address-family ipv6-unicast export vpn
set vrf name NMS protocols bgp address-family ipv6-unicast import vpn
set vrf name NMS protocols bgp address-family ipv6-unicast label vpn export 'auto'
set vrf name NMS protocols bgp address-family ipv6-unicast rd vpn export '65001:1'
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute connected
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute ospfv3
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn export '65001:10003'
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn import '65001:10003'

VPNv6 Configuration:

set protocols bgp listen range 192.168.0.0/23 peer-group 'PG-MPLS'
set protocols bgp peer-group PG-MPLS remote-as '65001'
set protocols bgp peer-group PG-MPLS update-source 'dum0'
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-reflector-client

Explanation:

• On VyOS you have to define a specific VRF table ID. On Cisco I assume that the table ID is created automatically.

• On VyOS you can have different router ID’s per VRF. On Cisco it is one global ID for all VRFs.

• On VyOS you can also configure a different AS number per VRF. On Cisco it is one AS for all VRFs.

  • Note: You can set local-as per neighbor, but that is not the same thing.

Cisco IOS-XE Spoke Router

! VRF Configuration
!
vrf definition MGMT
 rd 65001:10
 !
 address-family ipv4
  route-target export 65001:10003
  route-target import 65001:10003
 exit-address-family
 !
 address-family ipv6
  route-target export 65001:10003
  route-target import 65001:10003
 exit-address-family
!
! BGP Configuration:
!
router bgp 65001
 neighbor 192.168.0.1 remote-as 65001
 neighbor 192.168.0.1 transport connection-mode active
 neighbor 192.168.0.1 update-source Loopback0
 !
  address-family vpnv6
   neighbor 192.168.0.1 activate
   neighbor 192.168.0.1 send-community both
  exit-address-family
 !
   address-family ipv6 vrf MGMT
    redistribute ospf 1
    aggregate-address 2001:DB8:1234:A000::/52
    aggregate-address 2001:DB8:1234:A010::/60 summary-only
   exit-address-family
 !
!

Explanation:

• Unlike VyOS, Cisco divides the VRF parameters and the BGP parameters into different sections.

MPLS Configuration

So far there is not much action yet. MPLS needs to be activated on the tunnel interface for routes inside the VRFs to be exchanged.

VyOS Hub Router

set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp interface tun3 mpls forwarding

set protocols mpls interface 'tun3'
set protocols mpls ldp interface 'tun3'
set protocols mpls ldp router-id '192.168.0.1'
set protocols mpls ldp discovery transport-ipv4-address '192.168.0.1'
set protocols mpls ldp targeted-neighbor ipv4 enable

Explanation:

• The configuration is for MPLS with LDPv4, because the underlay is IPv4 only.

• No matter how hard I try I can’t establish dynamic LDP neighbor relationship over the tunnel interface. Therefore I had to enable targeted LDP. Only one side needs to specify neighbor address though. Read the appendix for more information.

• I think this is the minimum required configuration. MPLS is so buggy that if I make any changes to it, VRF traffic stops and not even rollback works without requiring a reboot.

Source: https://docs.vyos.io/en/latest/configuration/protocols/mpls.html

Cisco IOS-XE Spoke Router

interface Tunnel3
 mpls ip
!
mpls ldp neighbor 192.168.0.1 targeted
mpls ldp router-id Loopback0

Verification and Troubleshooting

Let’s see if IPv6 traffic inside the VRF can get through the tunnel. I don’t have any pingable address on the VyOS hub router at the moment so I’m creating a temporary dummy interface for testing:

set interfaces dummy dum1 address '2001:DB8:1234:A000::1/64'
set interfaces dummy dum1 vrf 'NMS'

Note: ‘redistribute connected’ under the BGP process has already been added.

Here I’m pinging the dummy interface from the Cisco IOS-XE Spoke router:

SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Fail! What could it be? In the routing table, we can see there is a route to the prefix:

SAUNA-RO1#show ipv6 route vrf MGMT 2001:DB8:1234:A000::/64
Routing entry for 2001:DB8:1234:A000::/64
  Known via "bgp 65001", distance 200, metric 0, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::C0A8:1%default indirectly connected
      MPLS label: 80
      From C0A8:1::
      opaque_ptr 0x7F4C505408
      Last updated 00:02:54 ago

Note: I don’t know why the route is not summarized to 2001:DB8:1234:A000::/60 as i have configured ‘aggregate address’ but that is a minor detail right now.

There is something odd with that next hop… It points towards a link local address! That link local address is the one on LBS-RO1 dummy dum0, configured earlier:

vyos@LBS-RO1:~$ show interfaces dummy dum0 
dum0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether c2:13:bb:d2:22:a4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/32 scope global dum0
       valid_lft forever preferred_lft forever
    inet6 fe80::c0a8:1/64 scope link
       valid_lft forever preferred_lft forever
    Description: BGP-RouterID

On the VyOS side it looks like this:

vyos@LBS-RO1:~$ show ipv6 route vrf NMS 2001:DB8:1234:A010::/60
Routing entry for 2001:db8:1234:a010::/60
  Known via "bgp", distance 200, metric 2, vrf NMS, best
  Last update 02:31:01 ago
  r ::ffff:c0a8:3(vrf default) (recursive), label 43, weight 1
  r   192.168.2.3, via tun3(vrf default) onlink, label 43, weight 1

This problem reminds me about when I configured BGP between my Cisco router and pfSense firewall.

The solution

The solution is to manually change the advertised next-hop to the tunnel interface, but that’s an IPv4 address!

You may have noticed in the output above that it has a strange IPv6 address as next-hop (::ffff:c0a8:3). The prefix ::ffff:0:0/96 is used for IPv4-mapped IPv6 addresses. c0a8:3 actually represents 192.168.0.3. But the real next-hop should be 192.168.2.3, hence why the route is marked as recursive.

It’s interesting although BGP is standard based, you can spot different behaviors between vendors. Anyway, let’s configure a route-map that manually sets the next-hop to be the tunnel interface on the hub router:

VyOS Hub Router route-map configuration

First create the route-map:

set policy route-map RM_DMVPN_OUT description 'set next-hop tunnel3'
set policy route-map RM_DMVPN_OUT rule 10 action 'permit'
set policy route-map RM_DMVPN_OUT rule 10 set community add 'no-export'
set policy route-map RM_DMVPN_OUT rule 10 set ipv6-next-hop global '::ffff:c0a8:201'

Note: “community add no-export” makes sure that prefixes advertised between iBGP peers doesn’t get advertised outside the AS.

Then apply it to the VPNv6 neighbor:

set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-map export 'RM_DMVPN_OUT'

Now let’s verify what happens on the Cisco router:

SAUNA-RO1#show ipv6 route vrf MGMT 2001:DB8:1234:A000::/64
Routing entry for 2001:DB8:1234:A000::/64
  Known via "bgp 65001", distance 200, metric 0, type internal
  Route count is 1/1, share count 0
  Routing paths:
    192.168.2.1%default indirectly connected
      MPLS label: 80
      From C0A8:1::
      opaque_ptr 0x7F4C505408
      Last updated 00:01:38 ago

SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Still can’t ping though, so I probably need to set the next hop manually on the spoke router as well.

Cisco IOS-XE Route-map Configuration

Route-map configuration:

route-map RM_DMVPN_OUT permit 10
 set community no-export
 set ipv6 next-hop ::FFFF:192.168.2.3

Note: I actually typed “::ffff:c0a8:203” but the running configuration is displaying it as “192.168.2.3”.

Then apply it to the bgp vpnv6 neighbor:

 address-family vpnv6
  neighbor 192.168.0.1 route-map RM_DMVPN_OUT out

The moment of truth

Before pinging the remote address, let’s check how the route is looking at the hub side:

vyos@LBS-RO1:~$ show ipv6 route vrf NMS 2001:DB8:1234:A010::/60
Routing entry for 2001:DB8:1234:a010::/60
  Known via "bgp", distance 200, metric 2, vrf NMS, best
  Last update 00:04:23 ago
  * ::ffff:c0a8:203, via tun3(vrf default) onlink, label 43, weight 1

It’s not marked as recursive anymore. Let’s do the ping! drum roll please….

SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

vyos@LBS-RO1:~$ ping 2001:DB8:1234:A010::1 vrf NMS
PING 2001:DB8:1234:A010::1(2001:db8:1234:a010::1) 56 data bytes
64 bytes from 2001:db8:1234:a010::1: icmp_seq=1 ttl=62 time=2.33 ms
64 bytes from 2001:db8:1234:a010::1: icmp_seq=2 ttl=62 time=2.35 ms
64 bytes from 2001:db8:1234:a010::1: icmp_seq=3 ttl=62 time=2.40 ms
^C
--- 2001:DB8:1234:A010::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 2.330/2.361/2.401/0.029 ms

Success! You have no idea how many hours I have spent before this moment but wow that is satisfying!

Appendix

Special Mention

I want to thank Level Zero Networking for their blogs on a similar MPLSVPN setup. It was a good foundation to start with for making this lab.

• Part 1: https://lev-0.com/2024/01/08/dynamic-multipoint-vpn-with-zerotier-and-vyos/

• Part 2: https://lev-0.com/2024/01/09/dynamic-multipoint-vpn-with-zerotier-and-vyos-part-2-mpls/

Caveats discovered

Confusion about enabling MPLS LDP on Cisco IOS-XE

There is some confusion about how to enable MPLS on a tunnel interface. According to Cisco, you can use either ‘mpls bgp forwarding’ or the ‘mpls nhrp’ command, but the documentation is very unclear:

Source: https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/ip-multicast/b-ip-multicast/m-6vpe-6pe-mpls-over-dmvpn-phase2.html

• ‘mpls nhrp’ apparently enables tag switching, which I assume means TDP, which is Cisco proprietary.

• The documentation doesn’t explicitly say if ‘mpls bgp forwarding’ enables TDP or LDP. If they claim you can use either of them, I will assume it also enables TDP.

• ‘mpls ip’ apparently enables LDP, so that is the command I use.

Multicast issue on the VyOS tunnel interface

Whatever I do I can’t get MPLS LDP neighborship up without using targeted LDP. I discovered when troubleshooting through native FRR console that these log messages are appearing:

LBS-RO1# terminal monitor 
2025-03-17 19:00:51.868 [WARN] ldpd: [HD2R1-Z7JJT] send_packet: error sending packet to 224.0.0.2
2025-03-17 19:00:56.870 [WARN] ldpd: [HD2R1-Z7JJT] send_packet: error sending packet to 224.0.0.2

At the moment I can’t do anything about it so I just have to live with it.

Bug Report issued

I have created the following bug report: https://vyos.dev/T7266

Full Configuration

Down below is the full configuration, including part 1:

VyOS Hub Router Configuration

set interfaces dummy dum0 address '192.168.0.1/32'
set interfaces dummy dum0 address 'FE80::c0a8:1/64'
set interfaces dummy dum0 description 'BGP-RouterID'
set interfaces dummy dum0 ipv6 address no-default-link-local

set interfaces ethernet eth1 vif 3000 address '2001:DB8:1234::3/64'
set interfaces ethernet eth1 vif 3000 address 'fe80::3/64'
set interfaces ethernet eth1 vif 3000 address '10.0.1.3/26'
set interfaces ethernet eth1 vif 3000 description 'VLAN 3000 External'
set interfaces ethernet eth1 vif 3000 ipv6 address no-default-link-local

set interfaces tunnel tun3 address '192.168.2.1/32'
set interfaces tunnel tun3 enable-multicast
set interfaces tunnel tun3 encapsulation 'gre'
set interfaces tunnel tun3 ip adjust-mss '1360'
set interfaces tunnel tun3 ipv6 adjust-mss '1340'
set interfaces tunnel tun3 mtu '1400'
set interfaces tunnel tun3 parameters ip key '3'
set interfaces tunnel tun3 source-interface 'eth1.3000'

set protocols nhrp tunnel tun3 authentication 'secret'
set protocols nhrp tunnel tun3 holdtime '60'
set protocols nhrp tunnel tun3 multicast 'dynamic'
set protocols nhrp tunnel tun3 network-id '3'
set protocols nhrp tunnel tun3 redirect
set protocols nhrp tunnel tun3 registration-no-unique

set vpn ipsec profile DMVPN bind tunnel 'tun3'
set vpn ipsec esp-group DMVPN-ESP lifetime '3600'
set vpn ipsec esp-group DMVPN-ESP mode 'transport'
set vpn ipsec esp-group DMVPN-ESP pfs 'dh-group2'
set vpn ipsec esp-group DMVPN-ESP proposal 1 encryption 'aes256'
set vpn ipsec esp-group DMVPN-ESP proposal 1 hash 'sha1'
set vpn ipsec ike-group DMVPN-IKE key-exchange 'ikev2'
set vpn ipsec ike-group DMVPN-IKE lifetime '28800'
set vpn ipsec ike-group DMVPN-IKE proposal 1 dh-group '2'
set vpn ipsec ike-group DMVPN-IKE proposal 1 encryption 'aes256'
set vpn ipsec ike-group DMVPN-IKE proposal 1 hash 'sha1'
set vpn ipsec interface 'eth1.3000'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile DMVPN esp-group 'DMVPN-ESP'
set vpn ipsec profile DMVPN ike-group 'DMVPN-IKE'

set protocols bgp parameters log-neighbor-changes
set protocols bgp parameters router-id '192.168.0.1'
set protocols bgp listen range 192.168.0.0/23 peer-group 'PG-MPLS'
set protocols bgp listen range 192.168.2.0/23 peer-group 'PG-DMVPN'
set protocols bgp peer-group PG-DMVPN address-family ipv4-unicast route-reflector-client
set protocols bgp peer-group PG-DMVPN passive
set protocols bgp peer-group PG-DMVPN password 'secret'
set protocols bgp peer-group PG-DMVPN remote-as '65001'
set protocols bgp peer-group PG-MPLS address-family ipv4-vpn route-reflector-client
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-map export 'RM_DMVPN_OUT'
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-reflector-client
set protocols bgp peer-group PG-MPLS remote-as '65001'
set protocols bgp peer-group PG-MPLS update-source 'dum0'
set protocols bgp system-as '65001'
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp interface tun3 mpls forwarding

set protocols mpls interface 'tun3'
set protocols mpls ldp discovery transport-ipv4-address '192.168.0.1'
set protocols mpls ldp interface 'tun3'
set protocols mpls ldp router-id '192.168.0.1'
set protocols mpls ldp targeted-neighbor ipv4 enable

set vrf name NMS table '110'
set vrf name NMS protocols bgp parameters router-id '10.1.255.1'
set vrf name NMS protocols bgp system-as '65001'
set vrf name NMS protocols bgp address-family ipv6-unicast aggregate-address 2001:DB8:1234:A000::/60 summary-only
set vrf name NMS protocols bgp address-family ipv6-unicast export vpn
set vrf name NMS protocols bgp address-family ipv6-unicast import vpn
set vrf name NMS protocols bgp address-family ipv6-unicast label vpn export 'auto'
set vrf name NMS protocols bgp address-family ipv6-unicast rd vpn export '65001:1'
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute connected
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute ospfv3
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn export '65001:10003'
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn import '65001:10003'

set policy route-map RM_DMVPN_OUT description 'set next-hop tunnel3'
set policy route-map RM_DMVPN_OUT rule 10 action 'permit'
set policy route-map RM_DMVPN_OUT rule 10 set ipv6-next-hop global '::ffff:c0a8:201'

Notes:

• VyOS software version used: VyOS 1.5-rolling-202502110508

• Interface “dummy dum1” used for testing not included.

Cisco IOS-XE Spoke Router Configuration

interface Loopback0
 description BGP_RouterID
 ip address 192.168.0.3 255.255.255.255
 ipv6 address FE80::C0A8:3 link-local
!
interface GigabitEthernet0/0/1.3000
 description FW-EXTERNAL
 encapsulation dot1Q 3000
 vrf forwarding EXTERNAL
 ip address 10.0.1.2 255.255.255.192
 ipv6 address FE80::2 link-local
 ipv6 address 2001:DB8:1234::2/64
 ipv6 nd ra suppress all
!
interface Tunnel3
 ip address 192.168.2.3 255.255.254.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication secret
 ip nhrp network-id 3
 ip nhrp holdtime 60
 ip nhrp nhs 192.168.2.1 nbma 10.0.1.3
 ip nhrp redirect
 ip tcp adjust-mss 1360
 ipv6 mtu 1400
 ipv6 tcp adjust-mss 1340
 mpls ip
 tunnel source GigabitEthernet0/0/1.3000
 tunnel mode gre multipoint
 tunnel key 3
 tunnel path-mtu-discovery
 tunnel vrf EXTERNAL
 tunnel protection ipsec profile DMVPN
!
!
crypto ikev2 proposal IKEV2-PROPOSAL 
 encryption aes-cbc-256
 integrity sha1
 group 2
!
crypto ikev2 policy IKEV2-POLICY 
 match fvrf any
 proposal IKEV2-PROPOSAL
!
crypto ikev2 keyring DMVPN-KEYRING
 peer DMVPN-PEER
  address 0.0.0.0 0.0.0.0
  pre-shared-key secret
!
crypto ikev2 profile DMVPN-IKE
 match fvrf any
 match identity remote address 0.0.0.0 
 authentication remote pre-share
 authentication local pre-share
 keyring local DMVPN-KEYRING
 dpd 30 5 on-demand
!
crypto ipsec transform-set DMVPN-ESP esp-aes 256 esp-sha-hmac 
 mode transport
!
crypto ipsec profile DMVPN
 set transform-set DMVPN-ESP 
 set ikev2-profile DMVPN-IKE
!
!
vrf definition MGMT
 rd 65001:10
 !
 address-family ipv4
  route-target export 65001:10003
  route-target import 65001:10003
 exit-address-family
 !
 address-family ipv6
  route-target export 65001:10003
  route-target import 65001:10003
 exit-address-family
!
router bgp 65001
 bgp router-id 192.168.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.0.1 remote-as 65001
 neighbor 192.168.0.1 transport connection-mode active
 neighbor 192.168.0.1 update-source Loopback0
 neighbor 192.168.2.1 remote-as 65001
 neighbor 192.168.2.1 transport connection-mode active
 neighbor 192.168.2.1 password secret
 !
 address-family ipv4
  network 192.168.0.3 mask 255.255.255.255
  neighbor 192.168.2.1 activate
 exit-address-family
 !
 address-family vpnv6
  neighbor 192.168.0.1 activate
  neighbor 192.168.0.1 send-community both
  neighbor 192.168.0.1 route-map RM_DMVPN_OUT out
 exit-address-family
 !
 address-family ipv6 vrf MGMT
  redistribute ospf 1
  aggregate-address 2001:DB8:1234:A010::/60 summary-only
  aggregate-address 2001:DB8:1234:A000::/52
 !
!
route-map RM_DMVPN_OUT permit 10 
 set ipv6 next-hop ::FFFF:192.168.2.3

Note: Cisco IOS-XE software version used: 17.03.04a

Possible Improvements

DIA = Direct Internet Access

As I mentioned in part 1, DIA can be enabled on the spoke router. But to do that securely it requires ZBF = Zone-Based Firewall configuration. That is a future article in itself.

Stronger security with TLS certificates

Once I have a functioning PKI infrastructure, I will make an update using SSL certificates instead of pre-shared keys.

Dual hub and multiple spokes

There are still some unanswered questions about if spoke-to-spoke traffic will work since I don’t have a second spoke to test with.

Also, dual hub is nice for testing redundancy and load balancing.