Monday April 14th 2025
This part is covering MPLS, VRF and advanced BGP configuration.
Recap of what was configured in part 1:
GRE Multipoint tunnel interface with NHRP
Securing the tunnel with IPSec and IKEv2, using shared secrets
Loopback / Dummy interfaces that will be used for router ID
iBGP peering between hub and spoke routers
Note: the 2nd spoke is just for show.
So basically the underlying infrastructure is operational. Now it’s time to transport different VRFs on top of the DMVPN tunnel.
Oh by the way; Leo told me that this part wasn’t possible:
Design
Overlay Design
The main benefit of implementing MPLSVPN over DMVPN is that it reduces the amount of tunnel interfaces needed. Without MPLS, you have to create one tunnel interface per VRF, which get old pretty fast. With MPLS enabled, you can transport multiple VRFs between sites using a single GRE tunnel interface.
The only drawback of tunneling MPLS over DMVPN is that it takes up 4 extra bytes of MTU.
Explanation:
MPLSVPN is an overlay network and it does not matter what the underlying network is; if it’s ethernet, GRE or some third party solution like Zero Tier, the configuration is the same.
Only prefixes for IPv6 will be included. For IPv4 prefixes you would use the VPNv4 Address-family.
To further explain the details behind how MPLSVPN works, examine this drawing:
Explanation:
This is an example of two VRFs and how they separate their routing tables.
The RD = Route Distinguisher is locally significant and does not have to match on all routers. It works like an ID number for a MPLSVPN routing table of a specific VRF.
Note: RD is only required when configuring MPLSVPN
The routers export their own routes towards a RT = Route-Target, and import routes from others routers with the same RT.
Note: Import and Export RT values doesn’t have to be the same but it makes sense if you want all DMVPN to be able to speak directly to each other.
Route targets can also be used to leak routes between VRFs. That is useful for configuring DIA = Direct Internet Access (not covered in this article).
But you probably know all and just want to know how it’s configured. This article is not about teaching all the concepts about MPLSVPN anyway.
Configuration
VRF Configuration
VRF Table
I’m going to configure the 3 VRFs shown in the overlay design:
VyOS Hub Router:
Cisco IOS-XE Spoke Router:
Note: The reason why the names and RDs are different between hub and spoke are of historical reasons. The important thing is that the route targets matches.
VyOS Hub Router
I’m only showing configuration for one VRF:
set vrf name NMS table '110'
set vrf name NMS protocols bgp parameters router-id '10.1.255.1'
set vrf name NMS protocols bgp system-as '65001'
set vrf name NMS protocols bgp address-family ipv6-unicast aggregate-address 2001:DB8:1234:A000::/60 summary-only
set vrf name NMS protocols bgp address-family ipv6-unicast export vpn
set vrf name NMS protocols bgp address-family ipv6-unicast import vpn
set vrf name NMS protocols bgp address-family ipv6-unicast label vpn export 'auto'
set vrf name NMS protocols bgp address-family ipv6-unicast rd vpn export '65001:1'
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute connected
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute ospfv3
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn export '65001:10003'
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn import '65001:10003'
VPNv6 Configuration:
set protocols bgp listen range 192.168.0.0/23 peer-group 'PG-MPLS'
set protocols bgp peer-group PG-MPLS remote-as '65001'
set protocols bgp peer-group PG-MPLS update-source 'dum0'
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-reflector-client
Explanation:
On VyOS you have to define a specific VRF table ID. On Cisco I assume that the table ID is created automatically.
On VyOS you can have different router ID’s per VRF. On Cisco it is one global ID for all VRFs.
On VyOS you can also configure a different AS number per VRF. On Cisco it is one AS for all VRFs.
Note: You can set local-as per neighbor, but that is not the same thing.
Cisco IOS-XE Spoke Router
! VRF Configuration
!
vrf definition MGMT
rd 65001:10
!
address-family ipv4
route-target export 65001:10003
route-target import 65001:10003
exit-address-family
!
address-family ipv6
route-target export 65001:10003
route-target import 65001:10003
exit-address-family
!
! BGP Configuration:
!
router bgp 65001
neighbor 192.168.0.1 remote-as 65001
neighbor 192.168.0.1 transport connection-mode active
neighbor 192.168.0.1 update-source Loopback0
!
address-family vpnv6
neighbor 192.168.0.1 activate
neighbor 192.168.0.1 send-community both
exit-address-family
!
address-family ipv6 vrf MGMT
redistribute ospf 1
aggregate-address 2001:DB8:1234:A000::/52
aggregate-address 2001:DB8:1234:A010::/60 summary-only
exit-address-family
!
!
Explanation:
Unlike VyOS, Cisco divides the VRF parameters and the BGP parameters into different sections.
MPLS Configuration
So far there is not much action yet. MPLS needs to be activated on the tunnel interface for routes inside the VRFs to be exchanged.
VyOS Hub Router
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp interface tun3 mpls forwarding
set protocols mpls interface 'tun3'
set protocols mpls ldp interface 'tun3'
set protocols mpls ldp router-id '192.168.0.1'
set protocols mpls ldp discovery transport-ipv4-address '192.168.0.1'
set protocols mpls ldp targeted-neighbor ipv4 enable
Explanation:
The configuration is for MPLS with LDPv4, because the underlay is IPv4 only.
No matter how hard I try I can’t establish dynamic LDP neighbor relationship over the tunnel interface. Therefore I had to enable targeted LDP. Only one side needs to specify neighbor address though. Read the appendix for more information.
I think this is the minimum required configuration. MPLS is so buggy that if I make any changes to it, VRF traffic stops and not even rollback works without requiring a reboot.
Source: https://docs.vyos.io/en/latest/configuration/protocols/mpls.html
Cisco IOS-XE Spoke Router
interface Tunnel3
mpls ip
!
mpls ldp neighbor 192.168.0.1 targeted
mpls ldp router-id Loopback0
Verification and Troubleshooting
Let’s see if IPv6 traffic inside the VRF can get through the tunnel. I don’t have any pingable address on the VyOS hub router at the moment so I’m creating a temporary dummy interface for testing:
set interfaces dummy dum1 address '2001:DB8:1234:A000::1/64'
set interfaces dummy dum1 vrf 'NMS'
Note: ‘redistribute connected’ under the BGP process has already been added.
Here I’m pinging the dummy interface from the Cisco IOS-XE Spoke router:
SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Fail! What could it be? In the routing table, we can see there is a route to the prefix:
SAUNA-RO1#show ipv6 route vrf MGMT 2001:DB8:1234:A000::/64
Routing entry for 2001:DB8:1234:A000::/64
Known via "bgp 65001", distance 200, metric 0, type internal
Route count is 1/1, share count 0
Routing paths:
FE80::C0A8:1%default indirectly connected
MPLS label: 80
From C0A8:1::
opaque_ptr 0x7F4C505408
Last updated 00:02:54 ago
Note: I don’t know why the route is not summarized to 2001:DB8:1234:A000::/60 as i have configured ‘aggregate address’ but that is a minor detail right now.
There is something odd with that next hop… It points towards a link local address! That link local address is the one on LBS-RO1 dummy dum0, configured earlier:
vyos@LBS-RO1:~$ show interfaces dummy dum0
dum0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether c2:13:bb:d2:22:a4 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/32 scope global dum0
valid_lft forever preferred_lft forever
inet6 fe80::c0a8:1/64 scope link
valid_lft forever preferred_lft forever
Description: BGP-RouterID
On the VyOS side it looks like this:
vyos@LBS-RO1:~$ show ipv6 route vrf NMS 2001:DB8:1234:A010::/60
Routing entry for 2001:db8:1234:a010::/60
Known via "bgp", distance 200, metric 2, vrf NMS, best
Last update 02:31:01 ago
r ::ffff:c0a8:3(vrf default) (recursive), label 43, weight 1
r 192.168.2.3, via tun3(vrf default) onlink, label 43, weight 1
This problem reminds me about when I configured BGP between my Cisco router and pfSense firewall.
The solution
The solution is to manually change the advertised next-hop to the tunnel interface, but that’s an IPv4 address!
You may have noticed in the output above that it has a strange IPv6 address as next-hop (::ffff:c0a8:3). The prefix ::ffff:0:0/96 is used for IPv4-mapped IPv6 addresses. c0a8:3 actually represents 192.168.0.3. But the real next-hop should be 192.168.2.3, hence why the route is marked as recursive.
It’s interesting although BGP is standard based, you can spot different behaviors between vendors. Anyway, let’s configure a route-map that manually sets the next-hop to be the tunnel interface on the hub router:
VyOS Hub Router route-map configuration
First create the route-map:
set policy route-map RM_DMVPN_OUT description 'set next-hop tunnel3'
set policy route-map RM_DMVPN_OUT rule 10 action 'permit'
set policy route-map RM_DMVPN_OUT rule 10 set community add 'no-export'
set policy route-map RM_DMVPN_OUT rule 10 set ipv6-next-hop global '::ffff:c0a8:201'
Note: “community add no-export”
makes sure that prefixes advertised between iBGP peers doesn’t get advertised outside the AS.
Then apply it to the VPNv6 neighbor:
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-map export 'RM_DMVPN_OUT'
Now let’s verify what happens on the Cisco router:
SAUNA-RO1#show ipv6 route vrf MGMT 2001:DB8:1234:A000::/64
Routing entry for 2001:DB8:1234:A000::/64
Known via "bgp 65001", distance 200, metric 0, type internal
Route count is 1/1, share count 0
Routing paths:
192.168.2.1%default indirectly connected
MPLS label: 80
From C0A8:1::
opaque_ptr 0x7F4C505408
Last updated 00:01:38 ago
SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Still can’t ping though, so I probably need to set the next hop manually on the spoke router as well.
Cisco IOS-XE Route-map Configuration
Route-map configuration:
route-map RM_DMVPN_OUT permit 10
set community no-export
set ipv6 next-hop ::FFFF:192.168.2.3
Note: I actually typed “::ffff:c0a8:203” but the running configuration is displaying it as “192.168.2.3”.
Then apply it to the bgp vpnv6 neighbor:
address-family vpnv6
neighbor 192.168.0.1 route-map RM_DMVPN_OUT out
The moment of truth
Before pinging the remote address, let’s check how the route is looking at the hub side:
vyos@LBS-RO1:~$ show ipv6 route vrf NMS 2001:DB8:1234:A010::/60
Routing entry for 2001:DB8:1234:a010::/60
Known via "bgp", distance 200, metric 2, vrf NMS, best
Last update 00:04:23 ago
* ::ffff:c0a8:203, via tun3(vrf default) onlink, label 43, weight 1
It’s not marked as recursive anymore. Let’s do the ping! drum roll please….
SAUNA-RO1#ping vrf MGMT 2001:DB8:1234:A000::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A000::1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
vyos@LBS-RO1:~$ ping 2001:DB8:1234:A010::1 vrf NMS
PING 2001:DB8:1234:A010::1(2001:db8:1234:a010::1) 56 data bytes
64 bytes from 2001:db8:1234:a010::1: icmp_seq=1 ttl=62 time=2.33 ms
64 bytes from 2001:db8:1234:a010::1: icmp_seq=2 ttl=62 time=2.35 ms
64 bytes from 2001:db8:1234:a010::1: icmp_seq=3 ttl=62 time=2.40 ms
^C
--- 2001:DB8:1234:A010::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 2.330/2.361/2.401/0.029 ms
Success! You have no idea how many hours I have spent before this moment but wow that is satisfying!
Appendix
Special Mention
I want to thank Level Zero Networking for their blogs on a similar MPLSVPN setup. It was a good foundation to start with for making this lab.
Part 1: https://lev-0.com/2024/01/08/dynamic-multipoint-vpn-with-zerotier-and-vyos/
Part 2: https://lev-0.com/2024/01/09/dynamic-multipoint-vpn-with-zerotier-and-vyos-part-2-mpls/
Caveats discovered
Confusion about enabling MPLS LDP on Cisco IOS-XE
There is some confusion about how to enable MPLS on a tunnel interface. According to Cisco, you can use either ‘mpls bgp forwarding’
or the ‘mpls nhrp’
command, but the documentation is very unclear:
‘mpls nhrp’
apparently enables tag switching, which I assume means TDP, which is Cisco proprietary.The documentation doesn’t explicitly say if ‘
mpls bgp forwarding’
enables TDP or LDP. If they claim you can use either of them, I will assume it also enables TDP.
‘mpls ip’
apparently enables LDP, so that is the command I use.
Multicast issue on the VyOS tunnel interface
Whatever I do I can’t get MPLS LDP neighborship up without using targeted LDP. I discovered when troubleshooting through native FRR console that these log messages are appearing:
LBS-RO1# terminal monitor
2025-03-17 19:00:51.868 [WARN] ldpd: [HD2R1-Z7JJT] send_packet: error sending packet to 224.0.0.2
2025-03-17 19:00:56.870 [WARN] ldpd: [HD2R1-Z7JJT] send_packet: error sending packet to 224.0.0.2
At the moment I can’t do anything about it so I just have to live with it.
Bug Report issued
I have created the following bug report: https://vyos.dev/T7266
Full Configuration
Down below is the full configuration, including part 1:
VyOS Hub Router Configuration
set interfaces dummy dum0 address '192.168.0.1/32'
set interfaces dummy dum0 address 'FE80::c0a8:1/64'
set interfaces dummy dum0 description 'BGP-RouterID'
set interfaces dummy dum0 ipv6 address no-default-link-local
set interfaces ethernet eth1 vif 3000 address '2001:DB8:1234::3/64'
set interfaces ethernet eth1 vif 3000 address 'fe80::3/64'
set interfaces ethernet eth1 vif 3000 address '10.0.1.3/26'
set interfaces ethernet eth1 vif 3000 description 'VLAN 3000 External'
set interfaces ethernet eth1 vif 3000 ipv6 address no-default-link-local
set interfaces tunnel tun3 address '192.168.2.1/32'
set interfaces tunnel tun3 enable-multicast
set interfaces tunnel tun3 encapsulation 'gre'
set interfaces tunnel tun3 ip adjust-mss '1360'
set interfaces tunnel tun3 ipv6 adjust-mss '1340'
set interfaces tunnel tun3 mtu '1400'
set interfaces tunnel tun3 parameters ip key '3'
set interfaces tunnel tun3 source-interface 'eth1.3000'
set protocols nhrp tunnel tun3 authentication 'secret'
set protocols nhrp tunnel tun3 holdtime '60'
set protocols nhrp tunnel tun3 multicast 'dynamic'
set protocols nhrp tunnel tun3 network-id '3'
set protocols nhrp tunnel tun3 redirect
set protocols nhrp tunnel tun3 registration-no-unique
set vpn ipsec profile DMVPN bind tunnel 'tun3'
set vpn ipsec esp-group DMVPN-ESP lifetime '3600'
set vpn ipsec esp-group DMVPN-ESP mode 'transport'
set vpn ipsec esp-group DMVPN-ESP pfs 'dh-group2'
set vpn ipsec esp-group DMVPN-ESP proposal 1 encryption 'aes256'
set vpn ipsec esp-group DMVPN-ESP proposal 1 hash 'sha1'
set vpn ipsec ike-group DMVPN-IKE key-exchange 'ikev2'
set vpn ipsec ike-group DMVPN-IKE lifetime '28800'
set vpn ipsec ike-group DMVPN-IKE proposal 1 dh-group '2'
set vpn ipsec ike-group DMVPN-IKE proposal 1 encryption 'aes256'
set vpn ipsec ike-group DMVPN-IKE proposal 1 hash 'sha1'
set vpn ipsec interface 'eth1.3000'
set vpn ipsec profile DMVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile DMVPN esp-group 'DMVPN-ESP'
set vpn ipsec profile DMVPN ike-group 'DMVPN-IKE'
set protocols bgp parameters log-neighbor-changes
set protocols bgp parameters router-id '192.168.0.1'
set protocols bgp listen range 192.168.0.0/23 peer-group 'PG-MPLS'
set protocols bgp listen range 192.168.2.0/23 peer-group 'PG-DMVPN'
set protocols bgp peer-group PG-DMVPN address-family ipv4-unicast route-reflector-client
set protocols bgp peer-group PG-DMVPN passive
set protocols bgp peer-group PG-DMVPN password 'secret'
set protocols bgp peer-group PG-DMVPN remote-as '65001'
set protocols bgp peer-group PG-MPLS address-family ipv4-vpn route-reflector-client
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-map export 'RM_DMVPN_OUT'
set protocols bgp peer-group PG-MPLS address-family ipv6-vpn route-reflector-client
set protocols bgp peer-group PG-MPLS remote-as '65001'
set protocols bgp peer-group PG-MPLS update-source 'dum0'
set protocols bgp system-as '65001'
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp interface tun3 mpls forwarding
set protocols mpls interface 'tun3'
set protocols mpls ldp discovery transport-ipv4-address '192.168.0.1'
set protocols mpls ldp interface 'tun3'
set protocols mpls ldp router-id '192.168.0.1'
set protocols mpls ldp targeted-neighbor ipv4 enable
set vrf name NMS table '110'
set vrf name NMS protocols bgp parameters router-id '10.1.255.1'
set vrf name NMS protocols bgp system-as '65001'
set vrf name NMS protocols bgp address-family ipv6-unicast aggregate-address 2001:DB8:1234:A000::/60 summary-only
set vrf name NMS protocols bgp address-family ipv6-unicast export vpn
set vrf name NMS protocols bgp address-family ipv6-unicast import vpn
set vrf name NMS protocols bgp address-family ipv6-unicast label vpn export 'auto'
set vrf name NMS protocols bgp address-family ipv6-unicast rd vpn export '65001:1'
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute connected
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute ospfv3
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn export '65001:10003'
set vrf name NMS protocols bgp address-family ipv6-unicast route-target vpn import '65001:10003'
set policy route-map RM_DMVPN_OUT description 'set next-hop tunnel3'
set policy route-map RM_DMVPN_OUT rule 10 action 'permit'
set policy route-map RM_DMVPN_OUT rule 10 set ipv6-next-hop global '::ffff:c0a8:201'
Notes:
VyOS software version used: VyOS 1.5-rolling-202502110508
Interface “dummy dum1” used for testing not included.
Cisco IOS-XE Spoke Router Configuration
interface Loopback0
description BGP_RouterID
ip address 192.168.0.3 255.255.255.255
ipv6 address FE80::C0A8:3 link-local
!
interface GigabitEthernet0/0/1.3000
description FW-EXTERNAL
encapsulation dot1Q 3000
vrf forwarding EXTERNAL
ip address 10.0.1.2 255.255.255.192
ipv6 address FE80::2 link-local
ipv6 address 2001:DB8:1234::2/64
ipv6 nd ra suppress all
!
interface Tunnel3
ip address 192.168.2.3 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication secret
ip nhrp network-id 3
ip nhrp holdtime 60
ip nhrp nhs 192.168.2.1 nbma 10.0.1.3
ip nhrp redirect
ip tcp adjust-mss 1360
ipv6 mtu 1400
ipv6 tcp adjust-mss 1340
mpls ip
tunnel source GigabitEthernet0/0/1.3000
tunnel mode gre multipoint
tunnel key 3
tunnel path-mtu-discovery
tunnel vrf EXTERNAL
tunnel protection ipsec profile DMVPN
!
!
crypto ikev2 proposal IKEV2-PROPOSAL
encryption aes-cbc-256
integrity sha1
group 2
!
crypto ikev2 policy IKEV2-POLICY
match fvrf any
proposal IKEV2-PROPOSAL
!
crypto ikev2 keyring DMVPN-KEYRING
peer DMVPN-PEER
address 0.0.0.0 0.0.0.0
pre-shared-key secret
!
crypto ikev2 profile DMVPN-IKE
match fvrf any
match identity remote address 0.0.0.0
authentication remote pre-share
authentication local pre-share
keyring local DMVPN-KEYRING
dpd 30 5 on-demand
!
crypto ipsec transform-set DMVPN-ESP esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile DMVPN
set transform-set DMVPN-ESP
set ikev2-profile DMVPN-IKE
!
!
vrf definition MGMT
rd 65001:10
!
address-family ipv4
route-target export 65001:10003
route-target import 65001:10003
exit-address-family
!
address-family ipv6
route-target export 65001:10003
route-target import 65001:10003
exit-address-family
!
router bgp 65001
bgp router-id 192.168.0.3
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 192.168.0.1 remote-as 65001
neighbor 192.168.0.1 transport connection-mode active
neighbor 192.168.0.1 update-source Loopback0
neighbor 192.168.2.1 remote-as 65001
neighbor 192.168.2.1 transport connection-mode active
neighbor 192.168.2.1 password secret
!
address-family ipv4
network 192.168.0.3 mask 255.255.255.255
neighbor 192.168.2.1 activate
exit-address-family
!
address-family vpnv6
neighbor 192.168.0.1 activate
neighbor 192.168.0.1 send-community both
neighbor 192.168.0.1 route-map RM_DMVPN_OUT out
exit-address-family
!
address-family ipv6 vrf MGMT
redistribute ospf 1
aggregate-address 2001:DB8:1234:A010::/60 summary-only
aggregate-address 2001:DB8:1234:A000::/52
!
!
route-map RM_DMVPN_OUT permit 10
set ipv6 next-hop ::FFFF:192.168.2.3
Note: Cisco IOS-XE software version used: 17.03.04a
Possible Improvements
DIA = Direct Internet Access
As I mentioned in part 1, DIA can be enabled on the spoke router. But to do that securely it requires ZBF = Zone-Based Firewall configuration. That is a future article in itself.
Stronger security with TLS certificates
Once I have a functioning PKI infrastructure, I will make an update using SSL certificates instead of pre-shared keys.
Dual hub and multiple spokes
There are still some unanswered questions about if spoke-to-spoke traffic will work since I don’t have a second spoke to test with.
Also, dual hub is nice for testing redundancy and load balancing.