Linux: Configure MPLSoDMVPN Part 3: Enable routing over the DMVPN tunnel

How-To: Linux MPLSoDMVPN Part 3

Jun 02, 2025

Monday June 2 2025

When I finished part 1 and part 2, I thought the worst was over. However it turned out that this part took just as long as the other ones, if not more! There were a few pitfalls, caveats and limitations discovered that I will cover in the appendix.

The story so far

In part 1 I configured the interfaces and some required iptables rules for optimal GRE tunnel performance. Linux Debian works best, even if it’s a bit unintuitive to configure network settings compared to Ubuntu.

In part 2 I configured the DMVPN tunnel interface between FRR hub router and a Cisco Spoke router. I also configured the IPSec profile using a modified version of Strongswan.

Now we are up to the point where we need routing.

Underlay Routing Configuration

Design

Explanation:

The underlay routing protocol is going to be OSPFv2.
The iBGP neighborship to the uplink router is out of scope for this project.
Dummy0 on the Linux servers and Loopback0 on the Cisco routers are used for Router ID’s.
A linknet between the servers is implemented so that the spoke router won’t become a transit router for traffic destined between servers.
The high-availability setup is out of scope for this project. Therefor the configuration might not be optimized for dual-hub at this moment.

OSPF Configuration

Linux FRR Hub router OSPF Configuration

interface ens23.40
 no ip ospf passive
exit
!
interface gre1
 description DMVPN Tunnel Interface
 ip ospf hello-interfal 10
 ip ospf dead-interval 40
 ip ospf network point-to-multipoint
 multicast enable
 no ip ospf passive
!
router ospf
 ospf router-id 192.168.0.1
 passive-interface default
 network 10.14.0.0/24 area 0
 network 192.168.0.1/32 area 0
 network 192.168.1.1/32 area 0

Explanation:

Interface ens23.40 is the linknet between the Linux servers
Only configuration related to OSPF is included under the GRE tunnel interface. Complete configuration will be available in the appendix.

Cisco Spoke Router OSPF Configuration

interface Loopback0
 description RouterID
 ip address 192.168.0.3 255.255.255.255
 ip ospf 1 area 0
!
interface Tunnel1
 description DMVPN Tunnel Interface
 ip ospf network point-to-multipoint
 ip ospf dead-interval 40
 ip ospf hello-interval 10
 ip ospf 1 area 0
!
router ospf 1
 router-id 192.168.0.3
 passive-interface default
 no passive-interface Tunnel1

Verification

SAUNA-RO1#show ip route ospf
...

Gateway of last resort is 10.14.255.0 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 4 masks
O        10.14.0.0/24 [110/1010] via 192.168.1.2, 00:14:56, Tunnel1
                      [110/1010] via 192.168.1.1, 06:16:53, Tunnel1
      192.168.0.0/32 is subnetted, 3 subnets
O        192.168.0.1 [110/1010] via 192.168.1.1, 06:32:33, Tunnel1
O        192.168.0.2 [110/1010] via 192.168.1.2, 00:15:19, Tunnel1
      192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
O        192.168.1.1/32 [110/1000] via 192.168.1.1, 06:32:33, Tunnel1
O        192.168.1.2/32 [110/1000] via 192.168.1.2, 00:15:19, Tunnel1

As we can see, both servers Dummy interface address are in the routing table. Let’s try a ping from Cisco Lo0 to FRR Dummy0 for assurance:

SAUNA-RO1#ping 192.168.0.1 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

Linux configuration changes

So far no bigger challenges, but now it starts to become interesting. Before doing anything more in FRR, a few changes in the Linux system are required.

Up to this point only internal traffic destined to the server itself has been needed. For Linux to forward traffic destined to other hosts, forwarding has to be configured. All those settings can be enabled with sysctl.

IPv6 and IPv4 forwarding

You may have noticed that inside the running configuration of FRR, you can find “no ip forwarding” and “no ipv6 forwarding”. Those settings cannot be altered from FRR.

The settings needed to be changed can instead be found inside /etc/sysctl.conf. Uncomment these lines to enable IPv4 and IPv6 forwarding:

~$ sudo nano /etc/sysctl.conf
------------------------------------------------------------------------
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
net.ipv6.conf.all.forwarding=1

Activate the new settings with this command:

~$ sudo sysctl -p

Source: https://linuxconfig.org/how-to-turn-on-off-ip-forwarding-in-linux

MPLS forwarding

Load MPLS modules

MPLS modules may be needed to load before specific settings can be edited.

sudo modprobe -v mpls_router mpls_gso mpls_iptunnel

To make these modules load at boot, edit /etc/modules-load.d/modules.conf or alternatively create your own file:

$ sudo nano /etc/modules-load.d/mpls.conf
------------------------------------------------------------------------
mpls_gso
mpls_router
mpls_iptunnel

Change MPLS Settings

MPLS settings can also be enabled in the same file as ip forwarding, but I chose to create a dedicated file for mpls settings:

~$ sudo nano /etc/sysctl.d/mpls.conf
------------------------------------------------------------------------
net.mpls.platform_labels = 1048575
net.mpls.conf.gre1.input = 1
net.mpls.conf.ens23/40.input = 1

Activate with:

~$ sudo sysctl -p /etc/sysctl.d/mpls.conf

Source: https://stackoverflow.com/questions/31926342/iproute2-commands-for-mpls-configuration

Caveat Discovered

MPLS doesn’t get activated on the GRE tunnel interface without manually running “sysctl -p” after a reboot. This is likely cause by some related GRE module or something loading after MPLS. The sysctl is only run once.

Add `--v6-with-v4-nexthops` to Zebra config

Zebra is the protocol that communicates between FRR and the Linux network kernel.

To allow IPv6 routes with a IPv4 next-hop to be installed in the kernel routing table, a specific zebra option has to be added to /etc/frr/daemons:

~$ sudo nano /etc/frr/daemons
------------------------------------------------------------------------
...
zebra_options="  -A 127.0.0.1 -s 90000000 --v6-with-v4-nexthops"

Source: https://docs.frrouting.org/en/latest/zebra.html#invoking-zebra

Now I can start configuring FRR for MPLS and BGP.

Overlay Routing Configuration

Design

Explanation:

3 VRF’s defined. The Route targets are the same for export and import.
It’s important to know that BGP next-hop behaviour is different on Linux FRR and Cisco IOS-XE. Therefore:
- The BGP source address for the hub routers will be the gre1 interface addresses.
- The BGP source address for the spoke router will be tunnel1. The reason for this is because the Linux kernel allegedly doesn’t support recursive routing lookups. This will be mentioned in the appendix.
- The Cisco router is not as picky. It will install the BGP routes regardless if the next-hop is tunnel1 or the hub routers dummy interface.
I have not tested this design with a FRR spoke router yet.

MPLS Configuration

FRR Hub Router MPLS Configuration

mpls ldp
 router-id 192.168.0.1
 dual-stack cisco-interop
 !
 address-family ipv4
  discovery targeted-hello accept
  discovery transport-address 192.168.0.1
  label local allocate host-routes
  !
  interface ens23.40
  exit
  !
  interface gre1
  exit
  !
 exit-address-family
 !
exit
!
interface gre1
 mpls bgp forwarding

Note: Even though multicast has been configured for the GRE tunnel with iptables and OSPF neighborship forms dynamically as a proof of that, same thing does not work for LDP. In the FRR terminal I can see these messages constantly:

2025-04-29 22:34:55.702 [WARN] ldpd: [HD2R1-Z7JJT] send_packet: error sending packet to 224.0.0.2

Cisco IOS-XE Spoke Router MPLS Configuration

Luckily there is a workaround and that is to configure targeted LDP on the spoke router:

mpls ldp neighbor 192.168.0.1 targeted
mpls ldp neighbor 192.168.0.2 targeted
mpls ldp router-id Loopback0
!
interface Tunnel1
 mpls ip

MPLS Verification

Linux FRR Hub Router

sauna-vm1# show mpls ldp neighbor 
AF   ID              State       Remote Address    Uptime
ipv4 192.168.0.3     OPERATIONAL 192.168.0.3     03:33:43
ipv4 192.168.0.2     OPERATIONAL 192.168.0.2     03:33:45

BGP Configuration

FRR Hub Router BGP related Configuration

A Route-Map to manually define the next-hop is needed on the hub router. Otherwise the routes won’t be propagated to the Cisco Spoke router.

route-map RM_VPNV6_OUT permit 10
 set as-path prepend 65001
 set ipv6 next-hop global ::ffff:192.168.1.1
exit
!
route-map RM_VPNV4_OUT permit 10
 set as-path prepend 65001
 set ip next-hop 192.168.1.1
exit

Explanation:

A default route is advertised from the hub routers. For some reason, that route becomes preferred over the eBGP default route installed on the spoke router, even if eBGP routes should be preferred over iBGP routes. I found out that “set as-path prepend 65001” fixes that problem. Alternatively, i could just filter out the default route.
For IPv4 next-hop I’m using the tunnel interface address.
For IPv6 next-hop I’m using the IPv4-mapped IPv6 address corresponding to the tunnel interface address.

BGP Configuration:

vrf NMS
 ipv6 route 2001:db8:1234:a000::/60 blackhole tag 4 254
exit-vrf
!
router bgp 65001
 bgp router-id 192.168.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor PG-MPLS peer-group
 neighbor PG-MPLS remote-as 65001
 neighbor PG-MPLS passive
 neighbor PG-MPLS update-source gre1
 neighbor PG-MPLS timers 15 45
 neighbor 192.168.0.2 remote-as 65001
 neighbor 192.168.0.2 update-source dummy0
 bgp listen range 192.168.1.0/24 peer-group PG-MPLS
 !
 address-family ipv4 unicast
  network 192.168.0.1/32
 exit-address-family
 !
 address-family ipv4 labeled-unicast
  neighbor PG-MPLS activate
 exit-address-family
 !
 address-family ipv4 vpn
  neighbor PG-MPLS activate
  neighbor PG-MPLS route-reflector-client
  neighbor PG-MPLS route-map RM_VPNV4_OUT out
  neighbor 192.168.0.2 activate
 exit-address-family
 !
 address-family ipv6 vpn
  neighbor PG-MPLS activate
  neighbor PG-MPLS route-reflector-client
  neighbor PG-MPLS soft-reconfiguration inbound
  neighbor PG-MPLS route-map RM_VPNV6_OUT out
  neighbor 192.168.0.2 activate
 exit-address-family
exit
!
router bgp 65001 vrf NMS
 bgp router-id 10.10.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 !
 address-family ipv4 unicast
  redistribute connected route-map RM_SET_COMMUNITIES
  label vpn export auto
  rd vpn export 65001:10
  rt vpn both 65001:10
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute static route-map RM_SET_COMMUNITIES
  label vpn export auto
  rd vpn export 65001:10
  rt vpn both 65001:10
  export vpn
  import vpn
 exit-address-family
exit

Explanation:

The BGP configuration includes relevant parameters for MPLSVPN and one VRF for sample configuration.
The route-map included in the redistribute commands is optional. I’m using it to include some communities that can be used for route filtering.

Cisco IOS-XE Spoke Router BGP related Configuration

On Cisco, the VRF is configured separately:

vrf definition NMS
 rd 65001:10
 !
 address-family ipv4
  route-target export 65001:10
  route-target import 65001:10
 exit-address-family
 !
 address-family ipv6
  route-target export 65001:10
  route-target import 65001:10
 exit-address-family

BGP Configuration:

router bgp 65001
 bgp router-id 192.168.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.1.1 remote-as 65001
 neighbor 192.168.1.1 transport connection-mode active
 neighbor 192.168.1.1 update-source Tunnel1
 neighbor 192.168.1.2 remote-as 65001
 neighbor 192.168.1.2 transport connection-mode active
 neighbor 192.168.1.2 update-source Tunnel1
 !
 address-family ipv4
  network 192.168.0.3 mask 255.255.255.255
 exit-address-family
 !
 address-family vpnv4
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 send-community both
  neighbor 192.168.1.2 activate
  neighbor 192.168.1.2 send-community both
 exit-address-family
 !
 address-family vpnv6
  bgp scan-time 15
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 send-community both
  neighbor 192.168.1.2 activate
  neighbor 192.168.1.2 send-community both
 exit-address-family
 !
 address-family ipv4 vrf NMS
  aggregate-address 10.10.0.0 255.255.0.0 summary-only attribute-map RM_SET_COMMUNITIES
  redistribute ospfv3 1 route-map RM_SET_COMMUNITIES
 exit-address-family
 !
 address-family ipv6 vrf NMS
  redistribute ospf 1 route-map RM_SET_COMMUNITIES
  aggregate-address FC01::/16 summary-only attribute-map RM_SET_COMMUNITIES
  aggregate-address 2001:DB8:1234:A000::/52 summary-only attribute-map RM_SET_COMMUNITIES
 exit-address-family

Explanation:

All configuration related to MPLSVPN and one of the VRFs are included.
Notice that the BGP source address is set to tunnel1. Setting it to Loopback0 will not work (for IPv6 routes at least).

Overlay routing Verification

Note: SAUNA-VM2 has been turned off for simplicity.

Linux FRR Hub Router Overlay Routing Verification

First we can check the kernel routing table:

~$ ip -6 route show vrf NMS
anycast 2001:db8:1234:a001:: dev ens23.10 proto kernel metric 0 pref medium
2001:db8:1234:a001::/64 dev ens23.10 proto kernel metric 256 pref medium
blackhole 2001:db8:1234:a000::/60 dev lo proto static metric 20 pref medium
2001:db8:1234:a000::/52 nhid 1044  encap mpls  60 via ::ffff:192.168.1.3 dev gre1 proto bgp metric 20 onlink pref medium
2001:db8:1234:8000::/49 nhid 1051  encap mpls  59 via ::ffff:192.168.1.3 dev gre1 proto bgp metric 20 onlink pref medium
fc01::/16 nhid 1045  encap mpls  48 via ::ffff:192.168.1.3 dev gre1 proto bgp metric 20 onlink pref medium
anycast fe80:: dev ens19 proto kernel metric 0 pref medium
anycast fe80:: dev ens23.10 proto kernel metric 0 pref medium
fe80::/64 dev ens19 proto kernel metric 256 pref medium
fe80::/64 dev ens23.10 proto kernel metric 256 pref medium
multicast ff00::/8 dev ens19 proto kernel metric 256 pref medium
multicast ff00::/8 dev ens23.10 proto kernel metric 256 pref medium
default nhid 760 via fe80::1:1 dev ens19 proto bgp metric 20 pref medium

Pay attention to the routes marked with “encap mpls” and a next-hop with “::ffff:192.168.1.3”.

Verification from FRR:

sauna-vm1# show ipv6 route vrf NMS bgp
...
IPv6 unicast VRF NMS:
B>* ::/0 [200/0] via fe80::1:1, ens19, weight 1, 03:51:20
B>* 2001:db8:1234:8000::/49 [200/0] via ::ffff:192.168.1.3, gre1 (vrf default) onlink, label 59, weight 1, 00:23:17
B>* 2001:db8:1234:a000::/52 [200/0] via ::ffff:192.168.1.3, gre1 (vrf default) onlink, label 60, weight 1, 00:23:17
B>* fc01::/16 [200/0] via ::ffff:192.168.1.3, gre1 (vrf default) onlink, label 48, weight 1, 00:23:17

sauna-vm1# show bgp vrf NMS ipv6
...

     Network          Next Hop            Metric LocPrf Weight Path
 *>i ::/0             fe80::1:1                0    100      0 i
 *>  2001:db8:1234:8000::/49
                    ::ffff:192.168.1.3@0<
                                             0    100      0 65000 i
 *>  2001:db8:1234:a000::/52
                    ::ffff:192.168.1.3@0<
                                             0    100      0 i
 *>  2001:db8:1234:a000::/60
                    ::                       0         32768 ?
 *>  fc01::/16        ::ffff:192.168.1.3@0<
                                             0    100      0 i

sauna-vm1# show bgp vrf NMS ipv6 2001:db8:1234:a000::/52
BGP routing table entry for 2001:db8:1234:a000::/52, version 72
Paths: (1 available, best #1, vrf NMS)
  Not advertised to any peer
  Imported from 65001:10:2001:db8:1234:a000::/52
  Local, (aggregated by 65001 192.168.0.3)
    ::ffff:192.168.1.3 from :: (10.10.0.1) vrf default(0) announce-nh-self
      Origin IGP, metric 0, localpref 100, valid, sourced, local, atomic-aggregate, best (First path received)
      Community: 65001:2
      Extended Community: RT:65001:10
      Remote label: 60
      Last update: Wed Apr 30 12:56:17 2025

You will notice that the above output will look different on Cisco IOS-XE. For example “announce-nh-self” allegedly indicates that the router advertises itself as the next-hop to other neighbors, even though “next-hop-self” is not configured on the peer-group.

Cisco IOS-XE Spoke Router Overlay Routing Verification

SAUNA-RO1#show bgp vpnv6 unicast vrf NMS 2001:DB8:1234:A000::/60
BGP routing table entry for [65001:10]2001:DB8:1234:A000::/60, version 132
Paths: (2 available, best #1, table NMS, not advertised to EBGP peer, Advertisements suppressed by an aggregate.)
  Not advertised to any peer
  Refresh Epoch 1
  65001
    ::FFFF:192.168.1.1 (metric 1000) (via default) from 192.168.1.1 (192.168.0.1)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Community: 65001:1 no-export
      Extended Community: RT:65001:10
      mpls labels in/out nolabel/81
      rx pathid: 0, tx pathid: 0x0
      Updated on Apr 30 2025 17:48:54 CEST

Note that this output doesn’t have announce-nh-self and “from” displays the source BGP address of SAUNA-VM1. On SAUNA-VM1, it just says “from ::”. That’s just how it is.

SAUNA-RO1#show bgp vpnv6 unicast vrf NMS                         
...
Route Distinguisher: 65001:10 (default for vrf NMS)
 * i  ::/0             ::FFFF:192.168.1.1
                                                0    100      0 65001 i
 *>                    2001:DB8:1234:A::1
                                                0             0 65000 i
 *>   2001:DB8:1234:8000::/49
                      2001:464F:6F83:A::1
                                                0             0 65000 i
 s>i  2001:DB8:1234:A000::/60
                      ::FFFF:192.168.1.1
                                                0    100      0 65001 ?
 *>   2001:DB8:1234:A000::/52
                      ::                                 32768 i
 s>   2001:DB8:1234:A010::/60
                      FE80::CAFE:2             2         32768 ?
 s>   FC01::/64        FE80::CAFE:2             2         32768 ?
 *>   FC01::/16        ::                                 32768 i

Explanation:

Notice that AS 65001 is included in the AS-PATH for routes originating from SAUNA-VM1.
Also notice that the local default route is preferred over the one learnt from SAUNA-VM1.

SAUNA-RO1#show ipv6 route vrf NMS bgp
...
B   ::/0 [20/0], tag 65000
     via FE80::F2AD:4EFF:FE28:B1FF, GigabitEthernet0/0/1.3010
B   2001:DB8:1234:8000::/49 [20/0], tag 65000
     via FE80::F2AD:4EFF:FE28:B1FF, GigabitEthernet0/0/1.3010
B   2001:DB8:1234:A000::/52 [200/0]
     via Null0%default, directly connected
B   2001:DB8:1234:A000::/60 [200/0], tag 65001
     via 192.168.1.1%default, indirectly connected
B   FC01::/16 [200/0]
     via Null0%default, directly connected

End-to-End Verification

Here I’m pinging from a switch downstream from SAUNA-RO1:

SAUNA-SW1#ping 2001:DB8:1234:A001::1 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1234:A001::1, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:1234:A010::1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/2 ms

Appendix

Troubleshooting and Caveats

I don’t know how clear it is from this documentation that this project took way longer than expected. There wasn’t exactly any complete guide on how to set this up so there was a lot of trial and error, confusion and guessing to get it work.

Prefixes becomes invalid after one hour

For some reason, the IPv6 prefixes are removed from the routing table after a while:

sauna-vm1# show bgp vrf NMS ipv6
...
     Network          Next Hop            Metric LocPrf Weight Path
 *>i ::/0             fe80::1:1                0    100      0 i
  =  2001:db8:1234:8000::/49
                    ::ffff:192.168.1.3@0<
                                             0    100      0 65000 i
  =  2001:db8:1234:a000::/52
                    ::ffff:192.168.1.3@0<
                                             0    100      0 i
 *>  2001:db8:1234:a000::/60
                    ::                       0         32768 ?
  =  fc01::/16        ::ffff:192.168.1.3@0<
                                             0    100      0 i

Notice that “*>” is missing from prefixes learnt from SAUNA-RO1. It works fine when I’m actively pinging from time to time but when I have been away from the computer for a while, the routes are gone.

It definately corresponds with these events that can be seen on the Cisco router:

*Apr 30 22:08:39: %DMVPN-5-CRYPTO_SS:  Tunnel1: local address : 10.14.255.1 remote address : 10.4.254.41 socket is DOWN
*Apr 30 22:08:39: %DMVPN-5-NHRP_NHS_DOWN: Tunnel1: Next Hop Server : (Tunnel: 192.168.1.1 NBMA: 10.4.254.41 ) for (Tunnel: 192.168.1.3 NBMA: 10.14.255.1) is DOWN, Reason: External(NHRP: no error)
*Apr 30 22:08:40: %DMVPN-5-CRYPTO_SS:  Tunnel1: local address : 10.14.255.1 remote address : 10.4.254.41 socket is UP
*Apr 30 22:08:40: %DMVPN-5-NHRP_NHS_UP: Tunnel1: Next Hop Server : (Tunnel: 192.168.1.1 NBMA: 10.4.254.41) for (Tunnel: 192.168.1.3 NBMA: 10.14.255.1) is  UP

Those events happens exactly once every hour. Until right before the event, the routes are available. When the DMVPN restarts, the routes are lost.

It turned out it was because of incorrect IPSec config (Strongswan). Don’t worry, I found the problem before part 2 was released so the config there is up to date.

FRR wont’ install IPv6 BGP recursive routes into the routing table.

Normally I would use Loopback0 as the next-hop for iBGP routes and rely on the underlying OSPF infrastructure to guarantee next-hop reachability, in case there are backup routes. Sadly, this doesn’t seem to work with FRR.

Even though SAUNA-RO1 Loopback0 is reachable through OSPF and iBGP prefixes are marked as valid, the prefixes won’t be installed in the IPv6 routing table. This could be due to the fact that Linux doesn’t support recursive routing.

If I manually set the next-hop to the tunnel IP with a route-map, like I did when i configured the same setup with VyOS, the routes becomes invalid. Only occasionally I have seen some prefixes get installed in the routing table, but end-to-end reachability still not possible. When I tried an older version of FRR I noticed that the next-hop was marked “inaccessible” even if I could ping it.

Since I couldn’t use loopback0 as source IP for iBGP, and manipulating the next-hop didn’t work either, my solution was to use the tunnel IPs as BGP source addresses instead. This doesn’t affect the topology much as the tunnel interface must be reachable anyway.

Full Routing Configuration

This covers the full FRR and IOS-XE configuration (as related to this project), but does not cover all Linux system configuration covered in part 1 and 2.

Linux FRR Hub Router

Versions used:

FRRouting 10.3 (sauna-vm1) on Linux(6.8.0-58-generic).
FRRouting 10.3 (sauna-vm2) on Linux(6.1.0-34-amd64).

hostname sauna-vm1
!
nhrp nflog-group 1
nhrp multicast-nflog-group 2
!
!
route-map RM_SET_COMMUNITIES permit 10
 match tag 4
 set community 65001:1 no-export
!
route-map RM_VPNV6_OUT permit 10
 set as-path prepend 65001
 set ipv6 next-hop global ::ffff:192.168.1.1
exit
!
route-map RM_VPNV4_OUT permit 10
 set as-path prepend 65001
 set ip next-hop 192.168.1.1
exit
!
!
vrf NMS
 ipv6 route 2001:db8:1234:a000::/60 blackhole tag 4 254
exit-vrf
!
!
interface ens23.40
 no ip ospf passive
exit
!
interface gre1
 description DMVPN Tunnel Interface
 ip nhrp authentication sauna123
 ip nhrp holdtime 60
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 ip nhrp redirect
 ip nhrp shortcut
 ip ospf dead-interval 40
 ip ospf network point-to-multipoint
 mpls bgp forwarding
 multicast enable
 no ip ospf passive
 tunnel protection vici profile dmvpn
 tunnel source ens22
exit
!
router bgp 65001
 bgp router-id 192.168.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor PG-MPLS peer-group
 neighbor PG-MPLS remote-as 65001
 neighbor PG-MPLS passive
 neighbor PG-MPLS update-source gre1
 neighbor PG-MPLS timers 15 45
 neighbor 192.168.0.2 remote-as 65001
 neighbor 192.168.0.2 update-source dummy0
 bgp listen range 192.168.1.0/24 peer-group PG-MPLS
 !
 address-family ipv4 unicast
  network 192.168.0.1/32
 exit-address-family
 !
 address-family ipv4 labeled-unicast
  neighbor PG-MPLS activate
 exit-address-family
 !
 address-family ipv4 vpn
  neighbor PG-MPLS activate
  neighbor PG-MPLS route-reflector-client
  neighbor PG-MPLS route-map RM_VPNV4_OUT out
  neighbor 192.168.0.2 activate
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected route-map RM_SET_COMMUNITIES
 exit-address-family
 !
 address-family ipv6 vpn
  neighbor PG-MPLS activate
  neighbor PG-MPLS route-reflector-client
  neighbor PG-MPLS soft-reconfiguration inbound
  neighbor PG-MPLS route-map RM_VPNV6_OUT out
  neighbor 192.168.0.2 activate
 exit-address-family
exit
!
router bgp 65001 vrf NMS
 bgp router-id 10.10.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 !
 address-family ipv4 unicast
  redistribute connected route-map RM_SET_COMMUNITIES
  label vpn export auto
  rd vpn export 65001:10
  rt vpn both 65001:10
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute static route-map RM_SET_COMMUNITIES
  label vpn export auto
  rd vpn export 65001:10
  rt vpn both 65001:10
  export vpn
  import vpn
 exit-address-family
exit
!
!
router ospf
 ospf router-id 192.168.0.1
 passive-interface default
 network 10.14.0.0/24 area 0
 network 192.168.0.1/32 area 0
 network 192.168.1.1/32 area 0
exit
!
mpls ldp
 router-id 192.168.0.1
 dual-stack cisco-interop
 !
 address-family ipv4
  discovery targeted-hello accept
  discovery transport-address 192.168.0.1
  label local allocate host-routes
  !
  interface ens23.40
  exit
  !
  interface gre1
  exit
  !
 exit-address-family
 !
exit

IPSec profile config:

~$ sudo nano /etc/swanctl/swanctl.conf
------------------------------------------------------------------------
connections {
    dmvpn {
        proposals = aes256-sha256-modp2048
        version = 2
        rekey_time = 28800s
        keyingtries = 0
        local {
            auth = psk
            id = @sauna-vm1.bastuklubben.online
        }
        remote {
            auth = psk
            id = %any
        }
        children {
            dmvpn {
                esp_proposals = aes256-sha256-modp2048
                rekey_time = 3600s
                rand_time = 540s
                local_ts = dynamic[gre]
                remote_ts = dynamic[gre]
                mode = transport
                dpd_action = clear
                close_action = none
                start_action = none
            }
        }
    }

}
secrets {
    ike-1 {
        secret = my-pre-shared-secret
    }
}

Cisco IOS-XE

interface Loopback0
 description RouterID
 ip address 192.168.0.3 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet0/0/1.3014
 description FW-GLOBAL
 encapsulation dot1Q 3014
 ip address 10.14.255.1 255.255.255.254
 ipv6 address FE80::E:2 link-local
 ipv6 address 2001:DB8:1234:E::2/64
 ipv6 nd ra suppress all
!
interface Tunnel1
 description DMVPN Tunnel Interface
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication secret
 ip nhrp network-id 1
 ip nhrp holdtime 60
 ip nhrp nhs 192.168.1.1 nbma 10.4.254.41 multicast
 ip nhrp nhs 192.168.1.2 nbma 10.4.254.42 multicast
 ip nhrp redirect
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 ip ospf dead-interval 40
 ip ospf hello-interval 10
 ip ospf 1 area 0
 ipv6 mtu 1400
 ipv6 tcp adjust-mss 1340
 mpls ip
 tunnel source GigabitEthernet0/0/1.3014
 tunnel mode gre multipoint
 tunnel key 1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile DMVPN
!
! IPSec Configuration
!
crypto ikev2 proposal IKEV2-PROPOSAL 
 encryption aes-cbc-256
 integrity sha256
 group 14
!
crypto ikev2 policy IKEV2-POLICY 
 match fvrf any
 proposal IKEV2-PROPOSAL
!
crypto ikev2 keyring DMVPN-KEYRING
 peer DMVPN-PEER
  address 0.0.0.0 0.0.0.0
  pre-shared-key my-pre-shared-secret
 !
crypto ikev2 profile DMVPN-IKE
 match fvrf any
 match identity remote fqdn sauna-vm1.bastuklubben.online
 match identity remote fqdn sauna-vm2.bastuklubben.online
 identity local fqdn sauna-ro1.bastuklubben.online
 authentication remote pre-share
 authentication local pre-share
 keyring local DMVPN-KEYRING
 dpd 30 5 on-demand
!
crypto ipsec transform-set DMVPN-ESP esp-aes 256 esp-sha256-hmac 
 mode transport
!
crypto ipsec profile DMVPN
 set transform-set DMVPN-ESP 
 set ikev2-profile DMVPN-IKE
!
!
mpls ldp neighbor 192.168.0.1 targeted
mpls ldp neighbor 192.168.0.2 targeted
mpls ldp router-id Loopback0
!
router ospf 1
 router-id 192.168.0.3
 passive-interface default
 no passive-interface Tunnel1
!
!
router bgp 65001
 bgp router-id 192.168.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.1.1 remote-as 65001
 neighbor 192.168.1.1 transport connection-mode active
 neighbor 192.168.1.1 update-source Tunnel1
 neighbor 192.168.1.2 remote-as 65001
 neighbor 192.168.1.2 transport connection-mode active
 neighbor 192.168.1.2 update-source Tunnel1
 !
 address-family ipv4
  redistribute ospf 1 route-map RM_SET_COMMUNITIES
 exit-address-family
 !
 address-family vpnv4
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 send-community both
  neighbor 192.168.1.2 activate
  neighbor 192.168.1.2 send-community both
 exit-address-family
 !
 address-family ipv6
  redistribute ospf 1 route-map RM_SET_COMMUNITIES
 exit-address-family
 !
 address-family vpnv6
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 send-community both
  neighbor 192.168.1.2 activate
  neighbor 192.168.1.2 send-community both
 exit-address-family
 !
 address-family ipv4 vrf NMS
  aggregate-address 10.10.0.0 255.255.0.0 summary-only attribute-map RM_SET_COMMUNITIES
  redistribute ospfv3 1 route-map RM_SET_COMMUNITIES
 exit-address-family
 !        
 address-family ipv6 vrf NMS
  redistribute ospf 1 route-map RM_SET_COMMUNITIES
  aggregate-address FC01::/16 summary-only attribute-map RM_SET_COMMUNITIES
  aggregate-address 2001:DB8:1234:A000::/52 summary-only attribute-map RM_SET_COMMUNITIES
 exit-address-family
 !
!
route-map RM_SET_COMMUNITIES permit 10 
 set community 65001:2
!

Open Source Is Fun

Discussion about this post

Open Source Is Fun

Linux: Configure MPLSoDMVPN Part 3: Enable routing over the DMVPN tunnel

How-To: Linux MPLSoDMVPN Part 3

The story so far

Underlay Routing Configuration

Design

OSPF Configuration

Linux FRR Hub router OSPF Configuration

Cisco Spoke Router OSPF Configuration

Verification

Linux configuration changes

IPv6 and IPv4 forwarding

MPLS forwarding

Load MPLS modules

Change MPLS Settings

Caveat Discovered

Add --v6-with-v4-nexthops to Zebra config

Overlay Routing Configuration

Design

MPLS Configuration

FRR Hub Router MPLS Configuration

Cisco IOS-XE Spoke Router MPLS Configuration

MPLS Verification

Linux FRR Hub Router

BGP Configuration

FRR Hub Router BGP related Configuration

Cisco IOS-XE Spoke Router BGP related Configuration

Overlay routing Verification

Linux FRR Hub Router Overlay Routing Verification

Cisco IOS-XE Spoke Router Overlay Routing Verification

End-to-End Verification

Appendix

Troubleshooting and Caveats

Prefixes becomes invalid after one hour

FRR wont’ install IPv6 BGP recursive routes into the routing table.

Full Routing Configuration

Linux FRR Hub Router

Cisco IOS-XE

Discussion about this post

Add `--v6-with-v4-nexthops` to Zebra config