Network Redesign Part 2: Configuration Changes
The devil is in the details
Monday 30th June 2025
In Part 1 I gave an overview of what I wanted to accomplish. In this part I will go through some routing techniques I used to rebuild my network infrastructure.
Detailed Design
Notice that following configuration won’t be covered since they are already described in various posts already:
Interface configuration
VLAN configuration
Routing configuration
VPN configuration
Actually the only thing I will cover in detail is how to perform route filtering to prevent routing loops.
Underlay Routing
Explanation:
I’m using the same type of overlay/underlay topology as describe in my Configure MPLSoDMVPN on Linux series:
The routes from the firewall is only available to SAUNA-RO1 because I haven’t configured redistribution or default information originate under the OSPF process.
Why I am using IPv4 in the Underlay
Things would have been much simpler if I just ran IPv6 over the GRE tunnel. However, I have tried configure NHRPv6 and it just doesn’t work. You can see my ticket on Github about it. Therefore as a workaround I’m using MPLS to transport IPv6 packets over the GRE ipv4-only tunnel.
However, running MPLS over DMVPN is actually pretty neat. It scales well and it’s easy to apply route filtering. So even if NHRPv6 would have worked, I would probably stick to this design.
Ofcourse, being able to at least run NHRPv4 over a GREv6 tunnel would have been a major benefit.
IPSec Design
All underlay routing is configured in the Global routing table. However, I’m using VRF NMS on the Bastuklubben side as a front-door VRF. Technically, I don’t even need that route down to the pfSense firewall in the global routing table anymore and I’m considering if I should remove it (see part 1 layer 3 design).
Why I am using IPv4 to establish VPN tunnels
The same reason as stated above; NHRP only works on a GREv4 underlay.
Overlay Design
Explanation:
This is the same MPLSVPN configuration as described in Linux: Configure MPLSoDMVPN Part 3 but with a twist: On the Linux Servers I’m importing and exporting VPN routes using the global routing table. I’m not even sure that is possible on my Cisco router!
Configuration
Linux LAN Interface configuration
This is how I setup the LAN interface on the linux servers:
#SAUNA-VM1:
#NMS VLAN
auto ens20
iface ens20 inet static
address 10.10.0.1/24
iface ens20 inet6 static
address 2001:DB8:1234:9001::1/64
iface ens20 inet6 static
address 2001:DB8:1234:A001::1/64
iface ens20 inet6 static
address 2001:DB8:1234:B001::1/64
iface ens20 inet6 static
address FE80::A:1/64Note: The configuration is for debian servers. Those prefixes are also advertised in BGP, something that is covered down below.
Route Filtering Configuration
To prevent routing loops, some filtering has to be applied. For example if the default route from the firewall get leaked over the VPN tunnel, The tunnel will start flapping.
The core concepts are:
Routes learned from the Firewall should not be advertised to other BGP neighbors, except 2001:464F:6F83:8000::/49 should be advertised from Bastuklubben main site to cloud datacenter.
Routes learned internally on each site should be shared with each other, but not shared with the firewalls.
Host routes for SAUNA-VM1 and SAUNA-VM2 VLAN 10 addresses are necessary for optimal routing. Those host routes needs to be filtered so that they don’t become advertised redundantly.
Route-map Configuration
I solved this by not referencing a single prefix (except on the firewall). I prefer to do it that way since the same rules apply for both IPv4 and IPv6, and I don’t have to reconfigure the route-maps when there is a change in the address scheme.
SAUNA-FW01 Route-Maps
This is actually covered in IPv6 Multi-Homing Part 1. But I may have made some slight changes since then so I document the present FRR configuration here:
# Relevant BGP Configuration:
router bgp 65000
!
address-family ipv4
aggregate-address 10.0.0.0/13
aggregate-address 10.8.0.0/13
network 0.0.0.0/0
network 10.0.0.0/8
network 10.0.0.0/16
neighbor 10.0.10.3 route-map BGP_Permit_AS65001 out
neighbor 2001:db8:1234:1::3 route-map BGP_Permit_AS65002 out
exit-address-family
!
address-family ipv6
aggregate-address 2001:DB8:1234::/49
aggregate-address 2001:DB8:1234:8000::/49
network ::/0
network 2001:DB8:1234::/48
network 2001:DB8:1234::/52
neighbor 10.0.10.3 route-map BGP_Permit_AS65001 out
neighbor 2001:db8:1234:1::3 route-map BGP_Permit_AS65002 out
exit-address-family
# Route Maps:
route-map BGP_Permit_AS65001 permit 10
description Permit route advertisements for Bastuklubben
match ipv6 address prefix-list IPv6_SAUNA_AS65001
set community 65000:1
!
route-map BGP_Permit_AS65001 permit 20
description Permit IPv4 scopes for Bastuklubben
match ip address prefix-list IPv4_SAUNA_AS65001
set community 65000:1
!
route-map BGP_Permit_AS65001 permit 30
description IPv6 Permit Default Route
match ipv6 address prefix-list IPv6_Default
set community no-advertise
!
route-map BGP_Permit_AS65001 permit 40
description IPv4 Permit Default Route
match ip address prefix-list IPv4_Default
set community no-advertise
!
route-map BGP_Permit_AS65002 permit 10
description Permit Route advertisements for Libertas Solutions
match ipv6 address prefix-list IPv6_LBS_AS65002
set community 65000:1
!
route-map BGP_Permit_AS65002 permit 20
description Permit IPv4 Scopes for Libertas Solutions
match ip address prefix-list IPv4_LBS_AS65002
set community 65000:1
!
route-map BGP_Permit_AS65002 permit 30
description IPv6 Permit Default Route
match ipv6 address prefix-list IPv6_Default
set community 65000:1
!
route-map BGP_Permit_AS65002 permit 40
description IPv4 Permit Default Route
match ip address prefix-list IPv4_Default
set community 65000:1
# Related Prefix Lists:
ip prefix-list IPv4_Default seq 5 permit 0.0.0.0/0
ip prefix-list IPv4_Default seq 10 permit 10.0.0.0/16
!
ip prefix-list IPv4_LBS_AS65002 seq 5 permit 10.0.0.0/13
ip prefix-list IPv4_LBS_AS65002 description IPv4 Scope for Libertas Solutions
!
ip prefix-list IPv4_SAUNA_AS65001 seq 5 permit 10.8.0.0/13
ip prefix-list IPv4_SAUNA_AS65001 description IPv4 Scope for Bastuklubben
!
ipv6 prefix-list IPv6_Default seq 5 permit ::/0
ipv6 prefix-list IPv6_Default seq 10 permit 2001:DB8:1234::/52
!
ipv6 prefix-list IPv6_LBS_AS65002 seq 5 permit 2001:DB8:1234::/49
ipv6 prefix-list IPv6_LBS_AS65002 description IPv6 Scope for Libertas Solutions
!
ipv6 prefix-list IPv6_SAUNA_AS65001 seq 5 permit 2001:DB8:1234:8000::/49
ipv6 prefix-list IPv6_SAUNA_AS65001 description IPv6 Scope for BastuklubbenExplanation:
Two Route-Maps are defined, 4 rules each. If Bastuklubben and Libertas Solutions actually was located on different locations, there would only be one Route-Map per firewall.
The Rules are quite simple. For Bastuklubben:
For IPv6 and IPv4 default route and routes belonging to the firewall, i.e 10.0.0.0/16 and 2001:464F:6F83::/52 (See part 1 for reference), set community no-advertise, which means that route advertisement will stop at SAUNA-RO1.
For Bastuklubbens IPv4 and IPv6 scopes, i.e 10.8.0.0/13 and 2001:464F:6F83:8000::/49, set community 65000:1. That community identifies that the routes originates from the firewall
For Libertas Solutions
Same concept as for Bastuklubben, but with a few differences. Since the SAUNA-VMs are not the next-hops from the SAUNA-FW01, I can’t use the “no-advertise” community, as it then would stop at LBS-RO1. That’s why I’m only using community “65000:1”.
On the next-hop LBS-RO1, I’m converting that community to “no-advertise”
Note: For more information on how to configure routing on pfsense, I have these related posts:
LBS-RO1 Route-Maps
# Relevant BGP Configuration:
set vrf name NMS protocols bgp peer-group PG-NMS address-family ipv4-unicast route-map export 'RM_TENANTS_OUT'
set vrf name NMS protocols bgp peer-group PG-NMS address-family ipv6-unicast route-map export 'RM_TENANTS_OUT'
# Route-Map Configuration:
set policy route-map RM_TENANTS_OUT description 'Matches routes from firewall that should be filtered on next-hop'
set policy route-map RM_TENANTS_OUT rule 10 action 'permit'
... rule 10 match community community-list 'CL-SAUNA-FW'
... rule 10 set community replace 'no-advertise'
# Relevant Community-List:
set policy community-list CL-SAUNA-FW rule 10 action 'permit'
set policy community-list CL-SAUNA-FW rule 10 regex '65000:1'As mention earlier, I’m converting community “65000:1” into “no-advertise” when sending the prefixes downstream to tenants.
SAUNA-RO1 Route-Maps
# Related VRF Configuration:
vrf definition NMS
rd 65001:10
!
address-family ipv4
route-target export 65001:10
route-target import 65001:10
exit-address-family
!
address-family ipv6
route-target export 65001:10
route-target import 65001:10
exit-address-family
# Related BGP Configuration
router bgp 65001
!
address-family ipv4 vrf NMS
aggregate-address 10.10.0.0 255.255.0.0 summary-only attribute-map RM_SET_COMMUNITIES
redistribute ospfv3 1
neighbor 10.0.10.1 route-map NMSv4 out
exit-address-family
!
address-family ipv6 vrf NMS
aggregate-address 2001:DB8:1234:A000::/52 summary-only attribute-map RM_SET_COMMUNITIES
redistribute ospf 1
neighbor 10.0.10.1 route-map NMSv6 out
exit-address-family
!
address-family vpnv4
neighbor 192.168.1.1 route-map RM_DMVPNv4_OUT out
neighbor 192.168.1.2 route-map RM_DMVPNv4_OUT out
exit-address-family
!
address-family vpnv6
neighbor 192.168.1.1 route-map RM_DMVPNv6_OUT out
neighbor 192.168.1.2 route-map RM_DMVPNv6_OUT out
exit-address-family
# Route-Maps:
route-map RM_SET_COMMUNITIES permit 10
set community 65001:2
!
!
route-map RM_DMVPNv4_OUT permit 10
match community CL-SAUNA-FW
set as-path prepend 65001
set community 65001:2
!
route-map RM_DMVPNv4_OUT permit 20
match community CL-SAUNA-DC2
set as-path prepend 65001
!
!
route-map RM_DMVPNv6_OUT permit 10
match community CL-SAUNA-FW
set as-path prepend 65001
set community 65001:2
set ipv6 next-hop ::FFFF:192.168.1.3
!
route-map RM_DMVPNv6_OUT permit 20
match community CL-SAUNA-DC2
set as-path prepend 65001
set ipv6 next-hop ::FFFF:192.168.1.3
!
!
route-map NMSv4 permit 10
match community CL-SAUNA-DC2
!
!
route-map NMSv6 permit 100
match community CL-SAUNA-DC2
set ipv6 next-hop 2001:DB8:1234:A::2
# Relevant Community Lists:
ip community-list standard CL-SAUNA-FW permit 65000:1
ip community-list standard CL-SAUNA-DC1 permit 65001:1
ip community-list standard CL-SAUNA-DC2 permit 65001:2Explanation:
I’ll start addressing that the next-hop configuration under the route-maps are not relevant for route filtering but are crucial for routing to work. You can read about it in previously linked posts.
All community lists are not used but I have them defined in case I need to reference one of them later.
Route-Map
RM_SET_COMMUNITIESsets a community on all routes that I want to advertise to DC1 (this is DC2).Route-MAP RM_DMVPNvX_OUT filter only routes with the correct communities. The prefix 2001:DB8:1234:8000::/49 from the firewall also gets matched, and converted to 65001:2, as to appear it originates from DC2.
Route-Map NMSvX also filter that only routes that originates from DC2 will be advertised.
SAUNA-VM1 and VM2 Route-Maps
I’m Only showing SAUNA-VM1 configuration as it is very similar to VM2.
# Related Static Routes:
ipv6 route 2001:db8:1234:9000::/60 blackhole tag 65001 254
ipv6 route 2001:db8:1234:a000::/60 blackhole tag 65001 254
ipv6 route 2001:db8:1234:b000::/60 blackhole tag 65001 254
# Related BGP Configuration:
router bgp 65001
neighbor PG-DMVPN peer-group
neighbor PG-DMVPN remote-as 65001
neighbor PG-DMVPN passive
neighbor PG-DMVPN update-source gre1
neighbor PG-DMVPN timers 15 45
!
address-family ipv4
network 10.10.0.0/24 route-map RM_SET_COMMUNITIES
network 10.10.0.1/32 route-map RM_SET_COMMUNITIES
neighbor 192.168.0.1 route-map RM_FILTER_COMMUNITIES out
neighbor 2001:db8:1234:1000::1 route-map RM_LBS-RO1_OUT out
label vpn export auto
rd vpn export 65001:10
rt vpn both 65001:10
export vpn
import vpn
exit-address-family
!
address-family ipv4 vpn
neighbor PG-DMVPN activate
neighbor PG-DMVPN route-reflector-client
neighbor PG-DMVPN route-map RM_DMVPNv4_OUT out
exit-address-family
!
address-family ipv4 labeled-unicast
neighbor PG-DMVPN activate
exit-address-family
!
address-family ipv6 unicast
network 2001:db8:1234:a001::1/128 route-map RM_SET_COMMUNITIES
redistribute static route-map RM_SET_COMMUNITIES
neighbor 192.168.0.2 route-map RM_FILTER_COMMUNITIES out
neighbor 2001:db8:1234:1000::1 route-map RM_LBS-RO1_OUT out
label vpn export auto
rd vpn export 65001:10
rt vpn both 65001:10
export vpn
import vpn
exit-address-family
!
address-family ipv6 vpn
neighbor PG-DMVPN activate
neighbor PG-DMVPN route-reflector-client
neighbor PG-DMVPN soft-reconfiguration inbound
neighbor PG-DMVPN route-map RM_DMVPNv6_OUT out
exit-address-family
exit
# Route-Maps:
route-map RM_SET_COMMUNITIES permit 10
set community 65001:1
!
!
route-map RM_LBS-RO1_OUT permit 10
match community CL_SAUNA_DC1
set community 65001:2101 no-advertise
!
!
route-map RM_DMVPNv6_OUT permit 10
match community CL_SAUNA_DC1
set as-path prepend 65001
set community no-export additive
set ipv6 next-hop global ::ffff:192.168.1.1
!
!
route-map RM_DMVPNv4_OUT permit 10
match community CL_SAUNA_DC1
set as-path prepend 65001
set community no-export additive
set ip next-hop 192.168.1.1
!
!
route-map RM_FILTER_COMMUNITIES deny 10
match community CL_SAUNA_DC1
!
route-map RM_FILTER_COMMUNITIES deny 20
match community CL_SAUNA_DC2
!
route-map RM_FILTER_COMMUNITIES permit 100
Related Community Lists:
bgp community-list standard CL_LBS_FW1 seq 5 permit 65000:1
bgp community-list standard CL_SAUNA_DC1 seq 5 permit 65001:1
bgp community-list standard CL_SAUNA_DC2 seq 5 permit 65001:2Explanation:
Included is route target configuration to demonstrate that you can actually use the global routing table to import routes from SAUNA-RO1 VRF NMS.
Prefixes going to LBS-RO1 gets assigned community “no-advertise” to prevent the routes from being advertised to other tenants. It also sends a community that identifies the VM ID, just for metadata purposes.
BGP prefixes between SAUNA-VM1 and 2 gets filtered so they don’t end up causing asymmetrical routing. I’m honestly questioning the necessity of even having BGP neighborship between them.1
Host routes are implemented so that traffic from SAUNA-RO1 takes the most optimal path to reach the servers.
Prefixes from SAUNA-RO1 are prevented from reaching the uplink router LBS-RO1. Since it could be necessary to advertise routes from SAUNA-RO1 to other sites in the future, I intentionally left out the “no-advertise” community.
Prefixes advertised to SAUNA-RO1 will be sent with the communities “65001:1” and “no-export”. No-export means that the routes won’t be advertised to any eBGP peers. Actually they won’t be advertised anyway but I included it just for extra protection.
Verification
I realize that for every new site I would have to add another community, but generic branch offices doesn’t need to have a unique community. In the future I might add “65001:3” that matches all other sites and that is final.