IPv6 Multi-homing Part 1: Configure routing and filtering

How-To: pfSense/Cisco/VyOS IPv6 Multi-homing Part 1

May 05, 2025

Monday 5th May 2025

The goal is to setup an IPv6 only remote datacenter with local internet breakout. One of the datacenters handles multiple tenants.

High-Level Design

Explanation:

The client browses nginx.bastuklubben.online and gets forwarded to one of the sites, depending on the DNS round-robin algorithm.
The pfSense appliance at the site forwards the TCP packet to one of their local docker or podman nodes, depending on the configured load-balancing settings.
DC2 is the main prefix and does not need prefix translation
DC1 needs prefix translation for local internet breakout
The Kubernetes cluster (or whatever I end up with) is interconnected through a GRE VPN tunnel, but could be a DCI or MPLSVPN as well.
NPTv6 (or IPv6 NPT) is a protocol used for IPv6-to-IPv6 prefix translation. In this case, 3FFF:ABC::/48 needs to be translated to 3FFF:123::/48

Note: ‘3FFF::/20’ is a new IPv6 documentation prefix.

Low-Level Design

Bastuklubben Hybrid-cloud design

Bastuklubben will be used as the example for this setup. It’s a hybrid “cloud” solution because;

DC2 it’s totally autonomous;
DC1 shares a space with other tenants, which is a cloud concept (even though it’s all running from my basement).

Intra VRF routing

This is how intra VRF traffic (inside one VRF) between the 2 datacenters operates:

Explanation:

Note that the prefixes used here doesn’t match the high-level design but the concept is the same.
DC1 is the site where NPTv6 is necessary.
Traffic between sites use a VPN tunnel between each other
Traffic towards the internet will choose the closest firewall
On DC1, NPTv6 is configured for traffic coming from, or going to the Internet.

Inter VRF routing

This is how inter VRF traffic between the 2 datacenters operates:

Explanation:

All inter VRF traffic needs to pass through the firewall at DC2.
There is no backup route for inter VRF traffic at the moment. If DC2 becomes unavailable, only a default route to the Internet will be available in DC1. Although I can think of a way on how to fix that, I’ll save it for another post.

Configuration

Pre-Requisites

The related posts that have led up to this scenario are:

BGP routing between Cisco IOS-XE and pfSense
Basically all of the VyOS related posts I have written so far
Advanced Network configuration on Ubuntu 24.04

All of these posts can be found under the Documentation page

Configuration pfSense

In reality, there is only one pfSense firewall but the prefixes are divided on different subinterfaces. The configuration will be almost identical even if the sites would be geographically separated.

The inter VRF routing with BGP was already described in VyOS Network Configuration but the configuration needs some refinement.

Route filtering and manipulation

The default routes for both sites should not be advertised to other iBGP neighbors. They won’t be usable as backup anyway because they are only reachable over the VPN tunnel.

For DC1, the 3FFF:123:abc::/49 should also not be allowed to be advertised to other iBGP neighbors.

To achieve this, I have created route-maps that matches the mentioned prefixes and attaches the well-known BGP community “no-advertise” to them.

Prefix list configuration in pfSense

Under Services > FRR Global/Zebra > Prefix Lists are three prefix lists defined (for IPv6):

Route-Map Configuration

In the tab Route Maps, two Route Maps are defined:

Note: IPv4 configuration won’t be covered.

Apply Route-Map to BGP neighbors

Under Services > FRR BGP > Neighbors, the route-maps have been attached in the outgoing direction for neighbors that are route-leaking between each other.

Example from SAUNA_PUBLIC:

Example from LBS_PUBLIC:

VyOS Configuration

At the moment, routes from all tenants are getting advertised through the VPN tunnel configured in earlier posts:

That is not ideal. I need to somehow filter prefixes that belongs to a specific tenant. Again, BGP communities are my solution for everything. From DC1:

LBS prefixes from downstream routers gets marked with 65001:3
SAUNA prefixes from downstream routers gets marked with 65001:4

Then the router at DC2 can filter incoming routes with it’s corresponding community.

Note: I tried to think of a solution to apply filtering in the outbound direction but I haven’t figured out a simple solution for that yet.

VyOS DC1 configuration

Utilizing route tags

In the post Advanced Linux Network Configuration I attached tags on incoming OSPFv3 routes. I can match on those tags to identify which tenant those routes belong to.

Route map

I only need one route-map to map tags to communities:

set policy route-map RM_SET_COMMUNITIES description 'map route tag to community tag'

set policy route-map RM_SET_COMMUNITIES rule 10 action 'permit'
set policy route-map RM_SET_COMMUNITIES rule 10 match tag '3'
set policy route-map RM_SET_COMMUNITIES rule 10 set community add '65001:3'
set policy route-map RM_SET_COMMUNITIES rule 10 set community add no-export

set policy route-map RM_SET_COMMUNITIES rule 20 action 'permit'
set policy route-map RM_SET_COMMUNITIES rule 20 match tag '4'
set policy route-map RM_SET_COMMUNITIES rule 20 set community add '65001:4'
set policy route-map RM_SET_COMMUNITIES rule 20 set community add no-export

Note: the well-known community of no-export makes sure that the routes learnt will not be advertised to eBGP peers (i.e the pfSense firewall).

Then attach the route-map to redistribute commands under the BGP processes for each VRF:

set vrf name INT protocols bgp address-family ipv6-unicast redistribute ospfv3 route-map 'RM_SET_COMMUNITIES'
set vrf name NMS protocols bgp address-family ipv6-unicast redistribute ospfv3 route-map 'RM_SET_COMMUNITIES'
set vrf name PUB protocols bgp address-family ipv6-unicast redistribute ospfv3 route-map 'RM_SET_COMMUNITIES'

Cisco IOS-XE DC2 configuration

Community-list

On the Cisco router I created a community-list that matches on “65001:4”:

ip community-list standard CL_SAUNA permit 65001:4

Route-map

Then I created a route-map that matches the community list…

route-map RM_DMVPN_IN permit 10 
 match community CL_SAUNA

… and it gets applied on incoming MPLSVPN routes:

router bgp 65001
 address-family vpnv6
  neighbor 192.168.0.1 route-map RM_DMVPN_IN in

Verification

VyOS router DC1

vyos@LBS-RO1:~$ show ipv6 route vrf PUB bgp
...
VRF PUB:
B>* ::/0 [20/0] via fe80::f2ad:4eff:fe28:b1ff, eth1.3003
B>* 3fff:abc:123::/49 [20/0] via fe80::f2ad:4eff:fe28:b1ff, eth1.3003
B>* 3fff:abc:123:9::/64 [200/0] via ::ffff:c0a8:203, tun3
B>* 3fff:abc:123:3000::/52 [200/0] unreachable (blackhole)
B>* 3fff:abc:123:8000::/49 [200/0] via ::ffff:c0a8:203, tun3
B>* 3fff:abc:123:9000::/52 [200/0] via ::ffff:c0a8:203, tun3

Explanation:

This is the router on DC1. It sees routes from all tenants.
The default route is the local firewall and not one learnt via the VPN tunnel.
“3FFF:123:abc:8000::/49” is learnt via the VPN tunnel and not from the local firewall

The “no-advertise” community is set for “::/0” and “3FFF:123:abc:8000::/49”

vyos@LBS-RO1:~$ show bgp vrf PUB ipv6 3fff:123:abc::/49
BGP routing table entry for 3fff:123:abc::/49, version 476
Paths: (1 available, best #1, vrf PUB, not advertised to any peer)
  Not advertised to any peer
  65002 65000, (aggregated by 65000 10.10.1.6)
    3fff:123:abc:3::1 from 3fff:123:abc:3::1 (10.10.1.6)
    (fe80::f2ad:4eff:fe28:b1ff) (used)
      Origin incomplete, metric 0, valid, external, atomic-aggregate, best (First path received)
      Community: no-advertise
      Last update: Mon Mar 17 19:24:15 2025

Cisco IOS-XE router DC2

On DC2, there is no route learnt for “3fff:123:abc::/49”. Only local default route, “3fff:123:abc:8000::/49” and routes learnt from itself or via the VPN tunnel are present:

SAUNA-RO1#show ipv6 route vrf PUBLIC bgp
...
B   ::/0 [20/0], tag 65000
     via FE80::F2AD:4EFF:FE28:B1FF, GigabitEthernet0/0/1.3009
B   3FFF:123:ABC:8000::/49 [20/0], tag 65000
     via FE80::F2AD:4EFF:FE28:B1FF, GigabitEthernet0/0/1.3009
B   3FFF:123:ABC:9000::/52 [200/0]
     via Null0%default, directly connected
B   3FFF:123:ABC:9000::/60 [200/20]
     via 192.168.2.1%default, indirectly connected

Now is the routing on the edge network almost in order. However, there are a few unresolved tasks left:

SAUNA nodes in DC1 cannot reach Internet because 3FFF:123:abc:8000: is outside DC1’s prefix range. In the next part I will cover the NPTv6 configuration.
An issue with asymmetric routing between the SAUNA and LBS MGMT network have appeared after route filtering was applied. That can be solved with a common OOBM = Out-of-Bound MGMT network. That setup will be covered in future posts.

Open Source Is Fun

Discussion about this post