VyOS: Firewall bridge Configuration

How-To: VyOS Firewall

Thursday 5th June 2025

Disclaimer: I’m not using this solution anymore but I decided to documented it anyway as it includes useful information about VyOS bridge configuration. Next post will be the actual solution I went for.

One of the most important things when building a multitenant datacenter is to prevent tenants from speaking directly to each other.

Imagine living in an apartment complex without locks on the front doors. You can of course install your own lock on your own door but it doesn’t stop you from entering other tenants.

Design

Explanation:

• There are three tenants in the drawing. However, the configuration will only focus on SAUNA and LBS.

• TE3 is just a generic name for “Tenant 3”

• All three tenants are residing inside the same PVE cluster. In laymen terms: The same apartment complex.

• The tenants have specific firewall groups assigned. Only members of the same group are allowed to communicate directly with each other.

Overview

Let’s focus on only SAUNA and LBS inside one example VRF:

Route filtering and NPTv6 is already configured. The tenants only see their own prefixes plus the uplink LAN that they all share:

sauna-vm1:~$ sudo vtysh -c "show ipv6 route vrf PUB"
...
IPv6 unicast VRF PUB:
B>* ::/0 [200/0] via fe80::3:1, ens21, weight 1, 4d20h37m
C>* 2001:db8:1234:3000::/64 is directly connected, ens21, weight 1...
L>* 2001:db8:1234:3000::4:1/128 is directly connected, ens21, weight...
B>* 2001:db8:1234:8000::/49 [200/0] via ::ffff:192.168.1.3, gre1...
B>* 2001:db8:1234:9000::/52 [200/0] via ::ffff:192.168.1.3, gre1...
S>* 2001:db8:1234:9000::/60 [254/0] unreachable (blackhole), weight...
C>* 2001:db8:1234:9001::/64 is directly connected, ens23.30, weight...
L>* 2001:db8:1234:9001::1/128 is directly connected, ens23.30, weight..
C * fe80::/64 is directly connected, ens23.30, weight 1, 4d20h37m
C>* fe80::/64 is directly connected, ens21, weight 1, 4d20h37m

lbs-vm1:~$ sudo vtysh -c "show ipv6 route vrf PUB"
...
IPv6 unicast VRF PUB:
B>* ::/0 [200/0] via fe80::3:1, ens21, weight 1, 00:00:28
S>* 2001:db8:1234:3000::/60 [254/0] unreachable (blackhole), weight...
C>* 2001:db8:1234:3000::/64 is directly connected, ens21, weight 1...
L>* 2001:db8:1234:3000::1:1/128 is directly connected, ens21, weight...
C>* 2001:db8:1234:3001::/64 is directly connected, ens23.30, weight...
L>* 2001:db8:1234:3001::1/128 is directly connected, ens23.30, weight..
C * fe80::/64 is directly connected, ens23.30, weight 1, 00:00:34
C>* fe80::/64 is directly connected, ens21, weight 1, 00:00:34

Note: some output omitted for easier reading

But clients inside vrf PUB can still communicate:

sauna@sauna-vm1:~$ sudo ip vrf exec PUB ping -I 2001:DB8:1234:9001::1 2001:DB8:1234:3001::1
PING 2001:DB8:1234:3001::1 (2001:db8:1234:3001::1) from 2001:464f:6f83:9001::1 : 56 data bytes
64 bytes from 2001:db8:1234:3001::1: icmp_seq=1 ttl=62 time=0.894 ms
64 bytes from 2001:db8:1234:3001::1: icmp_seq=2 ttl=62 time=1.16 ms
64 bytes from 2001:db8:1234:3001::1: icmp_seq=3 ttl=62 time=1.09 ms
^C
--- 2001:DB8:1234:3001::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2015ms
rtt min/avg/max/mdev = 0.894/1.046/1.161/0.112 ms

With a traceroute, you can see that it SAUNA reaches LBS via the default gateway:

sauna@sauna-vm1:~$ sudo ip vrf exec PUB traceroute -s 2001:DB8:1234:9001::1 2001:DB8:1234:3001::1
traceroute to 2001:db8:1234:3001::1 (2001:db8:1234:3001::1), 30 hops max, 80 byte packets
 1  2001:db8:1234:3000::1 (2001:db8:1234:3000::1)  0.433 ms  0.394 ms  0.645 ms
 2  2001:db8:1234:3001::1 (2001:db8:1234:3001::1)  0.853 ms  0.841 ms  0.828 ms

It feels like there should be a feature that prevents the router from forwarding traffic on the same interface that it received it on. That would have been an easy fix. Alas, I haven’t found any feature that does that. Until a better solution becomes available, I have to fix this with firewall rules.

Planning

The first thing to plan for is how to configure the network and the firewall with as few rules as possible. The criterias are:

• Default route has to be permitted

• It should be possible to automate

• I can’t filter by using source and destination prefixes, as it would need to be updated everytime a new tenant is added.

• I can’t use seperate VLANs or VRFs per tenant because it doesn’t scale and there will be a lot of configuration. The plan is to use the shared segments.

The solution

The solution (for now) is to use seperate interfaces per tenant, but they are members of the same bridge domain:

Explanation:

• Instead of a shared segment for each tenant, they are separated by physical interfaces. The gateway IP however, is the same.

• This introduces the ability to do port isolation, meaning that local VLAN traffic between tenants are not allowed.

• Firewalling can be done on the bridge level. It will be explained when configuration starts.

• What is not so good with this solution is that I have to create a new bridge for every tenant and add it to the VyOS VM.

Configuration

Configure new Linux bridges

In Proxmox, I need to configure two linux bridges. One way i found out to do it is to create security zones and create separate VNets for each zone:

Step 1: Navigate to Datacenter > SDN > Zones. Click Add > Simple Zone

At the moment, only ID and MTU needs to be specified:

When finished, navigate to SDN and click Apply.

Step 2: Under SDN, navigate to VNets. Click Create

Explanation:

• Give it a Name and assign it to the corresponding zone.

• Port Isolation is not necessary here because it’s going to be enabled on VyOS instead.

• VLAN Aware makes it possible to configure subinterfaces on the guest OS.

Step 3: When finished, navigate to SDN and click Apply. When configuration of the VNets are completed, new linux bridges will be available to the hosts. Add these to the guests:

On the VyOS router:

When making changes to a VyOS instance, the MAC addresses will change. VyOS will interpret this as new interfaces. Therefore you should note down the existing MAC addresses. If you lost them, you can edit the startup configuration to include the new MAC addresses instead:

vyos@LBS-RO1:~$ /bin/sh
sh-5.2$ nano /opt/vyatta/etc/config/config.boot
------------------------------------------------------------------------
...
    }
    ethernet eth1 {
        hw-id "bc:24:11:c1:02:9b"
        mtu "9198"
    }
    ethernet eth2 {
        hw-id "bc:24:11:aa:7e:02"
        mtu "9198"
    }
...

Then reboot.

Source: https://forum.vyos.io/t/ethernet-interface-renumbering/4042

Debian Bridge interface configuration

Here is a sample configuration of the bridge interface and how to create VLAN subinterfaces:

~$ sudo nano /etc/network/interfaces
------------------------------------------------------------------------
# VRF PUB
auto PUB
iface PUB inet manual
        pre-up ip link add $IFACE type vrf table 130
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down
        post-down ip link del $IFACE

# Uplink Bridge
iface ens19 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 4000-4020
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

# PUB Linknet
auto ens19.4003
iface ens19.4003 inet6 static
        pre-up ip link set dev $IFACE master PUB
        address 2001:db8:1234:3000::1:1/64

iface ens19.4003 inet6 static
        address fe80::2101:3/64

iface ens19.4003 inet static
        pre-up ip link set dev $IFACE master PUB
        address 10.3.254.21/24

VyOS Configuration

Now when the underlying network infrastructure is in order, I can start configuring the bridge interface on the VyOS router.

VyOS Bridge Configuration

Step 1: Here is the basic bridge configuration:

set interfaces bridge br0 enable-vlan
set interfaces bridge br0 mac '00:00:00:00:01:01'
set interfaces bridge br0 member interface eth1 allowed-vlan '4000-4020'
set interfaces bridge br0 member interface eth1 isolated
set interfaces bridge br0 member interface eth2 allowed-vlan '4000-4020'
set interfaces bridge br0 member interface eth2 isolated
set interfaces bridge br0 mtu '9198'
set interfaces bridge br0 stp

Explanation:

• This configures a bridge interface that allows VLANs and STP.

• The member interfaces are then configured as isolated. That means that LBS and SAUNA can’t communicate directly to each other (on the same subnet).

Here is a sample of VLAN specific configuration. As you can see there is nothing special about it:

set interfaces bridge br0 vif 4003 address '2001:464F:6F83:3000::1/64'
set interfaces bridge br0 vif 4003 address 'FE80::3:1/64'
set interfaces bridge br0 vif 4003 address '10.3.254.1/24'
set interfaces bridge br0 vif 4003 description 'PUB-Linknet'
set interfaces bridge br0 vif 4003 ipv6 address no-default-link-local
set interfaces bridge br0 vif 4003 vrf 'PUB'

Source: https://docs.vyos.io/en/latest/configuration/interfaces/bridge.html

Verification

A ping from SAUNA-VM1 verifies that the default gateway and other VMs in the same group is pingable, but other tenants are not:

# Ping to LBS-RO1:
sauna-vm1:~$ sudo ip vrf exec PUB ping 2001:DB8:1234:3000::1
PING 2001:DB8:1234:3000::1 (2001:DB8:1234:3000::1) 56 data bytes
64 bytes from 2001:DB8:1234:3000::1: icmp_seq=1 ttl=64 time=0.441 ms
64 bytes from 2001:DB8:1234:3000::1: icmp_seq=2 ttl=64 time=0.537 ms
64 bytes from 2001:DB8:1234:3000::1: icmp_seq=3 ttl=64 time=0.437 ms
^C
--- 2001:DB8:1234:3000::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2025ms
rtt min/avg/max/mdev = 0.437/0.471/0.537/0.046 ms

# Ping to LBS-VM1
sauna@sauna-vm1:~$ sudo ip vrf exec PUB ping 2001:DB8:1234:3000::1101
PING 2001:DB8:1234:3000::1101 (2001:DB8:1234:3000::1101) 56 data bytes
From 2001:DB8:1234:3000::2101 icmp_seq=1 Destination unreachable: Address unreachable
From 2001:DB8:1234:3000::2101 icmp_seq=2 Destination unreachable: Address unreachable
From 2001:DB8:1234:3000::2101 icmp_seq=3 Destination unreachable: Address unreachable
^C
--- 2001:DB8:1234:3000::1101 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3074ms

# Ping to SAUNA-VM2
sauna@sauna-vm1:~$ sudo ip vrf exec PUB ping 2001:DB8:1234:3000::2102
PING 2001:DB8:1234:3000::2102 (2001:DB8:1234:3000::2102) 56 data bytes
64 bytes from 2001:DB8:1234:3000::2102: icmp_seq=1 ttl=64 time=0.501 ms
64 bytes from 2001:DB8:1234:3000::2102: icmp_seq=2 ttl=64 time=0.497 ms
64 bytes from 2001:DB8:1234:3000::2102: icmp_seq=3 ttl=64 time=0.518 ms

Configure Bridge Firewall

Even though port isolation is configured, it still doesn’t prevent the VyOS router from routing the traffic destined to prefixes beyond the local subnets.

That means that some Firewall logic has to be implemented.

Here is VyOS documenation on Bridge firewalling:
https://docs.vyos.io/en/latest/configuration/firewall/bridge.html

The best solution I could come up with to allow tenants to reach the router itself and the internet, but each others networks, was to allow only the source prefixes towards the interface where it originates:

set firewall group ipv6-network-group BR0 network '2001:DB8:1234:1000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:2000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:3000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:4000::/64'

set firewall bridge output filter rule 10 action 'accept'
... rule 10 description 'Allow guests to reach the router'
... rule 10 source group ipv6-network-group 'BR0'
set firewall bridge output filter rule 20 action 'drop'
... rule 20 description 'Block if LBS SRC not equals eth1'
... rule 20 outbound-interface name '!eth1'
... rule 20 source address '2001:DB8:1234::/49'
set firewall bridge output filter rule 30 action 'drop'
... rule 30 description 'Block if SAUNA SRC not equals eth2'
... rule 30 outbound-interface name '!eth2'
... rule 30 source address '2001:DB8:1234:8000::/49'

Explanation:

• If for example the router would try to forward a packet with a source IP of 2001:DB8:1234:9001::1 to any other bridge member than eth2, the packet will be dropped.

• Alternatively, I could have used source MAC addresses instead of source interface. I could then revert back to use one shared segment for all tenants, instead of different “physical” interfaces.

Note: I actually reverted back to the almost original setup, but still using the bridge. More on that in the next post.

Verification

# Ping from SAUNA-VM1 to Internet:
sauna@sauna-vm1:~$ sudo ip vrf exec PUB ping -I 2001:DB8:1234:9001::1 2620:FE::FE
PING 2620:FE::FE (2620:fe::fe) from 2001:DB8:1234:9001::1 : 56 data bytes
64 bytes from 2620:fe::fe: icmp_seq=1 ttl=59 time=5.25 ms
64 bytes from 2620:fe::fe: icmp_seq=2 ttl=59 time=4.84 ms
^C
--- 2620:FE::FE ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 4.838/5.043/5.249/0.205 ms

# Ping from SAUNA-VM1 VLAN 30 to LBS-VM1 VLAN 30
sauna-vm1:~$ sudo ip vrf exec PUB ping -I 2001:DB8:1234:9001::1 2001:DB8:1234:3001::1
PING 2001:DB8:1234:3001::1 (2001:DB8:1234:3001::1) from 2001:DB8:1234:9001::1 : 56 data bytes
^C
--- 2001:DB8:1234:3001::1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6125ms

Conclusion

Benefits:

• With the setup I don’t have to create additional VRFs or subnets.

• I don’t need to update ipsets with new information everytime a new tenant gets added

Drawbacks:

• I have to add one additional bridge interface per tenant, meaning that the VyoS router will potentially get many additional interfaces. Although from PVE’s perspective I can possibly use the new bridges for LAN traffic as well.

• I have to add one additional rule for each new tenant added. Although it is better than having to edit ipsets to include new tenants prefixes.

If nothing else, this is a good demonstration of the firewall capabilities of the VyOS router.

Coming up: Alternative solution

In the next post I’m reverting back to one shared segment and only using the PVE firewall.

Appendix

Alternative bridge firewall configuration

Here is a configuration matching destination IP and source MAC, instead of outbound interface:

set firewall group ipv6-network-group BR0 network '2001:DB8:1234:1000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:2000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:3000::/64'
set firewall group ipv6-network-group BR0 network '2001:DB8:1234:4000::/64'
set firewall group mac-group LBS mac-address '00:00:00:00:11:01'
set firewall group mac-group SAUNA mac-address '00:00:00:00:21:01'
set firewall group mac-group SAUNA mac-address '00:00:00:00:21:02'

set firewall bridge output filter rule 1 action 'accept'
... rule 1 description 'Allow guests to reach the router'
... rule 1 source group ipv6-network-group 'BR0'
set firewall bridge output filter rule 10 action 'drop'
... rule 10 description 'Block if LBS SRCIP not equals LBS DSTMAC'
... rule 10 destination group mac-group '!LBS'
... rule 10 source address '2001:DB8:1234::/49'
set firewall bridge output filter rule 20 action 'drop'
... rule 20 description 'Block if SAUNA SRCIP not equals SAUNA DSTMAC'
... rule 20 destination group mac-group '!SAUNA'
... rule 20 source address '2001:DB8:1234:8000::/49'

The benefit of this is that I don’t have to use separate “physical” interfaces per tenant.

A drawback is that while prefixes inside the tenants are unreachable, it doesn’t isolate the tenants from reaching IP’s on the shared segment.