PfSense: Create a management interface with separate FIB
How To: PfSense
10th June 2024.
Background
When I upgraded my pfSense firewall from 23.9.1 to 24.3.0, my policy-based routing got destroyed. Routing to the management network no longer worked properly.
I’m using the LAN interface as a Out-of-Bound MGMT port. I had set a rule that for any incoming traffic toward the LAN IP, send it through the LAN Gateway. This was for preventing asymmetric routing. For whatever reason, in the latest update that caused me to loose access to servers and network devices inside the same subnet as the LAN interface. Once I deactivated it, I had access again.
In retrospect, my setup may actually have caused assymetric routing, instead of preventing it:
Incoming connections to the firewall OOBM interface from the user network probably got forwarded directly to the LAN/OOBM interface, since the pfSense had an entry in the routing table for that. Then policy based routing sent the traffic through the MGMT LAN. My conclusion is that I was causing the thing I was trying to prevent. The update was a feature, not a bug. It always helps when you draw out things.
The Solution
I already had plans to change the OOBM configuration to use a separate FIB = Forwarding Information Base. Now I got the a reason to do so.
Cleanup
First I had to cleanup some configuration (You may skip this part if it’s not relevant to you).
Remove Policy-Based Routing and IP addresses from LAN
Step 1: In the GUI, navigate to System > Routing and remove the LAN Gateways:
Step 2: Go to Firewall > Rules > LAN and remove some rules:
Step 3: Go to Interfaces > LAN and remove any IP addresses and gateways. These needs to be set from CLI. I will explain later.
Now we can begin with the multible FIB configuration
OOBM Interface configuration
The default FIB (FIB 0) contains all routing information that the firewall has. A separate FIB will contain a different list of routes that are specific for assigned interfaces. Multiple FIBs have long been supported on the FreeBSD OS, but for whatever reason, still not been added as a feature to pfSense! Therefore, I highly encourage you to take a backup before proceeding, as these instructions are officially not supported by pfSense.
Set Boot Loader variables to increase the FIB count
In the GUI, go to Diagnostics > Edit File
Enter /boot/loader.conf.local in the Path to file to be edited box
Press Load to load current config, if any (you might get an error if there isn’t a file but that’s OK).
Enter following to enable multiple FIBs (comments are optional):
# Enable multiple FIBs
net.fibs=4
# Disable route-leaking between FIBs
net.add_addr_allfibs=0Note: This will enable 4 fibs (0-3).
Click Save. and then reboot.
Sources:
Set interface and routes for FIB 1
Assuming you want your MGMT interface on FIB 1 (FIB 0 being the default). Open a command prompt to the firewall, either through SSH or navigate to Diagnostics > Command Prompt to enter commands from the GUI. Enter following (adjust values to what suits you):
ifconfig mvneta0.4091 inet 10.10.1.6 netmask 255.255.255.0 fib 1
ifconfig mvneta0.4091 inet6 2001:464F:6F83:A010::6 prefixlen 64 fib 1
setfib 1 route add -6 2001:464F:6F83:A010::/64 -interface mvneta0.4091
setfib 1 route add 10.10.1.0/24 -interface mvneta0.4091
setfib 1 route add -6 ::/0 2001:464F:6F83:A010::1
setfib 1 route add 0.0.0.0/0 10.10.1.1You can verify configuration from shell:
[admin@SAUNA-FW01]: setfib 1 netstat -rn
Routing tables (fib: 1)
Internet:
Destination Gateway Flags Netif Expire
default 10.10.1.1 UGS mvneta0.4091
10.10.1.0/24 link#10 US mvneta0.4091
Internet6:
Destination Gateway Flags Netif Expire
default 2001:464f:6f83:a010::1 UGS mvneta0.4091
2001:464f:6f83:a010::/64 link#10 US mvneta0.4091Once you have verified that the configuration works, you need to make the changes persist through reboot.
Make a startup script with Shellcmd.
Interface and route configuration will be erased after reboot. One way to prevent that is to make a startup script. One could edit the configration.xml to include the commands previously entered, but an easier approach is to download the Shellcmd package.
Install and configure the Shellcmd Packet
Step 1: Navigate to System > Package Manager and Install the package called "shellcmd"
Step 2: Then navigate to Services > Shellcmd and enter following commands:
Click Save and reboot.
Source: https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html
Note: The reason why you have to set the IP addresses through Shellcmd, and not through the regular GUI interface settings, is because if you set it through GUI, the LAN prefix and host address will be registered in FIB 0, as well as in FIB 1. That causes routing issues for other devices in the management subnet (at least it does for me on version 24.3).
Appendix
Add RC variables
Alternatively, you could put interface and routing configuration into /etc/rc.conf.local, but I haven’t got it to work properly. I’m pretty sure the configuration is correct, but I’m not sure if I’m putting it in the right place, or if it’s possible to configure this way at all on pfSense. Custom system tunables like these are easier to configure on TrueNAS, which is also based on FreeBSD:
# Set IP Addresses on LAN Interface
ifconfig_mvneta0.4091="inet 192.168.1.100 netmask 255.255.255.0 fib 1"
ifconfig_mvneta0.4091="inet6 2001:464F:6F83:A010::6 prefixlen 64 fib 1"
# IPv4 Static routes
static_routes="OOBMPrefix OOBMDefault"
route_OOBMPrefix="-net 10.10.1.0/24 -interface mvneta0.4091 -fib 1"
route_OOBMDefault="-net 0.0.0.0/0 10.10.1.1 -fib 1"
# Ipv6 Static routes
ipv6_static_routes="OOBMPrefix OOBMDefault"
ipv6_route_OOBMPrefix="-net 2001:464F:6F83:A010::/64 -interface mvneta0.4091 -fib 1"
ipv6_route_OOBMDefault="-net ::/0 2001:464F:6F83:A010::1 -fib 1"I’m just putting the configuration here for reference. If anyone manage to get it to work, please comment about it.
Sources: