Advanced Linux Network Configuration on Ubuntu 24.04
How-To: Ubuntu Linux Server Administration
Monday 28th April 2025
In my previous post I wrote about VyOS routing configuration. This post builds on that by finishing the OSPFv3 peering.
Network Design
Layer 2 design
Explanation:
This post will cover:
Configuration for VLAN 10, 20 and 30. These will be used for containers.
VLAN configuration for the downlink Linux Server peering interfaces (4001 - 4003), from the Linux Server perspective
Only configuration for bastuklubben will be covered. “LBS” is just included in the drawing to show that the setup can scale to multiple servers.
The vmbr1 is a OVS bridge interface defined in Proxmox VE. To learn how to setup the network on the Proxmox side, view Proxmox VE Setup Part 2: Network
To prevent vlan 10, 20 and 30 from becoming the same VLAN 10, 20 and 30 on every servers, I create one additional OVS bridge per server. This will ensure that the internal networks will be isolated.
Note: One server represents one tenant, i.e. different organizations isolated from each other.
Layer 3 Design
Explanation:
The servers are running VRFs with different docker networks attached. The docker networks are not in the scope of this post.
OSPFv3 with FRR Routing is enabled between the VyOS router and the Server. In practice, the Ubuntu Server is mainly acting as a router.
One VM per network zone, or one VM with one VRF per network zone?
When designing this I thought about security best practices. Usually you wouldn’t want to mix internal, management and public services on the same machine. Before I used to create 3 different Servers and put them in respective zones.
Now when all services will be running inside containers, does it matter that much anymore? In addition, the different types of services will have separate docker networks divided into VRFs. If one container gets compromised in the public zone, I doubt the attacker will get that much further from there.
Configuration
Netplan Configuration
As I have written about in my Linux Tutorial course, Ubuntu uses Netplan which is a YAML based configuration file to configure the network settings. This is quite powerful, intuitive and supports almost everything, except dynamic routing protocols.
network:
renderer: networkd
version: 2
ethernets:
ens18: {}
ens19: {}
vrfs:
INT:
table: 120
interfaces:
- br4002
- br20
PUB:
table: 130
interfaces:
- br4003
- br30
bridges:
# Default RIB
br4001:
addresses:
- FE80::4001:3/64
interfaces:
- vlan4001
accept-ra: no
br10:
addresses:
- 2001:DB8:1234:A000::1/64
- FE80::10:1/64
- 10.10.0.1/24
interfaces:
- vlan10
accept-ra: no
# VRF INT
br4002:
addresses:
- FE80::4002:3/64
interfaces:
- vlan4002
accept-ra: no
optional: true
br20:
addresses:
- 2001:DB8:1234:B000::1/64
- FE80::20:1/64
- 10.11.0.1/24
interfaces:
- vlan20
accept-ra: no
# VRF PUB
br4003:
addresses:
- FE80::4003:3/64
interfaces:
- vlan4003
accept-ra: no
br30:
addresses:
- 2001:DB8:1234:9000::1/64
- FE80::30:1/64
- 10.9.0.1/24
interfaces:
- vlan30
accept-ra: no
vlans:
vlan10:
id: 10
link: ens19
vlan20:
id: 20
link: ens19
vlan30:
id: 30
link: ens19
vlan4001:
id: 4001
link: ens18
vlan4002:
id: 4002
link: ens18
vlan4003:
id: 4003
link: ens18Explanation:
The NMS zone will be the default routing table so it doesn’t need VRF configuration
One thing that you might notice is that there are no configured DNS servers. Read in the appendix to learn where the DNS has been defined.
Another thing that you might notice is that there are no static routes configured for the VRFs. In the Netplan VRF Example, the static route is configured under the VRF section. The problem with that is it doesn’t allow you to specify source interface, something that is required when using only link-local addresses. That is why I need to enable a dynamic routing protocol like OSPFv3.
FRR Configuration
Ubuntu Server doesn’t support dynamic routing protocols natively. Therefore I’m using FRR = Free Range Routing to complete the configuration. Since I already have defined the VRFs and interfaces in Netplan, I only need to configure OSPFv3.
Install FRR
To install the latest version, you can add the FRR debian repository:
# Add GPG key:
curl -s https://deb.frrouting.org/frr/keys.gpg | sudo tee /usr/share/keyrings/frrouting.gpg > /dev/null
# Add latest version:
FRRVER="frr-stable"
echo deb '[signed-by=/usr/share/keyrings/frrouting.gpg]' https://deb.frrouting.org/frr \
$(lsb_release -s -c) $FRRVER | sudo tee -a /etc/apt/sources.list.d/frr.listTo install FRR, run:
sudo apt update && sudo apt install frr frr-pythontoolsSource: https://deb.frrouting.org/
Setup FRR
Enable routing daemons:
sudo nano /etc/frr/daemons
bgpd=yes
ospfd=yes
ospf6d=yes
...Add following services to /etc/ services:
zebrasrv 2600/tcp # zebra service
zebra 2601/tcp # zebra vty
ripd 2602/tcp # RIPd vty
ripngd 2603/tcp # RIPngd vty
ospfd 2604/tcp # OSPFd vty
bgpd 2605/tcp # BGPd vty
ospf6d 2606/tcp # OSPF6d vty
ospfapi 2607/tcp # ospfapi
isisd 2608/tcp # ISISd vty
babeld 2609/tcp # BABELd vty
nhrpd 2610/tcp # nhrpd vty
pimd 2611/tcp # PIMd vty
ldpd 2612/tcp # LDPd vty
eigprd 2613/tcp # EIGRPd vty
bfdd 2617/tcp # bfdd vty
fabricd 2618/tcp # fabricd vty
vrrpd 2619/tcp # vrrpd vtyAnd then restart FRR:
sudo systemctl restart frrSource:
https://docs.frrouting.org/en/latest/setup.html
Note: I discovered another Substack bug. I can't write the filepath '/etc/' followed by 'services' without getting network error!
FRR Routing Configuration
My FRR configuration:
~$ sudo vtysh
# show running-config
ipv6 forwarding
!
ipv6 route 2001:db8:1234:1000::/60 blackhole tag 3 254
!
vrf INT
ipv6 route 2001:db8:1234:2000::/60 blackhole tag 3 254
exit-vrf
!
vrf PUB
ipv6 route 2001:db8:1234:3000::/60 blackhole tag 3 254
exit-vrf
!
interface br10
ipv6 ospf6 area 1
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 passive
exit
!
interface br4001
ipv6 ospf6 area 0
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 priority 0
exit
!
interface br20
ipv6 ospf6 area 1
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 passive
exit
!
interface br4002
ipv6 ospf6 area 0
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 priority 0
exit
!
interface br30
ipv6 ospf6 area 1
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 passive
exit
!
interface br4003
ipv6 ospf6 area 0
ipv6 ospf6 cost 10
ipv6 ospf6 network broadcast
ipv6 ospf6 priority 0
exit
!
router ospf6 vrf INT
ospf6 router-id 10.2.0.1
log-adjacency-changes
redistribute static
area 1 range 2001:db8:1234:2000::/60 not-advertise
area 1 stub
exit
!
router ospf6 vrf PUB
ospf6 router-id 10.3.0.1
log-adjacency-changes
redistribute static
area 1 range 2001:db8:1234:3000::/60 not-advertise
area 1 stub
exit
!
router ospf6
ospf6 router-id 10.1.0.1
log-adjacency-changes
redistribute static
area 1 range 2001:db8:1234:1000::/60 not-advertise
area 1 stub
exit
!Note: ip forwarding and ipv6 forwarding is not enabled by default and won’t get saved in when configuring from vtysh. Those commands have to be added directly into the config file, /etc/frr/frr.conf.
Explanation:
The br10, 20 and 30 interfaces are passive since no OSPF speakers are expected there
The br4001 - 4003 interfaces are the linknets to the VyOS router and has set priority 0 to make sure they never become the DRs = Designated Routers.
For OSPFv3 routes to be advertised you need to to either:
redistribute routes into OSPFv3
Make a summary route with
“area 1 range advertise”
There are two reasons why I use both redistribute static and area 1 range command:
When you re-distribute routes you can add a route-map or a tag to manipulate routes. The tags set in the static routes are still present after re-distribution.
When re-distributing static routes, the directly connected prefixes within the summary are also included. The “
area 1 range X:X:X:X::/60not-advertise”excludes all the directly connected prefixes within the summary route and only advertises the redistributed static route.
Verification
You can verify the routing table from FRR. Notice there is a default route learned via OSPF:
sauna-vm1# show ipv6 route vrf PUB
...
VRF PUB:
O>* ::/0 [110/10] via fe80::be24:11ff:fe65:3823, br4003, weight 1, 1d21h53m
O 2001:db8:1234:9000::/64 [110/10] is directly connected, br30, weight 1, 1d21h53m
C>* 2001:db8:1234:9000::/64 is directly connected, br30, 1d21h53m
sauna-vm1# show ipv6 ospf6 vrf PUB neighbor
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
10.3.255.1 1 00:00:39 Full/DR 03:48:03 br4003[DROther]You can also verify the routing table from the Linux shell with the following command:
sauna@sauna-vm1:~$ ip -6 route show vrf PUB
...
2001:db8:1234:9000::/64 dev br30 proto kernel metric 256 pref medium
...
default nhid 146 via fe80::be24:11ff:fe65:3823 dev br4001 proto ospf metric 20 pref mediumAnd you can ping from a specific VRF:
sauna@sauna-vm1:~$ ping -I PUB 2620:fe::fe
ping: Warning: source address might be selected on device other than: PUB
PING 2620:fe::fe (2620:fe::fe) from 2001:db8:1234:9000::1 PUB: 56 data bytes
64 bytes from 2620:fe::fe: icmp_seq=1 ttl=59 time=10.6 ms
64 bytes from 2620:fe::fe: icmp_seq=2 ttl=59 time=10.5 ms
...Appendix
DNS Workaround
For whatever reason, DNS won’t resolve with this setup:
br10:
addresses:
- 2001:464F:6F83:A000::1/64
- FE80::10:1/64
- 10.10.0.1/24
interfaces:
- vlan10
accept-ra: no
nameservers:
addresses:
- 2620:FE::FE
- 2620:FE::9It just times out:
sauna@sauna-vm1:~$ nslookup example.com
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
^CThe DNS server is reachable though:
sauna@sauna-vm1:~$ ping 2620:FE::FE
PING 2620:FE::FE (2620:fe::fe) 56 data bytes
64 bytes from 2620:fe::fe: icmp_seq=1 ttl=59 time=10.5 ms
64 bytes from 2620:fe::fe: icmp_seq=2 ttl=59 time=9.62 msIt might have something to do with the fact that the linknet 4001 doesn’t have a public IP. I found a workaround by setting the same DNS addresses within this file:
sudo nano /etc/systemd/resolved.conf
[Resolve]
DNS=2620:fe::fe
FallbackDNS=2620:fe::9sudo systemctl restart systemd-resolved
Now it starts to resolve DNS queries:
sauna@sauna-vm1:~$ nslookup example.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: example.com
Address: 23.215.0.136
Name: example.com
Address: 23.215.0.138
...DNS even works inside VRFs now:
sauna@sauna-vm1:~$ ping -I PUB example.com
ping: Warning: source address might be selected on device other than: PUB
PING example.com (2600:1408:ec00:36::1736:7f24) from 2001:db8:1234:9000::1 PUB: 56 data bytes
...systemd-networkd-wait-online timeout
When booting up a new Ubuntu clone, the service “systemd-networkd-wait-online” timer times out, even if all interfaces are working fine after boot finishes.
I read this article that explains the reason behind that service. I tried all the possible solutions but I never found the root cause. The network works fine regardless. I ended up disabling the service so I don’t have to wait 2 minutes for the servers to boot.
sudo systemctl disable systemd-networkd-wait-online.service
sudo systemctl mask systemd-networkd-wait-online.service