Docker Compose: Containerized router with VyOS
How-To: Docker Compose
In the previous post I described ways to reach the containers actual IP address through routing. In this post I will walk through all the steps on how to setup a containerized router with Docker Compose.
A few things before I begin:
I’m using an older image that is not from the official VyOS repository. The reason for that is that the instructions on how to setup an official container image was quite complex, while this one just works out of the box. The point with this post anyway is just to show that it can be done, not something that is necessarily best practice.
Alternatively you could use the FRR image. The compose configuration will almost be the same. I just wanted to try something new.
You can actually install Docker on VyOS (inverted scenario) but that is not part of the scope here.
Intended Design
Explanation:
The VyOS container and the nginx containers are part of the same docker compose project
The VyOS container exchange routes with the external GW over OSPFv3.
Network driver MACVLAN must be used on the outside interface pointing towards the external networks.
Network driver IPVLAN_L2 must be used on the inside interface, pointing towards the services.
My workstation is on a completely different network behind the external GW.
the nginx containers are just included for testing
Configuration
External router configuration
I’m using a Cisco Catalyst 3850 multilayer switch as an external gateway.
interface Vlan10
description MGMT-NETWORK
ip address 10.10.1.1 255.255.255.0
ipv6 address FE80::10:1 link-local
ipv6 address 2001:DB8:0:A010::1/64
ipv6 enable
ospfv3 priority 255
ospfv3 1 ipv4 area 1
ospfv3 1 ipv6 area 1
!
router ospfv3 1
router-id 10.10.1.1
!
address-family ipv4 unicast
exit-address-family
!
address-family ipv6 unicast
exit-address-family
!Explanation:
OSPFv3 is ready to peer with any OSPFv3 speaker in VLAN 10.
By default the network type is broadcast.
Priority 255makes sure that this router will always be the DR = Designated Router.
Docker Compose Configuration
Step 1: Create a work directory
I personally use the /var/local/ directory:
wl@sauna-nms:/var/local/bastuklubben.online/vyos-router-test$
Note: make sure to change owner from root to your user, if applicable
Step 2: Create related files
Nginx settings
I have modified the heading of default webpage for nginx to make it easier to verify that I have reached the correct container.
$ mkdir nginx
$ nano nginx/nginx1.index.html
...
<h1>Welcome to nginx example 1!</h1>
...Do the same for nginx2:.
$ nano nginx/nginx2.index.html
...
<h1>Welcome to nginx example 2!</h1>
...Optional: Environment variables
Optionally create a .env file. I’m using it only for changing the default project name:
nano .env
COMPOSE_PROJECT_NAME=vyosSummary:
These are the expected files in your working directory:
wl@sauna-nms:/var/local/bastuklubben.online$ tree -a vyos-router-test/
vyos-router-test/
├── .env
└── nginx
├── nginx1.index.html
└── nginx2.index.htmlStep 3: Create the Compose file
Create the compose.yml file with following content. Then read the notes below.
$ nano compose.yml
services:
router:
image: afla/vyos:1.4
volumes:
- /lib/modules:/lib/modules
- vyos-config:/opt/vyatta/etc/config/
privileged: true
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
networks:
outside:
ipv4_address: 10.10.1.254
ipv6_address: 2001:DB8:0:A010::FE
inside:
ipv4_address: 100.110.1.2
ipv6_address: 2001:DB8:0:A01A::2
nginx1:
image: nginx
volumes:
- ./nginx/nginx1.index.html:/usr/share/nginx/html/index.html
hostname: nginx1.bastuklubben.online
networks:
inside:
ipv4_address: 100.110.1.10
ipv6_address: 2001:DB8:0:A01A::A
nginx2:
image: nginx
volumes:
- ./nginx/nginx2.index.html:/usr/share/nginx/html/index.html
hostname: nginx2.bastuklubben.online
networks:
inside:
ipv4_address: 100.110.1.11
ipv6_address: 2001:DB8:0:A01A::B
volumes:
vyos-config:
networks:
outside:
internal: true
attachable: true
enable_ipv6: true
driver: macvlan
driver_opts:
parent: "enp0s3"
ipam:
config:
- subnet: 10.10.1.0/24
- subnet: 2001:DB8:0:A010::/64
inside:
attachable: true
enable_ipv6: true
driver: ipvlan
driver_opts:
parent: "enp0s3.110"
ipvlan_mode: "l2"
ipam:
config:
- subnet: 100.110.1.0/24
gateway: 100.110.1.1
- subnet: 2001:DB8:0:A01A::/80
gateway: 2001:DB8:0:A01A::1Notes about the “Outside” network:
The outside interface have the internal flag set, because it is unnecessary for the container to configure a default GW on this interface when we are planning to setup dynamic routing instead.
The outside interface is using MACVLAN driver because the router needs to have a unique MAC address, otherwise it will cause a routing loop.
Notes about the “Inside” network:
The subinterface enp0s3.110 will be created automatically. You don’t have to modify anything on the docker host or the hypervisor engine.
The IP addresses on the inside network interface for the router container has to be temporarily set to something else than 1 (The intended default GW). This is because even if the docker nodes aren’t setting any IP addresses when using the IPVLAN driver, the docker daemon still complains about the IP being used already:
⠴ Container vyos-router-1 Starting 3.6s
Error response from daemon: Address already in useStep 4: Start the compose project
$ docker compose up -d
[+] Running 5/5
✔ Network vyos_inside Created 0.1s
✔ Network vyos_outside Created 0.2s
✔ Container vyos-nginx1-1 Started 2.2s
✔ Container vyos-nginx2-1 Started 2.3s
✔ Container vyos-router-1 Started 2.4s Note: If you start it for the first time it probably takes a while longer.
VyOS configuration
Final step is to configure the newly built VyOS router.
Step 1: Enter the VyOS shell
$ docker exec -it vyos-router-1 vbash
vbash-4.1# vyos
vyos@vyos:~$Step 2: Verify interfaces
vyos@vyos:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - u/u
eth1 - u/u
lo 127.0.0.1/8 u/u
::1/128Which one of them are the outside interface? It might not necessarily be eth0, just because it was the first network listed in the compose file. One simple way is to see which interface has a unique mac address:
The Docker node:
$ ip address show enp0s3.110 | grep link/ether
link/ether 00:a0:98:37:1c:c0 brd ff:ff:ff:ff:ff:ffVyOS:
vyos@vyos:~$ show interface ethernet eth0 | grep eth | grep link
link/ether 00:a0:98:37:1c:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
vyos@vyos:~$ show interface ethernet eth1 | grep eth | grep link
link/ether 02:42:0a:0a:01:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0As you can see, eth1 must be the outside interface because the MAC address is different than the docker node, which is the second interface listed in the compose file. It has something to do with docker logics.
Step 3: Configure IP addresses
vyos@vyos:~$ configure
vyos@vyos# set interface ethernet eth1 address 2001:DB8:0:A010::FE/64
vyos@vyos# set interface ethernet eth1 address 10.10.1.254/24
vyos@vyos# set interface ethernet eth1 mtu 9198
vyos@vyos# set interface ethernet eth0 address 2001:DB8:0:A01A::1/80
vyos@vyos# set interface ethernet eth0 address 100.110.1.1/24
vyos@vyos# set interface ethernet eth0 mtu 9198
vyos@vyos# commit
vyos@vyos# save
Saving configuration to '/config/config.boot'...
Done
[edit]
vyos@vyos#Note: I have set MTU to 9198 globally in my network. For OSPF neighborship to form properly, one of the criteria is for MTU to match.
Verify reachability:
vyos@vyos# exit
exit
vyos@vyos:~$ ping 2001:DB8:0:A01A::A
PING 2001:DB8:0:A01A::A(2001:db8:0:a01a::a) 56 data bytes
64 bytes from 2001:db8:0:a01a::a: icmp_seq=1 ttl=64 time=0.164 ms
64 bytes from 2001:db8:0:a01a::a: icmp_seq=2 ttl=64 time=0.078 ms
...
vyos@vyos:~$ ping 2001:DB8:0:A010::1
PING 2001:DB8:0:A010::1(2001:db8:0:a010::1) 56 data bytes
64 bytes from 2001:db8:0:a010::1: icmp_seq=1 ttl=64 time=20.9 ms
64 bytes from 2001:db8:0:a010::1: icmp_seq=2 ttl=64 time=2.16 ms
...Step 4: Configure OSPFv3
vyos@vyos# set protocols ospfv3 parameters router-id 10.10.1.254
vyos@vyos# set protocols ospfv3 area 1 interface eth0
vyos@vyos# set protocols ospfv3 area 1 interface eth1
vyos@vyos# set protocols ospfv3 interface eth0 passive
vyos@vyos# set protocols ospfv3 interface eth0 ifmtu 9198
vyos@vyos# set protocols ospfv3 interface eth1 ifmtu 9198
vyos@vyos# set protocols ospfv3 redistribute connected
vyos@vyos# commit
[edit]
vyos@vyos# save
Saving configuration to '/config/config.boot'...
Done
[edit]
vyos@vyos#Explanation:
eth0 passivedisables sending of hello packets and is a best practice on LAN interfaces where no OSPF neighborships are expectedWith VyOS you have to redistribute connected routes into the OSPFv3 process for them to be advertised to the OSPF neighbors. Cisco devices just automatically advertise all routes on all OSPF enabled interfaces.
Verification
External router (Cisco)
SAUNA-SW1#show ipv6 route | i 2001:464F:6F83:A01A::/80
O 2001:464F:6F83:A01A::/80 [110/101]SAUNA-SW1#show ospfv3 neighbor
OSPFv3 1 address-family ipv4 (router-id 10.10.1.1)
Neighbor ID Pri State Interface ID Interface
10.14.1.1 0 FULL/ - 23 Vlan4010
OSPFv3 1 address-family ipv6 (router-id 10.10.1.1)
Neighbor ID Pri State Interface ID Interface
10.14.1.1 0 FULL/ - 23 Vlan4010
10.10.1.254 1 FULL/BDR 216 Vlan10Notice that OSPF neighborship has only formed over IPv6. With Cisco, the OSPFv3 configuration can be used to form neighborships over both IPv4 and IPv6. Main benefits include:
You don’t need to configure OSPFv2 in parallel.
You don’t need to configure IPv4 on link-nets. IPv4 destinations gets routed over IPv6 link-local addresses.
Sadly, it seems like neither VyOS or FRR is capable of doing the same. However, routing over IPv6 only for container networks are good enough for me.
VyOS
VyOS router commands to verify connection:
vyos@vyos:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 100.110.1.1/24 u/u
2001:DB8:0:a01a::1/80
eth1 10.10.1.254/24 u/u
2001:DB8:0:a010::fe/64
lo 127.0.0.1/8 u/u
::1/128vyos@vyos:~$ show ipv6 route ::/0
Routing entry for ::/0
Known via "ospf6", distance 110, metric 1, tag 1, best
Last update 02:13:16 ago
* fe80::10:1, via eth1, weight 1vyos@vyos:~$ show configuration
interfaces {
ethernet eth0 {
address 2001:DB8:0:A01A::1/80
address 100.110.1.1/24
mtu 9198
}
ethernet eth1 {
address 2001:DB8:0:A010::FE/64
address 10.10.1.254/24
mtu 9198
}
loopback lo {
}
}
protocols {
ospfv3 {
area 1 {
interface eth0
interface eth1
}
interface eth0 {
ifmtu 1500
network broadcast
passive
}
interface eth1 {
ifmtu 9198
}
redistribute {
connected {
}
}
}
}
...Connectivity test
Reachability from workstation to containers
As you can see, this workstation is on an entirely different network:
user@workstation:~$ ip address show enp3s0 | grep inet
inet 10.12.1.24/25 brd 10.12.1.127 scope global dynamic noprefixroute enp3s0
inet6 2001:464f:6f83:c010:4c12:7700:1e33:1967/64 scope global temporary dynamic
inet6 2001:464f:6f83:c010:ddfc:2e42:d6:7622/64 scope global dynamic mngtmpaddr noprefixroute
inet6 fe80::f7d6:c1db:55cb:9639/64 scope link noprefixrouteThe container is still reachable:
Reachability from container network to Internet
To test reachability from the inside network, one could exec into one of the containers and perform a ping command. But alas, the ping utility is not installed on the nginx image:
root@nginx1:/# ping google.com
bash: ping: command not foundYou can do curl instead:
root@nginx1:/# curl https://example.com
...
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>If niether tools are available, another alternative I found is by starting a container that can ping from the network and then cease to exist afterwards. This required the attachable option to be set to true on the inside network:
$ docker run --rm --network vyos_inside freelyit/ping -6 example.com
PING example.com (2606:2800:21f:cb07:6820:80da:af6b:8b2c): 56 data bytes
64 bytes from 2606:2800:21f:cb07:6820:80da:af6b:8b2c: seq=0 ttl=51 time=95.005 ms
64 bytes from 2606:2800:21f:cb07:6820:80da:af6b:8b2c: seq=1 ttl=51 time=94.785 ms
^Ccontext canceled