Monday 8th September
Background
I have a debian linux server that have docker apps and in addition operates as a router for a fedora VM behind it. This setup worked fine in lab but when I implemented the same design in my production environment, I struggled for hours to figure out why traffic from the “client” VM could not reach IP’s beyond the router VM.
Overview
Here is a Layer 2 diagram of the scenario:
The setup on the right (Bastuklubben) was working fine but the one on the left (LBS) was not.
Troubleshooting Steps
Layer 1 and layer 2 functionality are confirmed with pings and thus not need further explanation.
Layer 3 Routing
Traffic from the Linux VM behind the debian server could reach the interface vmbr2 of the debian server, indicating that the default GW is set correctly, but would not be forwarded to addresses beyond that, for example the vmbr2 interface on the upstream VyOS router.
The debian router itself had no problem pinging addresses on the Internet from vmbr3 as the source interface, so the VyOS router should know where to send the return traffic. I even verified to routing table.
Normally this indicates that IPv6 forwarding must be turned off in the kernel, but it was not as i verified with this command:
$ sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 1Even verified that the specific interfaces had IPv6 forwarding enabled:
$ sysctl net.ipv6.conf.ens19.forwarding
net.ipv6.conf.ens19.forwarding = 1
$ sysctl net.ipv6.conf.ens20.forwarding
net.ipv6.conf.ens20.forwarding = 1So the problem must be somewhere else
Layer 4 Security
Security parameters can quickly become complex and difficult to keep track on. In my environment there are 3 firewalls:
The pfSense appliance at the Internet Edge. Since traffic doesn’t even reach that firewall, I can safely assume that it’s not the root cause
Proxmox has firewall activated to filter traffic between virtual machines. I double-checked that those settings are not the issue because I have got stung by them in the past.
And then there is a client firewall as well…
For some reason, ip6tables was configured to block all forwarding of IPv6!
$ sudo ip6tables -n -L -v
...
Chain FORWARD (policy DROP 22271 packets, 1840K bytes)
...Comparison with IPv4:
$ sudo iptables -n -L -v
...
Chain FORWARD (policy ACCEPT 29 packets, 6852 bytes)
... After changing the rule to ACCEPT, traffic could finally flow through the debian server:
$ sudo ip6tables -L FORWARD -v -nMake the change persistent:
dpkg-reconfigure iptables-persistentWhy so many firewalls?
The deeper inside the network you get, the more granular control. The edge firewall takes care of threats coming from the outside, and the Proxmox firewall is used to block traffic between virtual machines.
The client firewall is mostly just an extra complexity but it is necessary for my DMVPN setup to work.
Final thoughts
This looks pretty easy in hindsight but I completely forgot about the client firewall. I struggled for hours and was very confused. Of course it had to be a firewall issue. It’s always that, or DNS, or both.
Just before I was about to re-image the whole VM I asked Leo for help and he reminded me about ip6tables.