Docker Networks Part 4: Reaching the containers IP address externally

Network Design

22nd July 2024

Updated 15th May 2025 - Added an update in the part about limitations when deploying a router as a container.

Most of the times with docker applications it’s enough to expose some ports to the docker node to have reachability to the services.

However, in some circumstances you might prefer to reach the actual IP address of the container itself. Then you have to setup routing capabilities between the docker node and the router on the LAN.

Routing the container networks

There are several ways one could perform routing on a Linux server.

Alternative 1: Install routing capabilities on the Docker node

One way is to install the FRR package and setup routing on all the docker nodes.

I have tried this and it works, but I don’t see how this will scale very well. Imagine a dozen more docker nodes on a LAN and all of them are sending hello messages and routing information. Not to mention the complex configuration needed.

Alternative 2: Use the IPVLAN Layer 2 Mode

Another way is to use the IPVLAN driver. Be cautious when using L2 mode though; you can risk causing Layer 2 loops in the network. But IPVLAN_L2 isn’t actually routing the traffic towards the container. The container is just bridged to the same LAN as the docker node.

Note:

• With the IPVLAN L3 Mode, you still need to enable routing capabilities on the docker node.

• The IPVLAN driver is also described in Part 1.

Alternative 3: Use a containerized router

You can use a containerized router image, for example a container version of FRR or VyOS that can do the routing. The containers then set the default gateway towards the router container.

Containerized router use cases

The benefit of having a containerized router is that you don’t need redundant routers (still nice to have though). Imagine you have 2 docker nodes working together with one containerized router with services behind it. If the node with the router goes down, the router container will just move over to the other node and the OSPF neighborship will re-establish.

At least this is how it should work in theory.

UPDATE: Another use case I found out recently is that Docker, or any container management tool that I know of, doesn’t support VRFs, but Linux does. If you could turn off the gateway creation on the host and let a router container do the forwarding instead, you could do interesting things! However, there are some limitations…

Limitations with using a containerized router

It is technically possible to setup this scenario, but practically not, at the moment.

In the perfect world, I would put the MACVLAN driver on the outside interface of the router (pointing towards external networks) and a regular bridge for the inside interface (towards services). The outside interface is no problem, however, the bridge interface will not work because the docker nodes will assign themselves an IP for the default GW, regardless if you specify it or not. This behavior can’t be modified.

---

UPDATE: This actually works now! Well, not really… but Docker introduced a new bridge option to disable automatic creation of IPv4 default gateways:

-o com.docker.network.bridge.inhibit_ipv4=true

To use it in docker compose:

networks:
  inside_int:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.inhibit_ipv4: "true"

However, there is still some issues left:

 ⠹ Container sauna-nms-ro1-1  Starting                                                                                   0.2s 
Error response from daemon: failed to set up container networking: Address already in use

There is no equivalent solution for IPv6. Even if you don’t assign an IPv6 gateway to the network, there is no mention of it in any inspect outputs, yet you can still see when running “ifconfig” that the host has assigned for example 2001:DB8:A019::1 to the bridge (the first IP in the scope).

And there is another issue as well: You can’t choose which containers should have a default gateway and which shouldn’t. It’s either all containers in the network have a default GW, or nobody has. I tried to add a default GW manually in the containers, but many of them have very limited functionality; they are designed to do one specific thing. For example niether “ip route” or “nmcli” is working on the nginx container. Then you have to create custom containers including one of those packages. However, if you have to make too much custom configuration to make something work, it’s not a good design anymore.

---

The only way I could make it work, with minimal complexity, was to use the IPVLAN_L2 driver on the inside interface. Then the services will set their default GW towards what usually would be the external router, but instead you point them to a subinterface where only the container router has an IP. However, I’m not sure if this will work with multiple docker nodes yet. That is why the bridge driver would have been the ideal choice.

In an upcoming post I will demonstrate how this is configured in detail.