IPv6 Routing to Downstream Router

[sifrmoja]

For clarification; I previously used a Mikrotik Router with this configuration and everything worked perfectly.

I am having bizarre issues with routing IPv6 to downstream routers from my OPNsense router. It works to a point and then causes all sorts of problems. It seems almost random.

Here is a guide I wrote about how it was all configured with my Mikrotik setup.

I have a couple of servers downstream from my OPNsense that are running Incus for containers and VMs. I manually set a IPv6 /64 network on the Linux bridges on those servers. In OPNsense I have created a disabled gateway with the IP of the server and then a route to the IPv6 subnet assigned to the linux bridge.
This gives access to the servers but then I have all kinds of issues with delays and I'm unable to traceroute between containers/VMs on the servers. The containers on one server have no issues with accessing everything sometimes and then it starts to fail.

I was hoping someone might have instructions on how they have configured this type of setup and I might see where I have gone wrong.

[Monviech (Cedrik)]

Random access issues between clients sound like NDP (NA, NS) not working correctly. Do packet captures to see if all neighbors can find each other.

[sifrmoja]

I seem to have it all working now but it is weird how I have had to make it work considering how basic it is to setup on RouterOS.

On each of my servers I have installed radvd. They then learn about each other and add a route. OPNsense does not learn about these routes. I still have to manually add gateways and routes.

[sifrmoja]

I am lost about what is happening here.

LAN receives an IPv6 /64 using Track Interface. Devices on the network receive their IPv6 with SLAAC.

I have some servers hosting Linux Containers. The containers network bridge has a static IPv6 /64 subnet configured from the /48 I receive from my ISP.

I have created a gateway on OPNsense for each of the servers IPv6 address. I then add a route to the static IPv6 /64 subnet specified on the network bridge with the matching gateway.

At this point everything appears to be working but...

The problem I am seeing is that connections from a device on the LAN network are not stable to a container via IPv6. I connect to a Minecraft server and then it drops after around 30 seconds and the firewall log is flooded with "Default deny / state violation rule" relating to that connection.

[Patrick M. Hausen]

That happens when the OPNsense, your client, and the server all share the same network. Client --> OPNsense --> Server. But the reply is routed directly bypassing OPNsense and so triggering a state violation sooner or later.

Would happen with IPv4, too, if the topology was the same.

Best move the servers into a DMZ (VLAN) so all traffic passes through OPNsense, literally.

[sifrmoja]

The servers have no direct connection to the bridges on each other. The only way they know how to get to each others bridge is through the OPNsense. I had this working on Mikrotik RouterOS by configuring the routes as shown in my write-up of it. Any device on my LAN network can only access the bridges through the OPNsense.

I may have a misunderstanding of how the Layer 2 traffic is functioning here. If I am not seeing a neighbour, from the bridge subnet, in the NDP table on any device in the LAN network, wouldn't that mean it isn't being sent to there on L2?

[sifrmoja]

My brain finally caught up. The link-local address of the server is added to the NDP table. My bad. I wonder why RouterOS didn't see this as an issue.