Dedicated Server + Proxmox + OPNSense + IPv6 Support for VMs in LAN

[eakteam]

Hi everyone. I am struggling setting up a working IPv6 network for VMs connected at OPNsense LAN interface.
The provider (Hetzner) has given me a /64 IPv6 network:  2a01:4f8:****:****::/64
Somehow I am able to create a working IPv6 network for Proxmox itself and OPNsense with below configurations:
Proxmox (Debian) /etc/network/interfaces:

Code Select Expand

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
        address 94.130.***.***/26
        gateway 94.130.***.***
        up route add -net 94.130.***.*** netmask 255.255.255.192 gw 94.130.***.*** dev enp0s31f6
       # route 94.130.***.***/26 via 94.130.***.***

iface enp0s31f6 inet6 static
        address 2a01:4f8:****:****:aaaa::11/128
        gateway fe80::1

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

iface vmbr0 inet6 static
        address 2a01:4f8:****:****:aaaa::1336/127
        up ip route add 2a01:4f8:****:****::/64 via 2a01:4f8:****:****:abcd::1337

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

iface vmbr1 inet6 manual

OK, now the hard part:

Ping from Proxmox to OPNSense works OK
Ping from OPNSense to Proxmox works OK
Ping from Proxmox to google: ping6 google.com is OK
Ping from OPNSense to google: ping6 google.com is OK
Ping from VM to OPNSense (2a01:4f8:****:****:aaaa::1337) works OK
Ping from VM to Gateway (2a01:4f8:****:****:aaaa::1336) FAILS
Ping from VMs to Google (ping6 google.com) FAILS

In OPNSense, at  IPv6 Configuration Type(LAN), choosed Static IPv6 with the following values: 2a01:4f8:****:****:0172:0016:0:0001/125
In OPNSense, at ISC DHCPv6: [LAN], enabled the services and added to Range from: 2a01:4f8:****:****:172:16:0:1 -> to 2a01:4f8:****:****:172:16:0:7
In OPNSense, at Router Advertisements: [LAN], choosed Managed and DNS Servers as following:
2001:4860:4860::8888
2001:4860:4860::4444

When trying to boot a VM e.g. in my case 1 Windows and 1 Ubuntu, they are getting an IPv6 address from DHCPv6 from OPNSense but cannot access or resolve the IPv6 adresses.

Ping shows current values:

Code Select Expand

ping6 google.com
PING google.com(fra24s05-in-x0e.1e100.net (2a00:1450:4001:828::200e)) 56 data bytes
--- google.com ping statistics ---
33 packets transmitted, 0 received, 100% packet loss, time 32747ms

What I am missing? Really getting tired for days with trying different configurations, but doesn't work. If anybody can assist I really appreciate that a looooot.

[Founding9371]

Hetzner offers /56 subnets. Make your life easier and ask for a /56 :)
It has a 1-time setup fee thats it.

[Homeyer]

Setting up IPv6 networking can be a real challenge, especially with all the configurations involved. It looks like you've done a lot of troubleshooting already, which is great. One thing to double-check is the routing on your Proxmox host. Make sure it's correctly forwarding IPv6 traffic to your OPNsense router.
Personally, I use this dedicated server India, and it gives me full control over my networking setup, which might help in your case. You might also want to check Hetzner's documentation or forums for specific IPv6 setup tips since they provided your address block.

[mooh]

None of your bridges has a bridge-port. How do you connect your VMs to a network? Please post a client's configuration.

Generally speaking, you're probably looking for something like this for your Proxmox server:

Code Select Expand

iface enp129s0f0 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp129s0f0
        bridge-stp off
        bridge-fd 0

iface vmbr0 inet static
        address 192.168.170.140/24
        gateway 192.168.170.1

Add IPv6 to taste. Connect your clients like this (Hardware section):

Code Select Expand

Network device      bridge=vmbr0

[Alpha_DE]

I needed quite some effort to get the combination of Proxmox 7 and OPNSense on a Hetzner root server running.

Finally, I ordered a /56 (10 Euro one time for assignment although RIPE tells one should get a /56 and not a /65, nice cash cow), created two interfaces (one for routing between the host and OPNSense) and the other one as internal v6 interface between the VMs.

That did work finally.

@eakteam I can give you a copy of my working /etc/network/interfaces but you need to order that additional /56 network.

During the weekend, I migrated all VM to a new machine with Proxmox 8.

Hetzner IP handling requires a manual hardware exchange to keep the existing v4/v6 IPs.

After that was done, I noticed the following day, that IPv6 inbound to the VM was again broken. Ping was possible, outbound from the VM did work and everything has been left untouched.

I just discovered the reason.

On creation of that internal IPv6 vmbr interface for v6 communication between the VMs (OPNsense and the rest), a routing rule was added for this internal /64 before the routing rule to get v6 traffic from the host to OPNsense was added.

This new rule (did not happen with Proxmox 7) took obviously precedence over the routing rule of v6 traffic to OPNsense.

After deleting it before adding the v6 OPNsense routing rule on the host, the problem was gone.

I can only recommend to check not only interface and iptables setting but especially the routing table.

On Proxmox 7 I had a similar issue on the v4 side, I needed to delete a rule there, too.

XXXX:XXXX:XXXX:XXXX:: the /64 network used solely on the host
YYYY:YYYY:YYYY:YY:: the /56 network used for the VM

The troubling rule is marked with ***, the removal of that rule solved it.

Code Select Expand

================ new (troubling) routing table ========================
XXXX:XXXX:XXXX:XXXX::2 dev eno1 proto kernel metric 256 pref medium
XXXX:XXXX:XXXX:XXXX::/64 dev vmbr1 proto kernel metric 256 pref medium
YYYY:YYYY:YYYY:YY01::/64 dev vmbr2 proto kernel metric 256 pref medium
*** YYYY:YYYY:YYYY:YY02::/64 dev vmbr3 proto kernel metric 256 expires 86246sec pref medium ***
YYYY:YYYY:YYYY:YY02::/64 via 2a01:4f8:191:fe01::3 dev vmbr2 metric 1024 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr1 proto kernel metric 256 pref medium
fe80::/64 dev vmbr2 proto kernel metric 256 pref medium
fe80::/64 dev vmbr3 proto kernel metric 256 pref medium
fe80::/64 dev eno1 proto kernel metric 256 pref medium
fe80::/64 dev vmbr4000 proto kernel metric 256 pref medium
default via fe80::1 dev eno1 proto kernel metric 1024 onlink pref medium
default via fe80::8031:e8ff:fe74:2ded dev vmbr3 proto ra metric 1024 expires 1646sec hoplimit 64 pref medium