Multiple WAN IP Addresses with different MAC but same Gateway on Hetzner Network

[luckylinux]

Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required.

It definitely does. An IPv6 interface will always have a link local address in addition to the global unicast address you assigned. So it can use another link local address on the same interface just fine.

See attached screenshots. This is OPNsense on bare metal, though.

I'm not doubting it works in your case. In mine it doesn't though for some weird reason. Maybe because the Linux Bridge (Switch) causes fe80::1 to become Ambiguous (is it Hetzner's Gateway, the Proxmox VE Gateway, etc) ?

Anyway, it works with the "Direct" Proxmox VE Host as a Gateway.

I also have both Link Local Address and Global Unicast Address in OPNSense:

Code: [Select]

2a01:XXXX:XXXX:XXXX:1::10/96
fe80::YYYY:YYYY:YYYY:YYYY/64
Weird Stuff ... Could also be because I disabled some multicast/unicast flooding in the Bridge though (I don't want to receive yet other MAC Address Abuse from Hetzner).

On another note, I don't understand why, unless I put a Firewall Rule in place (which another User on Proxmox Forums recommended to me), ALL other "NEIGHBOR" Hetzner Servers in my IPv4 Subnet are dumping traffic on me (like I am seeing & dropping Traffic that is destined to my Neighbors IP Address, and if I don't, Hetzner claims I have some non-authorized SOURCE MAC Addresses coming out of my Ethernet Port).

Why should I get traffic destined to another Server IP in the First Place ? That seems some weird Stuff going on.

I am receiving IPv4 TCP/UDP/ICMP Traffic with "Destination = Neighbor Server IP" (not destined to my Server !), of about all range of ports (for TCP/UDP I mean, NOT only the 32768-65535 Range, also e.g. port 8008 etc). Is this a Multicast Traffic ?

Do you also face this Issue at Hetzner ? Funnily the ARP Table (and NDT for IPv6 for that Matter) only shows "my" MAC Addresses / IPv6 Addresses (plus the IPv4 Gateway MAC Address for ARP) in OPNSense.

EDIT 1: By the Way, testing IPv6 via Interfaces -> Diagnostics -> Trace Route I always get "timeout reached", but from SSH into the Router, the following work Correctly:

Code: [Select]

# ICMP Traceroute
root@OPNSense:/ # traceroute6 -I 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 20 byte packets
 1  2a01:XXXX:XXXX:XXXX:1::1  0.149 ms *  0.211 ms
 2  2a01:XXXX::XXXX:XXXX:b  0.365 ms  2.220 ms  0.287 ms
 3  2a01:XXXX:0:XXXX::XXXX  0.571 ms  0.474 ms  0.341 ms
 4  2a01:XXXX:0:XXXX::XXXX  4.964 ms  4.898 ms  4.898 ms
 5  2001:4860:1:1::1a4  5.157 ms  5.038 ms  5.094 ms
 6  2a00:1450:8155::1  5.205 ms  5.219 ms  5.211 ms
 7  dns.google  5.204 ms  5.189 ms  5.201 ms

# UDP Traceroute
root@OPNSense:/ # traceroute6 -U -p 33434 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 28 byte packets
 1  2a01:XXXX:XXXX:XXXX:1::1  0.135 ms * *
 2  2a01:XXXX::XXXX:XXXX:b  0.457 ms  0.326 ms  0.304 ms
 3  coreXXX.XXXX.hetzner.com  11.941 ms  0.622 ms
    2a01:XXXX:XXXX:XXXX::XXXX  0.497 ms
 4  coreXXX.XXXX.hetzner.com  4.958 ms
    2a01:XXXX:XXXX:XXXX::XXXX  5.180 ms
    2a01:XXXX:XXXX:XXXX::XXXX  5.102 ms
 5  2001:4860:1:1::19c8  4.948 ms
    2001:4860:1:1::624  6.293 ms
    2001:4860:1:1::19c8  5.235 ms
 6  2a00:1450:8037::1  5.239 ms
    2a00:1450:8057::1  6.570 ms
    2a00:1450:8463::1  5.356 ms
 7  dns.google  5.270 ms  5.112 ms  5.248 ms

# TCP Traceroute gets Lost sometimes once it gets out of Hetzner Datacenter I guess
root@OPNSense:/ # traceroute6 -T 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 20 byte packets
 1  2a01:XXXX:XXXX:XXXX:1::1  0.158 ms *  0.250 ms
 2  2a01:XXXX::XXXX:XXXX:b  0.364 ms  2.267 ms  0.339 ms
 3  2a01:XXXX:XXXX:XXXX::XXXX  0.577 ms  0.495 ms
    coreXX.XXX.hetzner.com  0.453 ms
 4  coreXX.XXX.hetzner.com  5.096 ms  4.982 ms  5.170 ms
 5  *
    2001:4860:1:1::19c8  5.257 ms *
 6  * * *
 7  * * *


From the Host it seems to behave better with regards to TCP Traceroute:

Code: [Select]

root@Proxmox:~# traceroute6 -T -p 443 2001:4860:4860::8888
traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 68 byte packets
 1  2a01:XXXX::XXXX:XXXX:b (2a01:XXXX::XXXX:XXXX:b)  0.401 ms  0.398 ms *
 2  * * coreXX.XXX.hetzner.com (2a01:XXXX:XXXX:XXXX::XXXX)  0.395 ms
 3  coreXX.XXX.hetzner.com (2a01:XXXX:XXXX:XXXX::XXXX)  4.840 ms * *
 4  * * *
 5  * * 2a00:1450:814d::1 (2a00:1450:814d::1)  4.912 ms
 6  dns.google (2001:4860:4860::8888)  5.331 ms * *

Although I must say that when I'm doing TCP Traceroute from Proxmox VE, I can see in the Firewall Logs that it's "consistent". It's always targeting Port 443 as Instructed -> DPT=443

Whereas when I do TCP Traceroute from OPNSense, I can see in the Proxmox VE Host Firewall Logs that the Destination Port is changing A LOT during the Traceroute: DPT=444, DPT=445, DPT=446, ... , DPT=465, ...

So I don't know ... Is the Traceroute Implementation *that* different from GNU/Linux to FreeBSD ?

[Patrick M. Hausen]

When you spoof MAC addresses the interface is set to promiscuous mode which means you will see all traffic on that particular network destined at broadcast or multicast addresses or ones where the switch has not yet learned the correct port.

Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?

[luckylinux]

When you spoof MAC addresses the interface is set to promiscuous mode which means you will see all traffic on that particular network destined at broadcast or multicast addresses or ones where the switch has not yet learned the correct port.

Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?

I'm actually NOT spoofing (from inside OPNSense VM) the MAC Address and in OPNSense VM the Interface is NOT set to Promiscuous Mode.

The MAC Spoofing was done one level higher up (in Proxmox VE Network Configuration).

And according to Proxmox Forums Posts, this Part of /etc/network/interfaces should disable Promiscous Mode in GNU/Linux:

Code: [Select]

        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096

>> Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?
Why do you say that ("environment that is clearly not designed for that") ?

What would be the "Proper" way to handle it otherwise ? Your vswitch Idea seems a bit complicated/overkill/somewhat expensive from what I need.

Would it be better to order an additional NIC + LAN Connection, do PCIe passthrough to OPNSense of one of the NICs (depending on which IP can get assigned the MAC Address, as discussed before), so basically keep a Dedicated NIC for Proxmox VE, and one Dedicated NIC (via PCIe Passthrough) for OPNSense VM ?

That at least could get around the Double-Firewalling Issue I guess ...

[Patrick M. Hausen]

I have dedicated OPNsense hosts at Hetzner and private LAN connections to the equally dedicated web server hosts. But that might be overkill in your case.

When you order an additional private network interface you cannot connect that to the Internet as far as I know. Only to other equally private infrastructure.

I see another "proper" way out of your dilemma - routed setup. We do this for all our web servers at Hetzner. I assume (I don't use Proxmox) that Proxmox can create an internal bridge not connected to any physical interface that can serve as the uplink for VMs? Correct?
I also assume that Proxmox acts as a router for both IPv4 and IPv6.

So on the external Proxmox interface:

- configure the single IPv4 address you ordered with your server
- configure the IPv4 default gateway
- configure one IPv6 address from that /64 you got with your server, but with a /128 prefix length.
  E.g. dead:beef:dead:beef::1/128
- configure the IPv6 default gateway: fe80::1

You should be able to ping the Proxmox host via IPv4 and IPv6 from outside.

Now you would need to order at least one additional IPv4 /29 from Hetzner to have IPv4 for your VMs.
Let's assume it's 8.16.32.64/29.

- assign the first (or last, or any ...) address to that internal bridge, e.g. 8.16.32.65/29
- you can now use 8.16.32.66-70 as IP addresses for VMs connected to that bridge. 8.16.32.65 would be their default gateway
- now assign another address from your /64 to that same bridge, but with the correct /64 prefix length, e.g. dead:beef:dead:beef::2/64. This is the default gateway for your VMs for IPv6.

The VMs connected to that bridge should be reachable on their public IPv4 and IPv6 addresses that way.

We have >20 hosts setup exactly in that fashion, only that our "VMs" are in fact FreeBSD jails and the hosts are not Proxmox but plain FreeBSD.

HTH,
Patrick

[Patrick M. Hausen]

This is what we run with OPNsense at Hetzner:
(proServer is our own managed hosting product)

Code: [Select]

Private LAN                                      Virtual Switch                                       
   ┏━━━━┓             ┏━━━━━━━━━━━━━━━━┓             ┌────┐                                           
   ┃    ┃             ┃    OPNsense    ┃             │    │                                           
   ┃    ┃     10G     ┃    (EX 44)     ┃     1G      │    │                                           
   ┃    ┣━━━━━━━━━━━━━┃                ┃─────────────┤    │                                           
   ┃    ┃             ┃     HAproxy    ┃             │    │                                           
   ┃    ┃             ┗━━━━━━━━━━━━━━━━┛             │    │                                           
   ┃    ┃                      ┃                     │    │                                           
   ┃    ┃                      ┃                     │    │       Uplink              Public Services 
   ┃    ┃              HA-Sync ┃ 1G                  │    │━━━━━━━━━━━━━━━━━━▶    ◀════════════════════
   ┃    ┃                      ┃                     │    │     Public IP                             
   ┃    ┃                      ┃                     │    │                                           
   ┃    ┃             ┏━━━━━━━━━━━━━━━━┓             │    │                                           
   ┃    ┃             ┃    OPNsense    ┃             │    │                                           
   ┃    ┃     10G     ┃    (EX 44)     ┃     1G      │    │                                           
   ┃    ┣━━━━━━━━━━━━━┃                ┃─────────────┤    │                                           
   ┃    ┃             ┃     HAproxy    ┃             │    │                                           
   ┃    ┃             ┗━━━━━━━━━━━━━━━━┛             └────┘                                           
   ┃    ┃                                                                                             
   ┃    ┃                                                                                             
   ┃    ┃                                                                                             
   ┃    ┃                                                                                             
   ┃    ┃                                                                                             
   ┃    ┃             ┏━━━━━━━━━━━━━━━━┓                                                               
   ┃    ┃     10G     ┃ proServer Host ┃                                                               
   ┃    ┣━━━━━━━━━━━━━┫    (AX 102)    ┃                                                               
   ┃    ┃             ┃                ┃                                                               
   ┃    ┃             ┃ ┌────┐         ┃                                                               
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃                                                               
   ┃    ┃             ┃ └────┘  Mgmt   ┃                                                               
   ┃    ┃             ┃ ┌────┐         ┃                Uplink                                         
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━▶                         
   ┃    ┃             ┃ └────┘  Mgmt   ┃           Management only                                     
   ┃    ┃             ┃ ┌────┐         ┃                                                               
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃                                                               
   ┃    ┃             ┃ └────┘  Mgmt   ┃                                                               
   ┃    ┃             ┃   ...          ┃                                                               
   ┃    ┃             ┃                ┃                                                               
   ┃    ┃             ┗━━━━━━━━━━━━━━━━┛                                                               
   ┃    ┃             ┏━━━━━━━━━━━━━━━━┓                                                               
   ┃    ┃     10G     ┃ proServer Host ┃                                                               
   ┃    ┣━━━━━━━━━━━━━┫    (AX 102)    ┃                                                               
   ┃    ┃             ┃                ┃                                                               
   ┃    ┃             ┃ ┌────┐         ┃                                                               
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃                                                               
   ┃    ┃             ┃ └────┘  Mgmt   ┃                                                               
   ┃    ┃             ┃ ┌────┐         ┃                Uplink                                         
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━▶                         
   ┃    ┃             ┃ └────┘  Mgmt   ┃           Management only                                     
   ┃    ┃             ┃ ┌────┐         ┃                                                               
   ┃    ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃                                                               
   ┃    ┃             ┃ └────┘  Mgmt   ┃                                                               
   ┃    ┃             ┃   ...          ┃                                                               
   ┃    ┃             ┃                ┃                                                               
   ┗━━━━┛             ┗━━━━━━━━━━━━━━━━┛                                                               
                              ...                                                                     


[Patrick M. Hausen]

Why do you say that ("environment that is clearly not designed for that") ?

Because Hetzner routes based on MAC address. For example they "throw" your entire /64 at the MAC address of your server. Similar for the first IPv4 and all subsequent additional subnets you might order. They don't allow multiple external links outside of simple dedicated MAC addresses for VMs (as you found out) or completely custom built solutions (different department, definitely expensive).

But then that configuration (just throw all destination IPs at that particular MAC) is why the setup with the single /128 externally and an internal bridge with the correct /64 works as supposed to as long as the host (Proxmox in your case) performs the routing.

[luckylinux]

I have dedicated OPNsense hosts at Hetzner and private LAN connections to the equally dedicated web server hosts. But that might be overkill in your case.

When you order an additional private network interface you cannot connect that to the Internet as far as I know. Only to other equally private infrastructure.

I see another "proper" way out of your dilemma - routed setup. We do this for all our web servers at Hetzner. I assume (I don't use Proxmox) that Proxmox can create an internal bridge not connected to any physical interface that can serve as the uplink for VMs? Correct?
I also assume that Proxmox acts as a router for both IPv4 and IPv6.

So on the external Proxmox interface:

- configure the single IPv4 address you ordered with your server
- configure the IPv4 default gateway
- configure one IPv6 address from that /64 you got with your server, but with a /128 prefix length.
  E.g. dead:beef:dead:beef::1/128
- configure the IPv6 default gateway: fe80::1

You should be able to ping the Proxmox host via IPv4 and IPv6 from outside.

Now you would need to order at least one additional IPv4 /29 from Hetzner to have IPv4 for your VMs.
Let's assume it's 8.16.32.64/29.

- assign the first (or last, or any ...) address to that internal bridge, e.g. 8.16.32.65/29
- you can now use 8.16.32.66-70 as IP addresses for VMs connected to that bridge. 8.16.32.65 would be their default gateway
- now assign another address from your /64 to that same bridge, but with the correct /64 prefix length, e.g. dead:beef:dead:beef::2/64. This is the default gateway for your VMs for IPv6.

The VMs connected to that bridge should be reachable on their public IPv4 and IPv6 addresses that way.

We have >20 hosts setup exactly in that fashion, only that our "VMs" are in fact FreeBSD jails and the hosts are not Proxmox but plain FreeBSD.

HTH,
Patrick

Uhm ... I really wanted to avoid /29 Subnet if possible (quite expensive Setup Fee and I would "lose" one IP for the Proxmox VE Host/Bridge).

Too bad they don't allow another WAN Connection (I guess this is what they call "Uplink").

I see they however offer "5-port 1 Gbit switch", but I'm not sure they would allow this on the Uplink side. I can try to ask them though ...

Right now stuff is "kinda" working (with like 2 out of 3 TCP/UDP/ICMP Traceroutes working, but NOT All 3, both for IPv4 and IPv6), although we can agree it's a bit of a weird "fashion" with:
- IPv4 in Bridge Mode (IPv4 Forwarding DISABLED since it's not currently needed)
- IPv6 in Brouter (Bridge+Router) Mode and IPv6 forwarding ENABLED

(I just discovered some stuff on Ubuntu where /usr/bin/traceroute pointed to a different location than what I have on Proxmox VE -> Different traceroute / traceroute6 executable, different behavior, different results)

The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down (it broke down in the middle of an update a few days ago, I had to reinstall all packages from the Proxmox VE noVNC Console / Web Console for the VM), then I cannot repair anything anymore (besides going in Hetzner Rescue Console, disable PCIe Passthrough, then go back to the old setup [the one I currently have], fix OPNSense, then switch back PCIe passthrough and reboot again).

But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?

[Patrick M. Hausen]

But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?

Correct. 5 possible VMs with public IPv4, if you order a /29.

The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down ...

At least you can get a remote KVM for up to three hours for free if available. Works pretty well. I don't know if you can fix a broken Proxmox/OPNsense setup that way, but I would guess so.

[luckylinux]

But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?

Correct. 5 possible VMs with public IPv4, if you order a /29.

The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down ...

At least you can get a remote KVM for up to three hours for free if available. Works pretty well. I don't know if you can fix a broken Proxmox/OPNsense setup that way, but I would guess so.

Or double NAT with iptables & masquerading when using a single additional IPv4 Address in Routed Setup I guess (Proxmox VE Routes the /32 single IPv4 Address to say vmbr2 which is say 192.168.100.1, and then OPNSense VM takes it from there as an additional Interface).

EDIT 1: To be honest, for all the times I got locked out of the Server, I was ALWAYS (knock on wood ) able to Recover it from the Rescue Console. Just needed to git clone my set of Scripts for Linux Setup / Recovery (https://github.com/luckylinux/linux-setup), copy a Config File a have at hand at Home, then run the mounting and access the Proxmox VE Host filesystem to fix then Configuration .

[Patrick M. Hausen]

Now you are leaving my area of expertise because I have no idea how these vmbr things work exactly.

I always aim for a clean network setup even if more expensive, but then I work in a professional/enterprise context. We just order all hosts with a /29 by default and calculate that in.

[luckylinux]

Now you are leaving my area of expertise because I have no idea how these vmbr things work exactly.

I always aim for a clean network setup even if more expensive, but then I work in a professional/enterprise context. We just order all hosts with a /29 by default and calculate that in.

Fair enough, thanks for the tips .

Right now it's definitively NOT clean but I'd say it's working. UDP Traceroute needed a "Reject" Rule at the last hop on OPNSense to make it work. Still stumbled by the inconsistency on the TCP Implementation that OPNSense uses (traceroute -T flag actually uses ICMP ... duh).

I contacted Hetzner and they forwarded my Ticket to another Department. Let's see if they come back with something positive & not too expensive. I'm just a Homelabber after all  .

EDIT: Nope, not easy and not cheap . Either separate MAC (which I am already using with all the issues that come with it), or vswitch (but that's more expensive, traffic is limited, etc).

Maybe I'll just have to keep the "weird" setup as it is now. Plus maybe some fixes as you suggested @Patrick with Regards to IPv6.

[whenthelight]

Hey got same Problem with 3 IP's and Same Gateway, got ist solved.

You need seperate MAC for every IP and set it to the seperate Interface in Proxmox VM Conf for Opnsense.
Interfaces conf file add to Main vmbr0 with Main-IP
        up ip route add WAN2-IP dev vmbr0
        up ip route add WAN3-IP dev vmbr0

In Opnsense create Interfaces and set IP4 on DHCP then you are able to get same Gateway for Multiple Wan Interfaces. What a Heck :-)

Try and Error multiple Times and a lot of Beer :-)

Hope it helps you.

[luckylinux]

Hey got same Problem with 3 IP's and Same Gateway, got ist solved.

You need seperate MAC for every IP and set it to the seperate Interface in Proxmox VM Conf for Opnsense.
Interfaces conf file add to Main vmbr0 with Main-IP
        up ip route add WAN2-IP dev vmbr0
        up ip route add WAN3-IP dev vmbr0

In Opnsense create Interfaces and set IP4 on DHCP then you are able to get same Gateway for Multiple Wan Interfaces. What a Heck :-)

Try and Error multiple Times and a lot of Beer :-)

Hope it helps you.

Thanks for the Reply. I'm not fully sure this is correct either though.

It seems you are using a Bridge+Router Configuration (Brouter), and on top of that you are using the same vmbr0 so you are not routing between interfaces. You are using a Bridge (with Separate MAC for OPNSense) linked to the Physical Interface (eth0 in my case), which is different than my "pure" Bridged IPv4 Configuration. Although, as said before, I am using it in "Reverse" now (OPNSense spoofing the MAC Address of the Ethernet Interface, Proxmox VE Bridge using one of the additional MACs). But, if you are at Hetzner, you need to beware of the MAC Abuse Emails as well in your case, since in the end it's still (mostly) a Bridged Configuration.

It seems a bit what I was doing with the IPv6 /64 Subnet originally, where I was using the IPv6 IP from vmbr0 in OPNSense, but I am also not sure that was correct.

Routing within the same Interface (vmbr0) or Separate Interfaces (vmbr0 <-> vmbr1) ? Not sure ... For IPv4 and related MAC Abuse Emails I think you are not out of the Woods yet  .

This is my updated /etc/network/interfaces

Code: [Select]

auto lo
iface lo inet loopback

#auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        hwaddress XX:XX:XX:XX:XX:XX
        address XXX.XXX.XXX.proxmox
    netmask 255.255.255.192
    gateway XXX.XXX.XXX.hetznergateway
    bridge-ports eth0
    bridge-stp off
        bridge_waitport 0
    bridge-fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        pre-up ip addr flush dev eth0
        post-up ip addr flush dev eth0


iface vmbr0 inet6 static
        hwaddress XX:XX:XX:XX:XX:XX
        # First Address of the Main /64 Subnet
        address 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001
        netmask 128
        gateway fe80::1
        bridge-mcsnoop no
        bridge-stp off
        bridge_waitport 0
        bridge-fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        pre-up ip addr flush dev eth0
        post-up ip addr flush dev eth0
        up ip -6 route add fe80::1 dev vmbr0
        up ip -6 route add default via fe80::1 dev vmbr0
        down ip -6 route del default via fe80::1 dev vmbr0
        down ip -6 route del fe80::1 dev vmbr0


auto vmbr1
iface vmbr1 inet static
        address 0.0.0.0
        netmask 255.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096

iface vmbr1 inet6 static
# First Address of the separate :0001 /80 Subnet of the Main /64 Subnet
        address 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
        netmask 80
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0
        bridge-mcsnoop no
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        up ip -6 route add 2a01:XXXX:XXXX:XXXX:0000::1/128 via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 0
        down ip -6 route del 2a01:XXXX:XXXX:XXXX:0000::1/128 via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 0
        up ip -6 route add 2a01:XXXX:XXXX:XXXX:0001::/80 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001 dev vmbr1 metric 256
        down ip -6 route del 2a01:XXXX:XXXX:XXXX:0001::/80 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001 dev vmbr1 metric 256
        up ip -6 route add 2a01:XXXX:XXXX:XXXX::/64 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0010 dev vmbr1 metric 500
        down ip -6 route del 2a01:XXXX:XXXX:XXXX::/64 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0010 dev vmbr1 metric 500
        up ip -6 route add default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
        down ip -6 route del default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800


auto vmbr2
iface vmbr2 inet static
        address 0.0.0.0
        netmask 255.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096

iface vmbr2 inet6 static
        # First Address of the separate :0000 /64 Subnet of the Additional /56 Subnet
        address 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001
        netmask 64
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0
        bridge-mcsnoop no
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        up ip -6 route add 2a01:XXXX:XXXX:YYYY::/64 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001 dev vmbr2 metric 256
        down ip -6 route del 2a01:XXXX:XXXX:YYYY::/64 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001 dev vmbr2 metric 256
        up ip -6 route add 2a01:XXXX:XXXX:YYYY::/56 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0010 dev vmbr2 metric 500
        down ip -6 route del 2a01:XXXX:XXXX:YYYY::/56 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0010 dev vmbr2 metric 500
        up ip -6 route add default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
        down ip -6 route del default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800

I ordered an additional /56 IPv6 Subnet, so right now I am using:
- Bridging for IPv4 (net0 / vtnet0 on OPNSense is WAN_IPv4)
- Routing for IPv6 (net2 /vtnet2 on OPNSense is WAN_IPV6_SUBNET_56 and net3 / vtnet3 on OPNSense is WAN_IPV6_SUBNET_64)

However, strangely enough, the WAN_IPV6_SUBNET_64 stubbornly refuses to work in OPNSense, the Gateway can be pinged but doesn't lead to "outside access" (e.g Google DNS Servers). I am using /80 for routing Traffic (which should be fine without SLAAC and Static IPv6 only), but for some reason OPNSense -> Google IPv6 DNS Servers is not working.

You can argue I'm being a bit nitpicky, as with a /56 IPv6 Subnet I should have 256 /64 Networks .

But still, kinda weird that it does not work.

[luckylinux]

Actually ... in OPNSense VM both the /64 and the Additional /56 IPv6 Subnet I purchased work correctly, it's just that one Gateway appears down / unpingable (typically /64 Gateway appears down), if the other Gateway is used instead (I guess OPNSense cannot have 2 "default" Gateways, which is fair).

Although it's weird that they don't allow to monitor an External IP when the Gateway isn't being used  .

Is this a BUG or a "Feature" ?

I lost a couple of Days trying to understand why I cannot monitor 2 different Google DNS Servers ...

Turns out you need to [UNCHECK] Gateways -> <your Gateway> -> "Disable Host Route", so that traffic to that Monitor IP is forced out through the Specified Gateway. Of course this applies to ALL Traffic (even if you do a "manual" Traceroute etc), but unless you do that, the Gateway will appear down (most likely because the traffic will just go through the other Gateway).