Cloned opnsense VM, no internet access

[leonidas-o]

Hello guys,

I got a very strange behavior. I'm running opnsense OPNsense 22.7.9_3 on a VM, inside a proxmox cluster. The proxmox cluster had two nodes, I added a third one. Each node has an opnsense VM, so I backup'ed the opnsense VM on node1 and restored it on the new third proxmox node (deleted in Proxmox the networks devices MAC addresses so new ones get generated). It has no HA configuration, not for Proxmox and not for opnsense. So they all work independent from each other.
I also got a separate wireguard VM on each of my proxmox nodes and the network/interfaces file is the same except the IP addresses of course.

Code: [Select]

iface enp0s31f6 inet static
  address MYPUBLICIP
  ...
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2

...

auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
# OPNSense WAN - Proxmox LAN

auto vmbr11
iface vmbr11 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
# VM Net
vmbr10 is the WAN interface and vmbr11 the LAN interface for opnsense VM.

So I tried the following:
- I've started the third opnsense vm, updated the IPv6 addresses inside the UI, IPv4 matched already as well as the interface assignments.
- I've reset to factory default and copied a config.xml with the corrected IPv6 addresses from proxmox's opnsense vm node 1 over to the third one.

The result is always the same. I don't have internet access from within LAN.
- I can access the Host server where proxmox is running on via ssh and ping whatever I want, that works.
- I can connect over the internet to the wireguard VM which is running in the opnsense LAN, I can then ping all the VMs inside LAN network.
- I can enter the opnsense web UI (Interfaces, Diagnostic, Ping) and ping the proxmox HOST's public IP (from WAN or LAN interfaces)
  - However I can NOT ping e.g. 1.1.1.1 (from WAN or LAN interfaces) -> 3 packets transmitted, 0 packets received, 100.0% packet loss.

I remember I had the same issue when I tried to clone the VM from Proxmox node1 to proxmox node2. I ended up configuring opnsense from scratch after a factory reset. Recreating all rules, port-forwardings etc. by hand.
Is there some kind of cache or anything deep buried in the system, which could be responsible for causing such behavior when you clone the vm?
It doesn't make sense to me, I cloned the VM, I adapted the rules to match the public IPv4 address of node 3 and updated also the ipv6 addresses for the interface wan gateway etc. I can also connect to wireguard from outside and ping within LAN but I cannot go out. What am I missing?

[leonidas-o]

Okay after I tried with opnsense 23.1.8, reset to factory defaults, even with a completely new installation and all of these approaches were showing the same behaviour, I knew it must be something else.

Fortunately it is documented in the proxmox docs (I simply forgot about it). I was missing the following entries in my network/interfaces file:

Code: [Select]

  post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
  post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

On my first proxmox node it was working without these lines but the third node needed it. I think that could be because  the third proxmox node is running on a slightly newer version, so updating the first node would maybe cause the same issue. Nevertheless it is working now. I don't want to touch the VM again, so still don't know for sure if I can clone the VM, as the latest setup was a fresh installation.