Firewall is unalbe to reach other networks

[kerbaras]

I just finish setting up a network using opnsense as the main firewall. I'm runing latest 24.1.

For some reason at some point OPNSense stop reaching all networks. But all the hosts in the networks can still reach eachother, the internet and OPNSense. All firewall rules are open and i've reinstalled it 5 times from scratch.

Also Tried disabling NAT Transaltion and still get the same result (Just the hosts are not able to reach the internet)

A bit about the topology:

I have a server hosted in hetzner with 1 Public IPv4 and a /64 IPv6 that is hosting a proxmox installetion
- WAN: 172.31.225.0/24
- LAN: 172.16.0.0/24
- vLAN.101: 172.16.101.0/24

The proxmox server gets the Public IP and forwards all the pakets to the OPNSense gateway (172.31.225.1) and does NAT masquerade from (172.31.225.0/24) via the Public IP.

OPNSense has authority over the other networks and also is doing natting.
- Gateway: 172.16.0.1
- Proxmox: 172.16.0.2

EVERY HOST IN THE NETWORK HAS ACCESS TO THE INTERNET. All can ping 172.31.225.1 (Firewall) and get a response. However, OPNSense does not have access to the internet nor can it ping local and remote hosts. I cannot install updates via console nor reach the internet with curl.

I have connectivity to the WebInterface via public IPv6 and i can also reach it from one of the internal machines that is running a tunnel to my home network.

I have no idea what is happening. Right now i have all firewall rules open in pass and still i'm not able to see it working

Code: [Select]

$ ssh root@127.16.0.1
*** horizon.cloudlab: OPNsense 24.1.9_4 ***

 VPC (vtnet0)    -> v4: 172.16.0.1/24
                    v6: xxxx:xxx:xx:xxx:8000::2/68
 VPN (vtnet2)    -> v4: 172.16.101.1/24
 WAN (vtnet1)    -> v4: 172.31.255.1/24
                    v6: xxxx:xxx:xx:xxx:4000::2/66


Code: [Select]

root@horizon:~ # netstat -nr4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.31.255.2       UGS      vtnet1
127.0.0.1          link#5             UH          lo0
172.16.0.0/24      link#1             U        vtnet0
172.16.0.1         link#1             UHS         lo0
172.16.101.0/24    link#3             U        vtnet2
172.16.101.1       link#3             UHS         lo0
172.31.255.0/24    link#2             U        vtnet1
172.31.255.1       link#2             UHS         lo0

No internet access

Code: [Select]

opnsense@172.31.255.1: ~ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

Pinging firewall from firewall!!!!

Code: [Select]

opnsense@172.31.255.1: ~ # ping 127.31.225.1
PING 127.31.225.1 (127.31.225.1): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
^C
--- 127.31.225.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Pinging Proxmox from firewall

Code: [Select]

opnsense@172.31.255.1: ~ # ping 127.31.225.2
PING 127.31.225.2 (127.31.225.2): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
^C
--- 127.31.225.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

From the Proxmox Host:

Code: [Select]

proxmox@172.31.255.2: ~ # ip route
default via {PUBLIC_IP} dev enp41s0 proto kernel onlink
{PUBLIC_IP}/26 via {PUBLIC_GW} dev enp41s0
{PUBLIC_IP}/26 dev enp41s0 proto kernel scope link src {PUBLIC_IP}
172.16.0.0/24 dev vmbr1 proto kernel scope link src 172.16.0.2
172.16.101.0/24 dev vmbr0.101 proto kernel scope link src 172.16.101.2
172.31.255.0/24 dev vmbr0 proto kernel scope link src 172.31.255.2

I can reach the internet:

Code: [Select]

proxmox@172.31.255.2: ~ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=17.1 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=17.2 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 17.125/17.144/17.163/0.019 ms

I can ping OPNSense from proxmox and get a response:

Code: [Select]

proxmox@172.31.255.2: ~ # ping 127.31.255.1
PING 127.31.255.1 (127.31.255.1) 56(84) bytes of data.
64 bytes from 127.31.255.1: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from 127.31.255.1: icmp_seq=2 ttl=64 time=0.017 ms
^C
--- 127.31.255.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1029ms
rtt min/avg/max/mdev = 0.017/0.021/0.025/0.004 ms

From a machine in the vlan:

Code: [Select]

vm@172.16.101.10 ~ $ ip route
default via 172.16.101.1 dev eth0  metric 202
172.16.101.0/24 dev eth0 scope link  src 172.16.101.10


Code: [Select]

vm@172.16.101.10 ~ $ ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: seq=0 ttl=42 time=0.116 ms
64 bytes from 172.16.0.1: seq=1 ttl=42 time=0.130 ms
^C
--- 172.16.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.116/0.123/0.130 ms


Code: [Select]

vm@172.16.101.10 ~ $ ping 172.31.255.1
PING 172.31.255.1 (172.31.255.1): 56 data bytes
64 bytes from 172.31.255.1: seq=0 ttl=42 time=0.109 ms
64 bytes from 172.31.255.1: seq=1 ttl=42 time=0.103 ms
^C
--- 172.31.255.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.103/0.106/0.109 ms

and i can reach the internet:

Code: [Select]

vm@172.16.101.10 ~ $ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=42 time=17.336 ms
64 bytes from 1.1.1.1: seq=1 ttl=42 time=17.287 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 17.287/17.311/17.336 ms

From the Web Interface i cannot run ANY diagnostic tools and i'm not able to update nor install plugins as well.

What i've noticed is that if i leave the update command long-running, esporadically i do get a hit and a couple of connections come through. (But i was not able to replicate this). Also the setup was working at some point as i was able to update packages

I would appriciate some help as i've tried a lot of things and i'm still don't know what is happening.