Messages

Hi, I didn't install the k8s cluster yet,  It has not effect, it is just a name of the subnet/interface/bridge. I'm testing from two test VM's on each side. The tcpdump's were also quite clear, return traffic arrived on vtnet2, but didn't go out back to vtnet1. Same behavior if I turned of the firewall with pfctl. Routing was as shown in my original post, very basic. Fyi, I used tcpdump -i to specify the interface each time, had multiple tmux panes open for each interface on the opnsense, but also on each VM, so I could see the packets arrive and leave on each hop.

The firewall is a VM (running for almost a year), I turned it off and deployed a new VM and installed a fresh copy of OPNsense, recreated the config, same IP's and had no issues. I did upgrade the problematic one to 24.7 from 24.1 last night, to see if that helped, but it didn't.

For now I have to focus on a project that relies on this firewall to work, I will get back to troubleshooting the original one as I can't stand not knowing what the problem is :D

Thanks for commenting. I'll post back here if I figure out what was wrong.

[EDIT:]

So I have two LAN's, one called LAN (vtnet1 interface), one called K8S (vtnet2 interface). Hosts on any of these networks should be able to reach echt other. This worked a month ago, but I must have changed something.

I have drawn the issue in the included image. Basically I can send traffic from my host in LAN to another in K8S arrives there (proven with tcpdump), the reply traffic also arrives back on OPNsense, but then disappears, it doesn't come out of the LAN vtnet1 interface (also proven with tcpdump on both interfaces)



I only have one firewall rule on each interface: From: that_network, any, any allow. Besides the default rules that is.

If I tail -f /var/log/filter/latests.log I don't see any blocks (or check in the GUI).

Maybe it is because I'm sick at the moment, but I spend 3 hours on this and I feel like this should be simple :(

EDIT: FYI, if I disable the firewall I have the same behavior. Which makes me think this is a weird routing problem. But routing is very simple. I have no static routes, both subnets are connected directly, there is only one gateway, used by the WAN interface, pointing to the internet. Everything else points to the opnsense as a default gateway. This is the routing table of opnsense:

Code Select Expand

Internet:
Destination         Gateway            Flags     Netif Expire
default               10.0.0.1         UGS      vtnet0
10.0.0.0/24        link#1             U        vtnet0
10.0.0.2             link#7             UHS         lo0
10.0.1.0/24        link#2             U        vtnet1
10.0.1.1             link#7             UHS         lo0
192.168.0.0/24   link#3             U        vtnet2
192.168.0.1        link#7             UHS         lo0
127.0.0.1          link#7             UH          lo0


I had found your project, but dismissed it on account of it being Vagrant. Ill look into the config.xml part. Thank you.

Hi all.

So I need to deploy OPNSense on Proxmox and I need to do it in multiple environments. I also need to be able to (relatively) quickly breakdown and rebuilt each environment. FYI, OPNsense/HAproxy will act as an entry point to a  Kubernetes cluster behind it (on the same Proxmox cluster), it is quite a critical component.

I've got the whole thing automated using Ansible and Terraform (setup of Proxmox, deployment of VM's, install of Kubernetes, all the apps/containers, etc).

The only thing left is OPNsense, which can only be setup manually via the console. I did some searching, looked at FreeBSD cloud-init support, which is not great yet. So I kind of gave up on that for now, but I would be happy if at least I can boot an OPNsense image which would read some kind of setup file (like docker compose, cloud-init, helm chart, etc) to get an IP address and a SSH public key so I can reach it over the network and use Ansible or even Expect to do the rest. But unfortunately I can't figure out if this is possible. Any help/ideas is/are appreciated.