Hello,
I have been able to get a Linux VM running with vm-bhyve on OPNsense, but I'm having trouble getting a stable network connection to a VLAN.
What I see is when the VM boots up the firewall filters the VM's tap interface and deny entries against the tap interface show in the logs. After roughly 18-20 minutes, deny logs stop and packets flow through the VLAN rule set. I setup a script that makes DNS requests until it's successful, logs how long it takes, and reboots the VM. Out of ~35 iterations, most times DNS request succeed in the 18-20 minute range.
I used `vm switch create` to make a switch on a vlan interface `ix2_vlan66`. This creates a bridge interface, which I assigned in the UI, and an entry for it became available in the firewall rules section. In the firewall I create a single rule that allows everything. From what I can tell, that rule is used to allow DHCP to work; all other packets appear to be processed with the VLAN's rules.
Per instructions on "How to set up a LAN Bridge" [1] I set `net.link.bridge.pfil_member` to 0 and net.link.bridge.pfil_bridge to 1.
Any thoughts on what might be causing this? Ideally the correct firewall rules would be applied to the VM immediately without the delay. It seems like `net.link.bridge.pfil_member` and `net.link.bridge.pfil_bridge` tunables take effect on the tap interface/new members to the bridge after ~18 minutes.
Thank you!
[1] https://docs.opnsense.org/manual/how-tos/lan_bridge.html
I've been running my script to reboot the VM once a DNS request succeeds for the past few days. In general, across ~150 reboots, it takes about 20 minutes for packets to pass through the firewall. I've seen a couple 0 minute outliers, one 104 minute outlier, and seldomly it takes less than 5 minutes.
I don't really see a pattern around the timestamps to assume there's some cron job running that sorts things out.
Here's a sample of my script's output:
Oct 24 11:30:12 opnsense-pihole check-dns.sh[574]: worked! duration=20 minutes
Oct 24 11:52:40 opnsense-pihole check-dns.sh[574]: worked! duration=20 minutes
Oct 24 12:14:48 opnsense-pihole check-dns.sh[574]: worked! duration=19 minutes
Oct 24 12:36:36 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 12:54:52 opnsense-pihole check-dns.sh[574]: worked! duration=15 minutes
Oct 24 13:17:25 opnsense-pihole check-dns.sh[575]: worked! duration=20 minutes
Oct 24 13:37:43 opnsense-pihole check-dns.sh[575]: worked! duration=17 minutes
Oct 24 14:00:10 opnsense-pihole check-dns.sh[576]: worked! duration=20 minutes
Oct 24 14:03:14 opnsense-pihole check-dns.sh[575]: worked! duration=0 minutes
Oct 24 14:25:21 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 14:46:59 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 15:03:10 opnsense-pihole check-dns.sh[574]: worked! duration=13 minutes
Oct 24 16:50:33 opnsense-pihole check-dns.sh[575]: worked! duration=104 minutes
Oct 24 17:04:38 opnsense-pihole check-dns.sh[574]: worked! duration=11 minutes
Oct 24 17:26:51 opnsense-pihole check-dns.sh[574]: worked! duration=19 minutes
Oct 24 17:44:27 opnsense-pihole check-dns.sh[577]: worked! duration=15 minutes