Bhyve VM temporary firewall blocking on VM boot

[rreboto]

Hello,

I have been able to get a Linux VM running with vm-bhyve on OPNsense, but I’m having trouble getting a stable network connection to a VLAN.

What I see is when the VM boots up the firewall filters the VM's tap interface and deny entries against the tap interface show in the logs. After roughly 18-20 minutes, deny logs stop and packets flow through the VLAN rule set.  I setup a script that makes DNS requests until it’s successful, logs how long it takes, and reboots the VM.  Out of ~35 iterations, most times DNS request succeed in the 18-20 minute range.

I used `vm switch create` to make a switch on a vlan interface `ix2_vlan66`.  This creates a bridge interface, which I assigned in the UI, and an entry for it became available in the firewall rules section.  In the firewall I create a single rule that allows everything.  From what I can tell, that rule is used to allow DHCP to work; all other packets appear to be processed with the VLAN's rules.

Per instructions on “How to set up a LAN Bridge” [1] I set `net.link.bridge.pfil_member` to 0 and net.link.bridge.pfil_bridge to 1.

Any thoughts on what might be causing this?  Ideally the correct firewall rules would be applied to the VM immediately without the delay.  It seems like `net.link.bridge.pfil_member` and `net.link.bridge.pfil_bridge` tunables take effect on the tap interface/new members to the bridge after ~18 minutes.

Thank you!

[1] https://docs.opnsense.org/manual/how-tos/lan_bridge.html

[rreboto]

I've been running my script to reboot the VM once a DNS request succeeds for the past few days.  In general, across ~150 reboots, it takes about 20 minutes for packets to pass through the firewall.  I've seen a couple 0 minute outliers, one 104 minute outlier, and seldomly it takes less than 5 minutes.

I don't really see a pattern around the timestamps to assume there's some cron job running that sorts things out.

Here's a sample of my script's output:

Code: [Select]

Oct 24 11:30:12 opnsense-pihole check-dns.sh[574]: worked! duration=20 minutes
Oct 24 11:52:40 opnsense-pihole check-dns.sh[574]: worked! duration=20 minutes
Oct 24 12:14:48 opnsense-pihole check-dns.sh[574]: worked! duration=19 minutes
Oct 24 12:36:36 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 12:54:52 opnsense-pihole check-dns.sh[574]: worked! duration=15 minutes
Oct 24 13:17:25 opnsense-pihole check-dns.sh[575]: worked! duration=20 minutes
Oct 24 13:37:43 opnsense-pihole check-dns.sh[575]: worked! duration=17 minutes
Oct 24 14:00:10 opnsense-pihole check-dns.sh[576]: worked! duration=20 minutes
Oct 24 14:03:14 opnsense-pihole check-dns.sh[575]: worked! duration=0 minutes
Oct 24 14:25:21 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 14:46:59 opnsense-pihole check-dns.sh[575]: worked! duration=19 minutes
Oct 24 15:03:10 opnsense-pihole check-dns.sh[574]: worked! duration=13 minutes
Oct 24 16:50:33 opnsense-pihole check-dns.sh[575]: worked! duration=104 minutes
Oct 24 17:04:38 opnsense-pihole check-dns.sh[574]: worked! duration=11 minutes
Oct 24 17:26:51 opnsense-pihole check-dns.sh[574]: worked! duration=19 minutes
Oct 24 17:44:27 opnsense-pihole check-dns.sh[577]: worked! duration=15 minutes