kvm+opnsense+Vlan problem - must manually reload firewall rules after reboot.

[dieterarn]

Hi, I'm having a bizarre problem with my setup:

I've got several VLANs segregating things. These are setup on my hypervisor (KVM/PROXMOX) as Linux bridges associated to specific VLANs.  i like making all the VLANs separate bridges because i can control what other virtual machines on the host can connect to.Obviously OPNSENSE is running as a virtual machine.

I had some weird problems: any thing that was connected to "base lan" worked ok straight after reboot but anything bridged via a configured VLAN wouldn't. through trail and error I discovered that if i touched a firewall rule and caused a rule reload everything would start to work.  :o

Has anyone here tried to setup VLANs and OPNSENSE in visualized environments? Is there a way to at least automate my workaround and cause a rule reload after everything is up and running?

[dieterarn]

i've reproduced the problem a half dozen more times. i've also tired running:

pfctl -F all -f /etc/pf.conf

to flush the firewall rules and reload but it errors out:

Code Select Expand

root@OPNsense:~ # pfctl -F all -f /etc/pf.conf
rules cleared
nat cleared
5 tables deleted.
140 states cleared
source tracking entries cleared
pf: statistics cleared
pf: interface flags reset
pfctl: /etc/pf.conf: No such file or directory
pfctl: cannot open the main config file!: No such file or directory
pfctl: Syntax error in config file: pf rules not loaded


i was going to add :
@reboot echo /sbin/pfctl -F all -f /etc/pf.conf | at now + 5 minutes

but i don't think that will work. However as soon as i reset using option "11" everything starts to work ( but ssh session breaks - oh well).

Code Select Expand


*** OPNsense.ad.grassyshallows.com: OPNsense 18.7.8 (amd64/OpenSSL) ***

LAN (em1)       -> v4: 192.168.1.1/24 ... bridged from a physical interface
WAN (em0)       -> v4/DHCP4: 107.190.35.80/26 ... bridged from a physical interface
cameras (em5)   -> xxxx ... currently unused but a vlan bridge from the bridged physical interface LAN is on
carp (em3)      -> xxxx ... currently unused but a vlan bridge from the bridged physical interface LAN is on
management (em2) -> xxxx .. currently unused but a vlan bridge from the bridged physical interface LAN is on
webServices (em6) -> xxxx .. currently unused but a vlan bridge from the bridged physical interface LAN is on
wifi (em4)      -> v4: 192.168.4.1/24 ... active and the interface with problems ,  a vlan bridge from the bridged physical interface LAN is on...

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 11

Writing firmware setting...done.
Configuring login behaviour...done.
Configuring CRON...done.
Setting timezone...done.
Setting hostname: OPNsense.ad.grassyshallows.com
Generating /etc/hosts...done.
Generating /etc/resolv.conf...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring VLAN interfaces...done.
Configuring WAN interface...done.
Configuring LAN interface...done.
Configuring cameras interface...done.
Configuring carp interface...done.
Configuring management interface...done.
Configuring webServices interface...done.
Configuring wifi interface...done.
Setting up routes...done.
Configuring firewall.......done.
Starting DHCPv4 service...done.
Starting DHCPv6 service...done.
Starting router advertisement service...done.
packet_write_wait: Connection to 192.168.1.1 port 22: Broken pipe


I imagine option "11" is a script? can i call it from cron and do my hacky workaround? if so where does it live? is this a system bug?

[franco]

/etc/pf.conf -> /tmp/rules.debug