Repeated "Ethernet detached" events / Flapping caused by Sensei Zenarmor?

[BNaCl]

I have been having a problem with repeated ethernet detached events which seem to cause very brief interface flapping.

I have narrowed this down to Sensei/ZA as it does not happen when in bypass, but I cannot figure out how to fix and wondering if anyone can add any insight. It also should be noted that I am running in a somewhat unique config using their Bridge Mode which is labeled as "experimental" so there's that. However, I would think running a NGFW in a transparent filtering bridge config isn't unusual and should work.

Here is my setup:

• Protectli appliance with Intel(R) Core(TM) i3-7100U CPU, 8 GB RAM, Intel NICs.
• Two interfaces enabled in OPNs and configured as a transparent filtering bridge in OPNs as L2 bridge (EM1/LAN and EM0/WAN shown in logs), no L3 IP assigned.
• Third MGT interface assigned (this interface isn't protected by Sensei and doesn't have the issue)
• CRC/TSO/LRO disabled on all interfaces
• Firewall disabled as I don't need any of those features and want to remove unnecessary variables
• Sensei configured to protect the bridge (not the MGT interface)

Code Select Expand

2022-12-20T10:13:02-05:00 Error opnsense /usr/local/etc/rc.newwanip: Failed to detect IP for WAN[wan]
2022-12-20T10:13:02-05:00 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em0'
2022-12-20T10:13:02-05:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static wan(em0)
2022-12-20T10:13:01-05:00 Error opnsense /usr/local/etc/rc.newwanip: Failed to detect IP for LAN[lan]
2022-12-20T10:13:01-05:00 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1'
2022-12-20T10:13:01-05:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static lan(em1)
2022-12-20T10:12:59-05:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static wan(em0)
2022-12-20T10:12:58-05:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static lan(em1)

Thanks in advance!

[lilsense]

why is WAN bridged, or your other interface. That's problem. No need to bridge any of the interfaces.

[BNaCl]

I can see why my post was a bit misleading. The WAN and LAN are just labels on eth0 and 1. This is configured as a transparent filtering bridge between the WAN (handled by another edge FW/router) and the LAN.

[sy]

Hi,

Zenarmor uses netmap which is an Operating System subsystem to grab packets off the wire. Netmap is not fully compatible with bridge interfaces yet. Please remove bridge configuration on OPNsense and configure Zenarmor as bridge.

[BNaCl]

Ah OK makes sense. I will remove the bridge in OPNs. Should the interfaces that will be part of the Zenarmor bridge be enabled or disabled in OPNs?

[BNaCl]

Welp I can't get ZA to pass the traffic unless the bridge is also configured in OPNs. I have tried with the interfaces both enabled and disabled in OPNs. I know this worked in this config before and I am pretty sure I ran into this after the last update to 22.7.9_3 and that is why I had to create the bridge in OPNs. I am not sure what ver I was on before I updated, might have fallen behind a few revs, but not much. My theory is the ZA bridge functionality broke somewhere along the way. Another item of note is that the interface shows as em1 on the ZA status page as opposed to the configured bridge br0 which seems odd. 

Hope to get this sorted, really enjoy ZA and hope to use it, but I need the transparent bridging to work.

[mb]

Hi @BNaCI,

Currently, the way ZA L2 bridge works is it actually disconnects them from the OS, and does the bridging itself switching packets back and forth between ZA-configured bridge interfaces (it is independent of OS bridge).

The drawback is, when you don't have the ZA packet engine running, you don't have the bridge. In some of our deployments, we also utilize Silicom Bypass adapters for that. With them, you always have a hardware-assisted bridge even when you have your server hardware powered down.

To simplify things, together with OPNsense team, we're bringing netmap support to if_bridge(4). So basically, you'll be able to run zenarmor or Suricata IPS on an OS bridge interface.

WRT an ETA, we hope to provide a testable kernel in February '23.

[BNaCl]

Hi @mb, appreciate your response. I understand what you are describing, and can confirm it previously worked  as you detailed with bridge ONLY configured in ZA (not in OPNs). But it doesn't work this way now. I literally spent an entire afternoon troubleshooting after a recent OPNs upgrade because it wouldn't pass traffic in this config until I added the bridge in OPNs. My troubleshooting indicates something changed in one of the recent OPNs updates that affected the ZA behavior such that it will not pass traffic in a bridge config without it also configured in OPNs.

I can confirm that I have ZA running with the bridge configured and it will not pass traffic UNLESS I also have it configured in OPNs. When I remove the bridge in OPNs it won't pass traffic. I can delete the config in ZA and reconfigure and still no joy. I have tried rebooting after config changes, and also tried with interfaces enabled and disabled in OPNs... but it doesn't matter  (again , all these configs previously had worked). Like I said, this started happening after a recent update. I have reinstalled ZA and also completely reloaded OPNs from scratch so I am at a loss. It is completely possible I am missing something, but I am pretty sure my troubleshooting is sound. Would be happy to work with you guys on it to sort it out.       

[BNaCl]

Update - I no longer think this is related to OPNs versions as I went back to 22.1 and it still won't pass traffic. The only way it will pass traffic is if I also add the bridge in OPNs. Remove the OPNs bridge and traffic stops. The only theory I can come up with is that the ZA bridging deployment mode broke in a recent ZA update.

One odd thing I have noted is that on the ZA status page the protected interface shows as one of the interfaces (em1) as opposed to the bridge (br0). I could swear back when this was working it showed br0 in the status.

I realize that most likely not many are using this in a transparent bridge config, but would help me if someone could confirm this behavior.

[sy]

Hi @BNaCl,

Can you share a bug report from the upper right corner of Zenarmor GUI by selecting all checkboxes?

[BNaCl]

Bug report sent, thanks for taking a look.

[BNaCl]

Closing the loop on this mystery... Sunny Valley has confirmed this to be a netmap issue. My understanding is they will be developing a fix in the coming weeks which I have volunteered to beta test. It seems not too many are running in a transparent bridge config, but just for awareness I would say it isn't a viable deployment mode until further notice. This of course is simply based on my experience and understanding of the situation and is no way an official word from SV. 

[evanrich]

Did they ever fix this? I've had zA turned off for weeks now because everything I try causes it to flap non-stop.  Only passive mode works.  The last post says "a couple weeks" but i haven't seen an update to ZA in a long time.

[beki]

Hi evanrich,
netmap fix will be shipped soon by the OPNsense team.
https://forum.opnsense.org/index.php?topic=32114.60

[BNaCl]

@evanrich - Just making sure you aren't getting incorrect info here. From my understanding (albeit limited), it is quite likely there are two, unrelated issues going on here. The thread and fix @beki is referring to is NOT related to the issue I posted here. If you go to the beginning of that thread he referenced, you will see my post asking if it was related, which it is not. If your issue is similar to mine which is specific to Bridge Mode, then we are dealing with a different issue where if_bridge and netmap are not playing nicely (or some variation of that). @franco did reply to my query in that thread indicating they are working on if_bridge compatibility, but he seemed to indicate this was more of a workaround which didn't seem ideal. Again, this quickly got over my head, but I came away with the understanding Bridge mode really isn't a viable option for the time being, so I had to resort to reworking my network setup to use routed mode.

Hope this helps.