# OPNsense 26.7 regression: Intel X540-AT2 SR-IOV (ixv) interfaces stop working after FreeBSD 15.1 upgrade — VLAN RX / DHCP / ARP failure on VMware ESXi 8.0 U2
**Disclaimer:** This post was generated with assistance from ChatGPT following an extensive, interactive troubleshooting session. The investigation involved examining actual OPNsense configurations, packet captures, interface statistics, DHCP logs, historical kernel crash output, VMware ESXi diagnostics, Intel driver release notes, and controlled before/after testing. Technical observations are based on collected evidence, while hypotheses and potential root causes are explicitly identified as unconfirmed. I have reviewed the findings and removed identifying and security-sensitive information.
---
## 1. Summary
Hi everyone,
I'm looking for assistance from the OPNsense and FreeBSD community with what appears to be a regression affecting **Intel X540-AT2 SR-IOV virtual functions (`ixv`) following the OPNsense 26.7 major upgrade to FreeBSD 15.1**.
**The most important detail is that this exact SR-IOV configuration worked before the OPNsense 26.7 upgrade.**
Following the upgrade, networking failed across multiple interfaces using the Intel `ixv` driver. Clients could no longer communicate normally with the firewall or obtain DHCP addresses.
I restored operational connectivity by migrating the affected production interfaces to VMware VMXNET3 (`vmx`) adapters.
I have subsequently isolated the issue to a test interface, `ixv2`, attached to my IoT network, while retaining functional production connectivity through VMXNET3.
The issue persists on **OPNsense 26.7.7**, including after upgrading the ESXi Intel physical-function driver from `ixgben 1.20.2.0` to `1.22.1.0`.
### Key findings
1. Intel SR-IOV networking worked before OPNsense 26.7.
2. `ixv` interfaces became unusable following the major upgrade.
3. The `ixv2` interface is UP/RUNNING and records incoming and outgoing packets.
4. All incoming packets in a controlled capture contain VLAN 20 tags, while firewall-originated packets are untagged.
5. The interface is assigned directly to `ixv2`, without a guest VLAN subinterface, because VLAN 20 is configured on the VMware port group.
6. The guest interface's "Packets for Unknown Protocol" counter previously exactly matched its total received-packet counter.
7. Hundreds of DHCP Discover packets reach the `ixv2` packet capture, but no DHCP Offers or ARP replies are observed.
8. Zenarmor is not configured to inspect `ixv2`.
9. Upgrading the ESXi physical-function driver and rebooting the host did not resolve the issue.
10. VMXNET3 networking remains functional.
My leading hypothesis is an **interoperability regression involving FreeBSD 15.1 `ixv`/`iflib`, SR-IOV VLAN metadata, or VLAN receive-path processing**.
However, I have not established whether the underlying fault is in the FreeBSD VF driver, the ESXi PF/VF interaction, or another part of the network stack.
---
## 2. Environment
### OPNsense virtual machine
| Component | Details |
|---|---|
| Current OPNsense | 26.7.7 |
| Architecture | amd64 |
| Firmware commit | `f1a83f72b` |
| FreeBSD base | 15.1 |
| Previously affected OPNsense | 26.7.5 |
| Last known working | Before 26.7 major upgrade; exact version not recorded |
| Hypervisor | VMware ESXi 8.0 Update 2 |
| VM compatibility | ESXi 8.0 U2 virtual machine, hardware version 21 |
| Guest OS selection | FreeBSD 14 or later (64-bit) |
| vCPUs | 8 |
| Memory | 20 GB |
| Guest networking | 5 VMXNET3 and 5 SR-IOV network adapters |
| Test interface | `ixv2` |
| Test network | IoT, VLAN 20 |
| Guest MTU | 1500 |
| VMware port group VLAN | 20 |
| Guest VLAN subinterface | None |
The ESXi host is standalone, without vCenter.
### ESXi host
```text
Product: VMware ESXi
Version: 8.0.2
Build: 23825572
Update: 2
Patch: 40
```
### Physical NIC and driver
```text
Controller:
Intel Ethernet Controller X540-AT2
Physical Function:
PCI: 0000:03:00.0
Vendor ID: 8086
Device ID: 1528
Subvendor ID: 8086
Subdevice ID: 35c5
ESXi interface:
vmnic0
ESXi driver:
ixgben
Driver before test:
1.20.2.0
Driver after test:
1.22.1.0
NIC firmware:
4.04.0 0x800004f8
Physical link:
1000 Mbps
Full duplex
Link UP
Physical interface MTU:
1508
```
### SR-IOV virtual function
```text
Controller:
Intel X540 Ethernet Controller Virtual Function
Vendor ID: 8086
Device ID: 1515
Guest driver:
ixv
Test interface:
ixv2
```
The VMware VM hardware settings confirm that the affected guest interface is backed by an **SR-IOV passthrough network adapter** associated with the Intel X540 physical function.
The configured VMware adapter MAC was confirmed to match the OPNsense `ixv2` MAC. The actual MAC addresses are omitted here.
---
## 3. Regression timeline
### Before OPNsense 26.7
The Intel SR-IOV virtual functions were working.
The host, SR-IOV configuration, port groups, and VLAN arrangements were operational with the previous OPNsense release.
There was no intentional VMware VLAN configuration change associated with the OPNsense upgrade.
### Following the OPNsense 26.7 upgrade
Multiple Intel `ixv` interfaces stopped providing usable network connectivity.
Observed symptoms included:
- Clients unable to obtain DHCP leases.
- Loss of normal communications with the firewall on affected networks.
- Firewall services becoming inaccessible through affected interfaces.
- Interface connectivity problems despite driver attachment.
- Netmap-related kernel errors during troubleshooting.
- A historical kernel panic involving `iflib` and Netmap-related functions.
### Initial workaround
To restore connectivity, I reassigned the affected production networks to VMware VMXNET3 (`vmx`) adapters.
This restored functional networking.
### Controlled troubleshooting on 26.7.7
I retained all other production interfaces on the functioning VMXNET3 adapters and assigned only the IoT network to `ixv2`.
The problem remained reproducible without disrupting the other networks.
### ESXi physical-function driver upgrade
I upgraded the ESXi Intel driver:
```text
Before:
ixgben 1.20.2.0
After:
ixgben 1.22.1.0
```
The package passed the ESXi dry run and installed successfully.
The host rebooted successfully.
I confirmed that `ixgben 1.22.1.0` was active, the physical link was UP, and the SR-IOV VFs were active.
**Result: No improvement.**
Following the ESXi reboot and driver upgrade, IoT clients still cannot obtain DHCP addresses or communicate normally through `ixv2`.
A fresh packet capture shows the same apparent VLAN-tagging asymmetry as before.
---
## 4. Current ixv2 interface status
The interface successfully attaches and reports link UP.
The following statistics were collected before the ESXi driver update:
```text
Interface: ixv2
Flags:
UP
BROADCAST
RUNNING
SIMPLEX
MULTICAST
LOWER_UP
Status: UP
Media: 1000baseT full-duplex
Line rate: 1.00 Gbit/s
MTU: 1500
IPv4 configuration: Static /24
IPv6 configuration: Track6
VLAN tag: None
Packets received: 13267
Packets transmitted: 772
Input errors: 0
Output errors: 0
Input queue drops: 0
Output queue drops: 0
Packets for unknown proto: 13267
Send queue drops: 0
```
### Important observation: unknown-protocol counter
The number of packets classified as unknown protocol exactly matched the total received-packet counter:
```text
Packets received: 13267
Packets for unknown proto: 13267
```
I understand that FreeBSD maintains an `ifi_noproto` interface statistic.
I'm particularly interested in whether this counter is expected to behave this way when an `ixv` interface receives frames carrying an 802.1Q VLAN tag without a corresponding guest VLAN interface.
It would be helpful to understand whether this indicates failed network-layer protocol dispatch, VLAN demultiplexing behaviour, or potentially misleading accounting in the driver.
The captured frames are recognisable IPv4, IPv6 and ARP traffic, so the exact protocol-processing boundary is important.
---
## 5. Packet capture investigation
This has been the most useful part of the investigation.
I collected an actual PCAP directly from `ixv2` using the OPNsense Web UI.
### Capture settings
```text
Interface: ixv2
Address family: Any
Protocol: Any
Promiscuous: Disabled
Host filter: None
Port filter: None
Packet limit: 10000
```
### Initial capture results
The capture contained 926 Ethernet frames over approximately 82 seconds.
The original PCAP was decoded to inspect Ethernet headers, VLAN tags, ARP packets, IP protocols, and DHCP message types.
```text
Total captured frames: 926
Incoming frames: 872
Incoming VLAN 20 frames: 872
Incoming untagged frames: 0
Firewall-originated frames: 54
Firewall-originated tagged frames: 0
Firewall-originated untagged frames: 54
DHCP Discover: 364
Distinct DHCP client MAC addresses: 33
DHCP Offer: 0
DHCP ACK: 0
Incoming ARP requests: 102
Firewall-originated ARP requests: 27
ARP replies: 0
IPv4 frames: 477
IPv6 frames: 282
ARP frames: 129
```
The DHCP Discover count was confirmed by decoding DHCP message-type option 53 rather than merely counting UDP port 67 traffic.
**All captured incoming frames contained an IEEE 802.1Q VLAN 20 header. All firewall-originated frames were untagged.**
This is particularly interesting because the OPNsense interface is not configured with a VLAN subinterface.
The VMware port group is configured with VLAN ID 20, which would ordinarily mean that the guest's parent interface handles untagged traffic in an access-style configuration.
### Example of inbound traffic
An anonymised representation of a captured packet:
```text
Source: [IoT client MAC]
Destination: ff:ff:ff:ff:ff:ff
802.1Q VLAN:
VLAN ID 20
Priority 0
IPv4:
Source 0.0.0.0
Destination 255.255.255.255
UDP:
Source port 68
Destination port 67
DHCP:
Message type: Discover
```
### Example of outbound firewall traffic
```text
Source: [Firewall ixv2 MAC]
Destination: ff:ff:ff:ff:ff:ff
Ethernet:
Untagged
ARP:
Request who-has [IoT client IP]
Tell [IoT gateway IP]
```
No corresponding ARP reply was observed.
### Interpretation
The capture establishes that inbound network traffic reaches the guest's packet-capture point, but normal communication is not functioning.
It also confirms an apparent mismatch between tagged inbound traffic and untagged outbound traffic.
However, I recognise an important technical limitation: **a packet capture containing VLAN headers does not necessarily prove those headers were physically delivered unchanged by the NIC**. Depending on the driver and capture mechanism, VLAN metadata could potentially be reconstructed into captured Ethernet frames.
I would appreciate guidance on how FreeBSD's `ixv` and `iflib` handle this distinction and how to verify what is actually passed to the network stack.
---
## 6. Post-driver-update packet capture
After installing ESXi `ixgben 1.22.1.0` and rebooting the host, I captured traffic again from `ixv2`.
The exported text contains 100 packet records over approximately five seconds.
Results:
```text
Total packet records: 100
VLAN 20 tagged frames: 96
Untagged frames: 4
Client DHCP messages:
UDP 68 -> 67 19
ARP requests: 5
ARP replies: 0
Observed DHCP server replies: 0
```
The four untagged frames were firewall-originated ARP requests.
The tagged frames were incoming network traffic.
This reproduces the same basic asymmetry after the ESXi physical-function driver update.
**The driver upgrade did not restore DHCP or normal communication through `ixv2`.**
I can provide additional sanitised packet details or reproduce another capture if a maintainer recommends a specific capture filter or diagnostic procedure.
---
## 7. VMware VLAN and SR-IOV configuration
The IoT interface uses a VMware standard virtual switch and a port group with VLAN ID 20.
```text
VMware port group:
VLAN ID: 20
Adapter type:
SR-IOV passthrough
Physical function:
Intel X540-AT2
Guest interface:
ixv2
Guest VLAN configuration:
No VLAN subinterface
Guest MTU:
1500
```
### Port group security policies
```text
Promiscuous mode:
Accept
Forged transmits:
Accept
MAC address changes:
Accept
```
NIC teaming uses routing based on the originating virtual port ID.
The configuration worked before the OPNsense major upgrade.
The same host also uses VMXNET3 adapters successfully.
For this reason, I am hesitant to modify the port-group VLAN ID or create a VLAN interface inside OPNsense without first understanding whether such a change would merely work around a FreeBSD receive-path regression.
---
## 8. ESXi SR-IOV hardware statistics
I also examined the hardware virtual-function allocations and counters.
The Intel X540 physical function reports:
```text
Configured virtual functions: 25
Active virtual functions: 11
Inactive virtual functions: 14
```
Five active VFs share a common ESXi owner World ID, consistent with their assignment to the OPNsense VM.
The World ID changes following VM/host restarts, so it is not a persistent identifier.
### Per-VF statistics before the driver update
| Statistic | VF 0–3 | VF 4 |
|---|---:|---:|
| RX unicast packets | 0 | 62,068 |
| RX multicast packets | 0 | 19,109 |
| TX unicast packets | 0 | 3,588 |
| RX unicast bytes | 0 | 12,427,463 |
| TX unicast bytes | 0 | 329,914 |
| RX error drops | 0 | 0 |
| RX out-of-buffer drops | 0 | 0 |
| TX errors | 0 | 0 |
| TX discards | 0 | 0 |
These statistics were collected using:
```sh
esxcli network sriovnic vf stats -n vmnic0 -v <VF_ID>
```
Only VF 4 reported nonzero traffic counters among the first five VFs.
The exact ESXi VF ID-to-guest `ixv` interface mapping was not independently established.
There are also indications that some of these hardware counters may be incomplete or classified differently from the guest counters. For example, multicast packets were reported while multicast bytes remained zero.
I have therefore avoided drawing conclusions from the all-zero counters alone.
---
## 9. DHCP configuration and logs
OPNsense has ISC DHCPv4 configured and enabled for the IoT interface.
The configuration backup confirms:
```text
IoT interface:
Enabled
IPv4:
Static /24
ISC DHCPv4:
Enabled
Kea DHCPv4:
Disabled
Kea DHCPv6:
Disabled
```
The configured DHCP pool is relatively small, with a larger set of static mappings. That may warrant separate housekeeping but does not explain the absence of ARP replies or the apparent VLAN receive-path issue.
### DHCP log investigation
I reviewed an exported ISC DHCP log containing approximately 42,773 error-level records.
The overwhelming majority concern failures to perform dynamic DNS registrations:
```text
Unable to add forward map ... : NOTIMP
```
These occur across multiple networks, including networks functioning over VMXNET3.
There are also historical address-pool-related messages.
I believe the repeated dynamic DNS errors are a separate configuration problem.
Importantly, the exported log was filtered to error-level messages, so the absence of normal DHCP Discover/Offer entries is not evidence that the DHCP daemon did not receive the packets.
The packet capture independently confirms that many client DHCP Discover frames are visible on `ixv2`, but no replies appear during the capture.
I have not yet verified the DHCP daemon's live socket binding or performed a simultaneous DHCP application log and packet capture at debug verbosity.
---
## 10. Zenarmor, Netmap and interface offloading
Zenarmor is installed on this OPNsense VM.
It uses:
```text
Deployment:
Routed Mode (L3)
Operation:
Reporting + Blocking
Packet engine:
Emulated Netmap
```
### Protected interfaces
```text
vmx0 (LAN): Protected
vmx3 (Management): Protected
ixv2 (IoT): Not protected
vmx1 (VLAN trunk): Not protected
vmx4: Not protected
```
I verified this through the live Zenarmor Web UI.
**The failing `ixv2` interface is not selected for Zenarmor protection.**
Suricata IDS/IPS is disabled.
The OPNsense configuration also contains the following Netmap-related tunables, originally added by Zenarmor:
```text
dev.netmap.buf_num = 1000000
dev.netmap.ring_num = 1024
dev.netmap.admode = 2
dev.netmap.if_num = 512
dev.netmap.ring_size = 131072
```
Other relevant tunables include:
```text
net.isr.dispatch = deferred
net.isr.bindthreads = 1
net.isr.maxthreads = -1
net.inet.rss.enabled = 0
```
The OPNsense configuration also has hardware checksum offloading, TSO, LRO and VLAN hardware filtering disabled.
The guest interface reports Netmap among its advertised capabilities, although this does not demonstrate that the interface is actively attached to Netmap.
Zenarmor's interface selection makes direct interception of `ixv2` less likely to explain the current problem, but the earlier Netmap errors still warrant investigation.
---
## 11. Historical Netmap and kernel errors
During the initial troubleshooting, particularly on OPNsense 26.7.5, kernel logs included:
```text
netmap_do_regif ixv1:
error: rx_buf_maxsize == 0
```
Additional messages involved Netmap adapter registration and missing/invalid buffer-size information.
A historical kernel panic was also observed.
Relevant crash details included:
```text
Fatal trap 12:
page fault while in kernel mode
Fault virtual address:
0x1
Current process:
swi1: netisr 0
```
The crash investigation included the following functions:
```text
generic_mbuf_dtor
mb_free_ext
m_freem
iflib_completed_tx_reclaim
iflib_txq_drain
iflib_if_transmit
```
There were also historical messages involving:
```text
generic_mbuf_dtor:
Error: no netmap adapter on device
```
The previous `rx_buf_maxsize == 0` error referenced `ixv1`, whereas my current controlled test uses `ixv2`.
I have not established that the kernel panic, Netmap registration errors and current packet-processing problem share a single root cause.
However, the combination raises questions about changes to `iflib`, mbuf handling, driver capabilities and Netmap compatibility following the FreeBSD upgrade.
I can supply a sanitised kernel trace if requested.
---
## 12. Tests performed and outcomes
| Test | Outcome |
|---|---|
| Upgrade OPNsense to 26.7.7 | Issue persists |
| Replace affected network paths with VMXNET3 | Connectivity restored |
| Retest isolated IoT network using `ixv2` | Failure reproduced |
| Verify `ixv2` attachment and link state | UP/RUNNING |
| Verify VMware SR-IOV adapter and MAC mapping | Confirmed |
| Verify VMware VLAN configuration | VLAN 20 |
| Confirm DHCP configured on IoT | Confirmed |
| Capture incoming DHCP traffic | Hundreds of Discover messages visible |
| Capture firewall-originated traffic | Untagged ARP requests visible |
| Capture DHCP Offers or ARP replies | None observed |
| Compare unknown-protocol and RX counters | Exact match in initial snapshot |
| Check Zenarmor protected interfaces | `ixv2` not selected |
| Inspect ESXi VF allocations | VFs active |
| Inspect hardware VF counters | VF 4 shows activity; mapping incomplete |
| Upgrade ESXi PF driver to 1.22.1.0 | Successful |
| Reboot ESXi with upgraded PF driver | Successful |
| Verify active driver after reboot | 1.22.1.0 confirmed |
| Retest IoT connectivity after ESXi upgrade | Still broken |
| Repeat `ixv2` packet capture | VLAN-tagged ingress persists |
I have deliberately avoided changing VLAN assignments, disabling host-wide network functions or modifying the remaining production interfaces because they provide the working network access needed for debugging.
---
## 13. Current hypotheses
### A. FreeBSD 15.1 `ixv` / `iflib` regression
This is a leading possibility because the problem began after the OPNsense 26.7 major upgrade and affected multiple Intel SR-IOV interfaces.
The previous FreeBSD-based OPNsense release worked with the existing VMware configuration.
The issue persists despite upgrading the ESXi physical-function driver.
I'd like to understand which changes in `ixv`, `iflib` and related receive-path code could explain the symptoms.
### B. VLAN metadata or VLAN receive-path regression
The most distinctive evidence is:
```text
Incoming:
VLAN 20 tagged
Outgoing:
Untagged
Guest interface:
ixv2 without VLAN subinterface
VMware port group:
VLAN 20
Unknown-protocol counter:
Equal to total received packets
```
It appears possible that VLAN-tagged frames are reaching the guest capture but are not being dispatched to the intended IPv4, IPv6 or ARP handlers.
Whether this involves in-frame VLAN headers, driver metadata, VLAN hardware acceleration, or a change in BPF/iflib behaviour is unresolved.
### C. ESXi / X540 PF-VF interoperability
The host is running an Intel X540 physical function managed by `ixgben`, with FreeBSD `ixv` inside the guest.
There could be an interaction between host-side VF VLAN configuration and guest-side VLAN handling.
However, the configuration worked before 26.7, and upgrading the PF driver from 1.20.2.0 to 1.22.1.0 did not fix the problem.
### D. Netmap / driver interaction
The historical Netmap and mbuf errors may indicate a related regression.
However, Zenarmor is not configured to inspect the failing interface, and there is insufficient evidence to establish a direct connection between the historical kernel crash and the current VLAN delivery failure.
### E. DHCP/application-level issues
ISC DHCP has some separate configuration issues, particularly repeated dynamic DNS update errors.
But DHCP alone would not explain the broader inability to communicate or the lack of ARP replies, so I consider it a secondary line of investigation.
---
## 14. Questions for OPNsense / FreeBSD maintainers
I'd appreciate technical guidance on the following:
**1. ixv / iflib regression**
Are there known changes or regressions in FreeBSD 15.1 affecting Intel X540 SR-IOV VFs, particularly their VLAN receive path or protocol dispatch?
**2. VLAN handling**
When an ESXi SR-IOV VF is connected to a port group configured with VLAN 20, should the FreeBSD guest receive VLAN metadata separately or ordinary untagged Ethernet frames?
Could a change to hardware VLAN stripping or `iflib` metadata propagation explain the captured behaviour?
**3. Unknown-protocol counter**
Does the fact that `ifi_noproto` equals the entire received-packet count indicate VLAN demultiplexing failure, or could this be expected/misleading accounting with `ixv`?
**4. VF/PF compatibility**
Are there known interoperability issues between:
```text
ESXi ixgben 1.22.1.0
Intel X540-AT2
Firmware 4.04.0
FreeBSD 15.1 ixv
```
particularly around VF VLAN configuration, VLAN stripping, or PF/VF feature negotiation?
**5. Netmap**
Could the historical `rx_buf_maxsize == 0` and `generic_mbuf_dtor` errors be related to the current packet-processing problem?
What diagnostics would establish whether Netmap remains involved when an interface is not selected for Zenarmor protection?
**6. Most useful kernel diagnostics**
Which additional commands, counters or traces would help distinguish:
- VF hardware delivery problems.
- VLAN metadata processing errors.
- `iflib` receive-path issues.
- FreeBSD VLAN demultiplexing failures.
- ARP/IP protocol-dispatch failures.
- DHCP daemon/application problems.
**7. Controlled regression testing**
Would you recommend testing an alternative OPNsense/FreeBSD kernel, a particular `ixv`/`iflib` commit or a debug kernel?
If a known patch exists, I'm happy to test it on the isolated `ixv2` interface.
---
## 15. Potentially related upstream reports
Two reports identified during troubleshooting may be relevant:
**OPNsense source issue #314 — Netmap/iflib kernel panic**
https://github.com/opnsense/src/issues/314
This includes discussion of a `generic_mbuf_dtor` crash in a Netmap-related environment.
The hardware and configuration differ, so I'm not assuming it is the same defect.
**OPNsense core issue #10810 — VLAN traffic regression**
https://github.com/opnsense/core/issues/10810
This concerns VLAN traffic failing to reach the expected interface following an OPNsense 26.7 update.
Again, the affected hardware and VLAN configuration differ from my SR-IOV setup, but the packet-delivery symptoms may be relevant.
These links are provided as potentially useful references, not as confirmed diagnoses.
---
## 16. Additional evidence available
I have access to:
- OPNsense Web UI.
- OPNsense shell and VM console.
- VMware ESXi Host Client.
- ESXi SSH shell.
- Packet captures.
- OPNsense system and service logs.
- ESXi network and SR-IOV statistics.
- Historical kernel crash information.
I can reproduce the problem on `ixv2` while keeping the other networks operational through VMXNET3.
I'm happy to provide additional sanitised diagnostic output or conduct developer-recommended tests.
I have intentionally excluded the full firewall configuration backup, raw packet captures, internal network addresses, MAC addresses, hostnames, certificates, authentication information and other identifying details.
### Final observation
**The strongest evidence for a regression is that Intel SR-IOV worked before OPNsense 26.7, failed after the upgrade, continues to fail on 26.7.7, and remains broken after upgrading the ESXi physical-function driver.**
The packet captures suggest that receive-side VLAN handling is worth investigating before changing the DHCP service or VMware VLAN assignments.
I'd appreciate guidance on the most useful next debugging step or an upstream fix to test.
Thanks in advance for your help.
Possible for any help? Too long, I can summarize it but this is a legit issue. Everything was working before the freebsd was upgraded...