DAI problems on WAN?

[jberg]

Hello,

I work as a network engineer for a living, and discovered some issues on all hardware i have used to install opnsense, first i truly belived it was hardware fault on my part but now on the third server i have some issues.

First: Running opnsense on a Supermicro server with dual nics (32-bit Intel Atom quad core) installation etc was fine but up on connecting to a Metro network port from my local municipally owned network i ran in to issues.

On the switch side we saw Invalid arps, bascially the Opnsense router replied with ARP issues (see attachment, note wrong timestamp and spoofed mac, its not a dlink)

I switch to a new supermicro server, a x64 xeon with dual nics, same problem but this time all of a sudden it stopped (no configuration change on my end) and switchlog flood stopped.

So yesterday it was time to downgrade from a overkill server hardware to a smaller barebone itx sized thing off amazon, installation was fine and worked on the lan side. But of course when connected same issues occure. So this time the old working server was cold restarted and now the issue didn't dissappear on the previously working machine.

This really makes me scratch my head, at first i belived i fucked up the WAN/LAN side, but i didn't. And after that i noticed that i managed to get LAN-ip from the WAN interface, only for a second then link down and starts to get link flap on WAN and LAN interface.

Basically I think that there could be an issue with some wierd data occuring, triggering the switchport security config (see below)

Switchport config:
switchport access vlan 2003
switchport trunk native vlan 2003
switchport trunk allowed vlan 296,2003
switchport mode trunk
switchport block multicast
switchport block unicast
switchport port-security maximum 8
switchport port-security
switchport port-security aging time 900
switchport port-security violation protect
switchport port-security aging type inactivity
ip access-group 101 in
no logging event link-status
speed auto 100
duplex full
no snmp trap link-status
storm-control broadcast level 1.00
storm-control multicast level 1.00
mac access-group IPv4_only in
service-policy input KUND-INGRESS-QOS_100M
service-policy output KUND-EGRESS-QOS_100M
ip igmp max-groups 8
ip verify source
ip dhcp snooping limit rate 10

So according to the switch log its Dynamic Arp Inspection that blocks the port when this occurs. I have tried to replicate this in a lab but lack some hardware to compleat it at the moment.

Has anyone seen something like this before?

I will downgrade to PF-sense instead since i need the firewall back online.

Regards

[jberg]

Update: Pfsense has no issues like this, after installation and setup with equivalent config as the old opnsense it just worked, on both the x64 Supermicro server and the barebone from Amazon.

I think there are some leakage of traffic to wrong interfaces on current version of Opnsense. Curiously the pcap i generated at the same time as link flap was on going shows only common and expected arp traffic.

Regards.