WAN interface flapping with 22.1.2

[Scuro]

As a new OPNSense user I can confirm that MAC spoofing causes the WAN to flap.
I am running a J4125 unit with Intel I225-V NICs on 22.1.6.

Logs:

Code: [Select]

2022-04-14T07:11:05-07:00 Error opnsense /interfaces.php: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igc0.pid' 'igc0'' returned exit code '1', the output was 'igc0: no link .............. giving up'
2022-04-14T07:10:22-07:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(igc0)
2022-04-14T07:10:21-07:00 Error opnsense /usr/local/etc/rc.linkup: Warning! dhcpd_dhcp4_configure() found no suitable IPv4 address on opt1
2022-04-14T07:10:21-07:00 Error opnsense /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
If I disable MAC spoofing it no longer flaps the WAN.

[grunge]

I need to spoof the mac address so still waiting for a fix

[subivoodoo]

I also need it... a fix would be nice!

Or is there at least an idea of the real issue? Intel driver [only?] issue with mac spoofing???

[devhunter55]

Thx all for the very good hint about "mac spoofing" .. i did not have got the chance to test it.
1rst of all, i wrote my ISP if "mac spoofing" is mandatory from his side.

on my side - UNBOUND is not working anymore since 22.1.2 or since UNBOUND version: unbound 1.15.0

Yesterday, i gave it a try again - and upgraded to 22.1.6.

I've got a lot of UNBOUND "overrides" in place & BlockLists.
I did disable the BlockLists - but this didn't help.

DNS is not working anymore - it seems that the DNS Resolver will switch from offline to online & vice versa in a very short time (what others called "flapping")

between i can connect to the WEB for a short time (but resolution is very, very slow).
..getting a WAN DHCP ip & and then it disappears again.

The whole machine gets very hot, CPU & unbound is about 100% - restart UNBOUND does not fix this issue.

Fortunately i'm using ZFS, so Restore is quick & easy - but full RESTORE was needed in every upgrade after version: 22.1.1 (with UNBOUND 1.15.0).

I tried the upgrades also with different hardware - same result - no chance to get UNBOUND working again - and DNS - of course is fundamental.

messages:
------------
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="32"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(igb1)
<27>1 2022-04-15T20:59:19+02:00 opnsense-host dhclient 96268 - [meta sequenceId="33"] dhclient already running, pid: 86990.
<26>1 2022-04-15T20:59:19+02:00 opnsense-host dhclient 96268 - [meta sequenceId="34"] exiting.
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="35"] /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/d
hclient.igb1.pid' 'igb1'' returned exit code '1', the output was 'dhclient already running, pid: 86990. exiting.'
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="36"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-04-15T20:59:19+02:00 opnsense-host dhcp6c 3104 - [meta sequenceId="37"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="38"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="39"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to lan
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="40"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="41"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to wan
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="42"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="43"] plugins_configure ipsec (,wan)
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="44"] plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="45"] plugins_configure dhcp ()
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="46"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="47"] plugins_configure dns ()
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="48"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="49"] plugins_configure dns (execute task : unbound_configure_do())
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="50"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: debug.pfftpproxy
<13>1 2022-04-15T20:59:24+02:00 opnsense-host dhcp6c 64283 - [meta sequenceId="51"] RTSOLD script - Sending SIGHUP to dhcp6c
<27>1 2022-04-15T20:59:36+02:00 opnsense-host upsmon 39698 - [meta sequenceId="52"] UPS [ups@localupsip]: connect failed: Connection failure: Operation timed out
<13>1 2022-04-15T20:59:49+02:00 opnsense-host configctl 68544 - [meta sequenceId="53"] event @ 1650049188.54 msg: Apr 15 20:59:48 opnsense-host config[87361]: [2022-04-15T20:59:48+02
:00][info] config-event: new_config /conf/backup/config-1650049188.5364.xml
<13>1 2022-04-15T20:59:49+02:00 opnsense-host configctl 68544 - [meta sequenceId="54"] event @ 1650049188.54 exec: system event config_changed
<27>1 2022-04-15T21:00:56+02:00 opnsense-host upsmon 39698 - [meta sequenceId="1"] UPS [ups@localupsip]: connect failed: Connection failure: Operation timed out
<29>1 2022-04-15T21:00:56+02:00 opnsense-host upsmon 39698 - [meta sequenceId="2"] UPS ups@localupsip is unavailable
<11>1 2022-04-15T21:01:19+02:00 opnsense-host configctl 87822 - [meta sequenceId="3"] error in configd communication  Traceback (most recent call last):   File "/usr/local/sbin/configctl
", line 66, in exec_config_cmd     line = sock.recv(65536).decode() socket.timeout: timed out
<11>1 2022-04-15T21:01:19+02:00 opnsense-host opnsense 99032 - [meta sequenceId="4"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(igb1)
<11>1 2022-04-15T21:01:19+02:00 opnsense-host opnsense 99032 - [meta sequenceId="5"] /usr/local/etc/rc.linkup: Clearing states for stale wan route on igb1
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="6"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(igb1)
<27>1 2022-04-15T21:01:20+02:00 opnsense-host dhclient 23026 - [meta sequenceId="7"] dhclient already running, pid: 86990.
<26>1 2022-04-15T21:01:20+02:00 opnsense-host dhclient 23026 - [meta sequenceId="8"] exiting.
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dh
client.igb1.pid' 'igb1'' returned exit code '1', the output was 'dhclient already running, pid: 86990. exiting.'
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-04-15T21:01:20+02:00 opnsense-host dhcp6c 30500 - [meta sequenceId="11"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to lan
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to wan
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="17"] plugins_configure ipsec (,wan)
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="18"] plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="19"] plugins_configure dhcp ()
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="20"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="21"] plugins_configure dns ()
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="22"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="23"] plugins_configure dns (execute task : unbound_configure_do())
<11>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: debug.pfftpproxy
<13>1 2022-04-15T21:01:24+02:00 opnsense-host dhcp6c 88779 - [meta sequenceId="25"] RTSOLD script - Sending SIGHUP to dhcp6c

[devhunter55]

https://forum.opnsense.org/index.php?topic=27372.15

[subivoodoo]

In my case it's really only an issue on Intel NIC's... switched my WAN to the onboard Realtek NIC and all is working now (mac spoofing with IDP on in IPS mode, OPNsense 22.1.6)

Is there any kernel or driver fix underway?

[tracerrx]

I still have to change the gateway monitor address on multiple firewalls every 3-4 days (all dual wan) or the gateway start to flap...

[bugvito]

In my case it's really only an issue on Intel NIC's... switched my WAN to the onboard Realtek NIC and all is working now (mac spoofing with IDP on in IPS mode, OPNsense 22.1.6)

Is there any kernel or driver fix underway?

What would be your intel NIC model? I'm having WAN issues (spoofing MACs on WAN vlans) when upgrading from 22.1.2 to 22.1.5-6 with the dreaded i225, but I'm not clear at this point the actual issue.

[tracerrx]

Code: [Select]

sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.5.%desc: Intel(R) I211 (Copper)
dev.igb.4.%desc: Intel(R) I211 (Copper)
dev.igb.3.%desc: Intel(R) I211 (Copper)
dev.igb.2.%desc: Intel(R) I211 (Copper)
dev.igb.1.%desc: Intel(R) I211 (Copper)
dev.igb.0.%desc: Intel(R) I211 (Copper)

sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.3.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.2.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.1.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.0.%desc: Intel(R) I210 Flashless (Copper)

[subivoodoo]

I have this issue on a X710 and on another system with I225... and no issues on a test VM (unraid/KVM virtio NIC) and also no issue on some old crappy realtek onboard NIC.

[foxmanb]

@Franco any update on this one? Will we see it resolved in a future release (soon hopefully?).

[devhunter55]

@Franco - yes - we would all appreciate if this could fixed soon 

(knocked out since 22.1.1 (with UNBOUND 1.15.0))

[bugvito]

I have this issue on a X710 and on another system with I225... and no issues on a test VM (unraid/KVM virtio NIC) and also no issue on some old crappy realtek onboard NIC.

Same here; my VM with VirtIO is fine for WAN with VLANs and MAC spoofing (2 USB dongles on proxmox, AQC111U), my bare metal with I225 has issues with this WAN+vlan+spoof setup.

[tracerrx]

Everyone having this issue seems to be using intel nics, so maybe this is just a FreeBSD 13 driver issue? Anyone have a howto to install the updated intel drivers that are included in FreeBSD 13.1?

[devhunter55]

this may be true in my case, too (but i think - as long we don't know the reason for - it is an assumption):

root@opnsense:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'

dev.igb.5.%desc: Intel(R) I210 (Copper)
dev.igb.4.%desc: Intel(R) I210 (Copper)
dev.igb.3.%desc: Intel(R) I210 (Copper)
dev.igb.2.%desc: Intel(R) I210 (Copper)
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)


---------------------------------

Intel® Ethernet Controller I210
I210 controllers support speeds up to 1GbE on a single port with advanced features such as Audio-Video Bridging (AVB), IEEE 802.1AS precision timestamping, Error Correcting Code (ECC) Packet Buffers, and Enhanced Management Interface options.