dhcpd messages appearing in log after upgrade to 23.1

[Headless1919]

After upgrading to 23.1, I have noticed that it takes much longer for inter-VLAN routing to start working when the firewall boots. All of my interface addresses are statically assigned (IPv4 and IPv6) and I have made no configuration changes since upgrading. When looking at the system log, I see messages similar to the below, repeating for about ~3 minutes - during which I have basic ping connectivity to the internet, but no DNS or reachability to anything else inside my network:

/usr/local/etc/rc.linkup: dhcpd_dhcp4_configure() found no suitable IPv4 address on opt7   

This also seems to delay Unbound starting, which would explain my DNS problems as I point to Unbound on the firewall.

Any ideas what might be causing this or what I can check/do to resolve?

[Headless1919]

I have tried various things but nothing has made any difference. It does seem to come right by itself, eventually, randomly - sometimes it necessitates reloading services via SSH. Changes include allowing all ICMPv6, disabling bogon blocking on WAN, removing dynamic DNS, nothing has helped.

At this point it seems like there is some kind of crash loop with the DHCPv6 server which is also affecting Unbound. Interfaces also seem to be changing state constantly between up and down which is completely breaking the network.

Can anyone help me with this? At this point. I will have no choice but to roll back to 22.7.

[Headless1919]

Can I PLEASE get some help with this? Nothing I have tried has made any difference. I thought resetting the tunables to default yesterday fixed it, but apparently not.

[franco]

The log messages have been raised in level from notice to warning. There were no functional changes.


Cheers,
Franco

[Taunt9930]

Have you seen the threads relating to certain tunables causing issues that have been hidden until this release? Have you set any additional/custom tunables related to the interface that might be causing a problem?

[Headless1919]

The log messages have been raised in level from notice to warning. There were no functional changes.


Cheers,
Franco

Thanks for clearing that up Franco. In that case, the messages might be circumstantial rather than an indication of the problem - but there is definitely something wrong, somewhere. That was the only "concerning" thing I could see in the logs. Is there anything else you can think of which might be causing this? Its almost like the interface is being disabled/reenabled constantly, leading to complete outage.

Have you seen the threads relating to certain tunables causing issues that have been hidden until this release? Have you set any additional/custom tunables related to the interface that might be causing a problem?

Yeah I saw those, thanks. Yesterday I defaulted all tunables, which I thought fixed it. Today - no joy. Have reset everything to default again and will test a little later today.

[franco]

I can make no other statement without addition information on what is actually not working.


Cheers,
Franco

[Headless1919]

I can make no other statement without addition information on what is actually not working.


Cheers,
Franco

Understood - I shall try to do a detailed write up later/tomorrow if or when the issue reoccurs. Will try to include as much detail as I can while still (hopefully) making sense.

[Headless1919]

In the interim - dmesg shows similar logs repeating, not sure if it means anything but it would seem consistent with losing connectivity:

igb1: permanently promiscuous mode disabled
738.093565 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: permanently promiscuous mode enabled
738.095867 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: link state changed to DOWN
igb1_vlan2: link state changed to DOWN
igb1_vlan3: link state changed to DOWN
igb1_vlan4: link state changed to DOWN
igb1_vlan5: link state changed to DOWN
igb1_vlan6: link state changed to DOWN
igb1_vlan7: link state changed to DOWN
igb1_vlan8: link state changed to DOWN
igb1_vlan9: link state changed to DOWN
igb1_vlan4090: link state changed to DOWN
igb1_vlan10: link state changed to DOWN
igb1_vlan15: link state changed to DOWN
738.374788 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: link state changed to UP
igb1_vlan2: link state changed to UP
igb1_vlan3: link state changed to UP
igb1_vlan4: link state changed to UP
igb1_vlan5: link state changed to UP
igb1_vlan6: link state changed to UP
igb1_vlan7: link state changed to UP
igb1_vlan8: link state changed to UP
igb1_vlan9: link state changed to UP
igb1_vlan4090: link state changed to UP
igb1_vlan10: link state changed to UP
igb1_vlan15: link state changed to UP
igb1: permanently promiscuous mode disabled

[franco]

Aha, netmap again.. is this Zenarmor or Suricata's doing?

As a general rule DO NOT select VLANs in their settings, only parents and / or try https://forum.opnsense.org/index.php?topic=32114.0


Cheers,
Franco

[Headless1919]

Aha, netmap again.. is this Zenarmor or Suricata's doing?

As a general rule DO NOT select VLANs in their settings, only parents and / or try https://forum.opnsense.org/index.php?topic=32114.0


Cheers,
Franco

I caught onto that shortly after posting yeah... for now, I have disabled Suricata and installed the netmap kernel patch (even though I do not see the generic_netmap_register messages). My Suricata config only has LAN selected (igb1). I do not have Zenarmor installed as yet.

Will be doing a cold boot in the next few hours, will report back on situation (latest tomorrow morning).

Thanks Franco, with luck this fixes it.

[franco]

If you haven't had VLAN selected that might not be enough. But we are working on more of those issues with netmap at the moment.

Disabling IPS mode should suffice for stability for the time being.


Cheers,
Franco

[Headless1919]

Preliminary feedback: after disabling Suricata and applying the netmap patch, the firewall booted normally without any issues. I do not want to call it resolved yet (since I thought it was resolved yesterday, only to find out not), but it looks positive.

Tomorrow, I shall test another cold boot to see if everything is still healthy. If so, I will enable Suricata in IDS mode, test again, and if everything is still working, revert to IPS mode and test once again.

Will report back in a couple of days with the results, maybe this will help to narrow down the problem.

@Franco let me know if you need anything in particular tested, happy to assist where I can.

[franco]

I am certain it's only IPS mode causing this. Have you set a schedule to update rules perhaps?


Cheers,
Franco

[Headless1919]

I am certain it's only IPS mode causing this. Have you set a schedule to update rules perhaps?


Cheers,
Franco

Cron job runs at 23:59 to update rules. So far so good with just IDS mode, will test IPS over the weekend and report back.

One question: having installed the kernel patch, I assume this will be reverted with the next minor release and I will need to apply again (if indeed that is part of the fix)?

Thanks for the help so far, much appreciated.