Messages

I understand that it is largely out of your hands - but at least know I know where the problem is and can manage it.

Thanks Franco, appreciate your guidance.

Got it, thanks Franco. At this point there is no suggestion that the kernel has or has not had a role in solving the problem.

Your suspicions were correct: as soon as Suricata is set to IPS mode, things start breaking. Suppose I will just have to leave it as is for the time being.

Happy to help if there is any sort of testing needed.

I am certain it's only IPS mode causing this. Have you set a schedule to update rules perhaps?


Cheers,
Franco

Cron job runs at 23:59 to update rules. So far so good with just IDS mode, will test IPS over the weekend and report back.

One question: having installed the kernel patch, I assume this will be reverted with the next minor release and I will need to apply again (if indeed that is part of the fix)?

Thanks for the help so far, much appreciated.

Preliminary feedback: after disabling Suricata and applying the netmap patch, the firewall booted normally without any issues. I do not want to call it resolved yet (since I thought it was resolved yesterday, only to find out not), but it looks positive.

Tomorrow, I shall test another cold boot to see if everything is still healthy. If so, I will enable Suricata in IDS mode, test again, and if everything is still working, revert to IPS mode and test once again.

Will report back in a couple of days with the results, maybe this will help to narrow down the problem.

@Franco let me know if you need anything in particular tested, happy to assist where I can.

Aha, netmap again.. is this Zenarmor or Suricata's doing?

As a general rule DO NOT select VLANs in their settings, only parents and / or try https://forum.opnsense.org/index.php?topic=32114.0


Cheers,
Franco

I caught onto that shortly after posting yeah... for now, I have disabled Suricata and installed the netmap kernel patch (even though I do not see the generic_netmap_register messages). My Suricata config only has LAN selected (igb1). I do not have Zenarmor installed as yet.

Will be doing a cold boot in the next few hours, will report back on situation (latest tomorrow morning).

Thanks Franco, with luck this fixes it.

In the interim - dmesg shows similar logs repeating, not sure if it means anything but it would seem consistent with losing connectivity:

igb1: permanently promiscuous mode disabled
738.093565 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: permanently promiscuous mode enabled
738.095867 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: link state changed to DOWN
igb1_vlan2: link state changed to DOWN
igb1_vlan3: link state changed to DOWN
igb1_vlan4: link state changed to DOWN
igb1_vlan5: link state changed to DOWN
igb1_vlan6: link state changed to DOWN
igb1_vlan7: link state changed to DOWN
igb1_vlan8: link state changed to DOWN
igb1_vlan9: link state changed to DOWN
igb1_vlan4090: link state changed to DOWN
igb1_vlan10: link state changed to DOWN
igb1_vlan15: link state changed to DOWN
738.374788 [ 851] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: link state changed to UP
igb1_vlan2: link state changed to UP
igb1_vlan3: link state changed to UP
igb1_vlan4: link state changed to UP
igb1_vlan5: link state changed to UP
igb1_vlan6: link state changed to UP
igb1_vlan7: link state changed to UP
igb1_vlan8: link state changed to UP
igb1_vlan9: link state changed to UP
igb1_vlan4090: link state changed to UP
igb1_vlan10: link state changed to UP
igb1_vlan15: link state changed to UP
igb1: permanently promiscuous mode disabled

I can make no other statement without addition information on what is actually not working.


Cheers,
Franco

Understood - I shall try to do a detailed write up later/tomorrow if or when the issue reoccurs. Will try to include as much detail as I can while still (hopefully) making sense.

The log messages have been raised in level from notice to warning. There were no functional changes.


Cheers,
Franco

Thanks for clearing that up Franco. In that case, the messages might be circumstantial rather than an indication of the problem - but there is definitely something wrong, somewhere. That was the only "concerning" thing I could see in the logs. Is there anything else you can think of which might be causing this? Its almost like the interface is being disabled/reenabled constantly, leading to complete outage.

Have you seen the threads relating to certain tunables causing issues that have been hidden until this release? Have you set any additional/custom tunables related to the interface that might be causing a problem?

Yeah I saw those, thanks. Yesterday I defaulted all tunables, which I thought fixed it. Today - no joy. Have reset everything to default again and will test a little later today.

Can I PLEASE get some help with this? Nothing I have tried has made any difference. I thought resetting the tunables to default yesterday fixed it, but apparently not.

I have tried various things but nothing has made any difference. It does seem to come right by itself, eventually, randomly - sometimes it necessitates reloading services via SSH. Changes include allowing all ICMPv6, disabling bogon blocking on WAN, removing dynamic DNS, nothing has helped.

At this point it seems like there is some kind of crash loop with the DHCPv6 server which is also affecting Unbound. Interfaces also seem to be changing state constantly between up and down which is completely breaking the network.

Can anyone help me with this? At this point. I will have no choice but to roll back to 22.7.

After upgrading to 23.1, I have noticed that it takes much longer for inter-VLAN routing to start working when the firewall boots. All of my interface addresses are statically assigned (IPv4 and IPv6) and I have made no configuration changes since upgrading. When looking at the system log, I see messages similar to the below, repeating for about ~3 minutes - during which I have basic ping connectivity to the internet, but no DNS or reachability to anything else inside my network:

/usr/local/etc/rc.linkup: dhcpd_dhcp4_configure() found no suitable IPv4 address on opt7   

This also seems to delay Unbound starting, which would explain my DNS problems as I point to Unbound on the firewall.

Any ideas what might be causing this or what I can check/do to resolve?

Hi Alex,

GUI, Services -> Net-SNMP ->  Display Version in OID. Tick that and poll, it should return the precise version.

The OID is also listed underneath (1.3.6.1.4.1.8072.1.3.2.3.1.2.7.118.101.114.115.105.111.110)

@i81b4u - you're right, I am using the term "RSA" too broadly (interchangeably), was referring specifically to KEX. Thanks ;)

@Fright, thanks - I should have clicked. Will check after the next patch/update to see what is there. Appreciated!

@Fright, I went to look into config.xml but it reflects exactly what the GUI shows, so I am not convinced it will function as an override. Will bear it in mind thought, thanks!

@i81b4u, RSA does not offer Perfect Forward Secrecy so I prefer to remove it, but that will work - thanks.

@Fright - thanks! I get it, bad idea not supported etc. ;) busy documenting though so if I do mess something up, it should be easy to reverse. Best part is, in these situations there really is only one person to blame if something goes wrong. Really appreciate your help!

@i81b4u - minus the fourth and six entries yeah (RSA). I don't entirely trust RSA anymore, even if TLS 1.2 is still secure there are varying degrees of cipher strength within the overall spec.