Interface problem on 23.1(_6)

[Roniro]

Hallo,

after the update to OPNsense 23.1 (and OPNsense 23.1_6) I got a strange behavior:

Everything works for a few seconds, than the main WAN interface gets errors on the console (photo in attachment) and the DNS (Unbound) fails permanently. The IP layer is still there.

Code: [Select]

% nslookup one.one.one.one                               :(  =100% =100%
Server:         10.77.10.1
Address:        10.77.10.1#53

** server can't find one.one.one.one: SERVFAIL

% ping 1.1.1.1                                     =100% =100%
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=17.1 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=14.4 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=14.3 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=54 time=13.9 ms
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 13.911/14.940/17.123/1.273 ms

The only way to fix this is to manually disable igb4. Then the gateway changes to ppp0 wich works as intended. If I re-enable igb4 the gateway changes back and everthing seems to be fine. At least for one hour ATM.

As you can see is my setup quite complex:

- igb0-3 is lagg0 and goes to my switch. There are some VLANs on it.
- igb4 is the Tier 1 WAN interface of my gateway group wich fails. It has a public static IP configured. It's connected to a cable modem wich itself gets converted to a fiber connection.
- ppp0 is a Tier 2 WAN interface via LTE. In this case the pingerd does not get an error, so the default gateway doesnt change.
- DNS is unbound using DoT and including blocklists.
- There is also surricata including etpro-telemetry and intrusion prevention. Promiscuous mode is enabled, not sure if needed because it runs on igb4 and ppp0 and those intefaves does not use VLANs.
 
But, this setup worked perfectly for a quite long time.

There is also a problem with ssh. If log into the firewall I'm no longer able to use the menu:

Code: [Select]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8
I input a number, the server responds, I see the number, the cursor goes to the start of the actual line an nothing happens.

I did the following:

- Updated to 23.1 via WebGUI. This situation.
- Reinstalled from USB (zfs) and imported the configuration, same thing.
- Installed 22.7.11 from USB and imported the configuration. Everthing was fine.
- Two days later update to 23.1 via ssh-console. Then updated to hotfix 23.1_6. Same problem. Figured out that dis- and re-enabling of igb3 helps.
- Got out of ideas and registered to the forum

I'm quite sure this is some kind of rare bug and not a configuration problem because it always worked and it's reproduceable, but as mentioned, I got out of ideas.