Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - mfalkvidd

#1
Quote from: franco on February 27, 2026, 08:06:23 AMThen the best option is the live boot into 25.7 to see if that kernel works (but I don't see a reason why it shouldn't) and install from there. I wouldn't recommend 26.1 from such an old install because it adds the ISC removal complication to the mix. Don't forget there is a config import feature in the installer.

Sound reasonable. Thanks! I'll do that as soon as I can schedule some downtime (don't have a HA setup).

I just noticed that /boot/kernel does not exist. So that's probably why the system says `Failed to load kernel 'kernel'`.
If i run `opnsense-update -u -V` the output ends like this:

+ echo -n 'Extracting kernel-23.7-amd64.txz...'
Extracting kernel-23.7-amd64.txz...+ rm -rf '/var/cache/opnsense-update/.sets.pending/kernel-*'
+ mkdir -p /var/cache/opnsense-update/.sets.pending
+ mv /var/cache/opnsense-update/69245/kernel-23.7-amd64.txz /var/cache/opnsense-update/.sets.pending
+ echo 23.7
+ echo ' done'
done
+ [ -n -k -a -z -u ]
+ [ -n -b -a -z -u ]
+ [ -b '=' -b -a -z -u ]
+ [ -n '' ]
+ [ -p '=' -P -a -z -u ]
+ [ -z '' ]
+ rm -rf /var/cache/opnsense-update/652 /var/cache/opnsense-update/69245
+ [ -k '=' -k -o -b '=' -b -o -n -u ]
+ echo 'Please reboot.'
Please reboot.

/var/cache/opnsense-update/.sets.pending/kernel-23.7-amd64.txz tests fine using tar -tf, and contains a kernel so the 25.1 live boot sounds like the most efficient way forward.
#2
Ran opnsense-update -u which completed ok. Rebooted and got same error as the first time (can't load kernel).
#3
Thanks. Will do, but through cli instead of web so I can see what is happening.

How can I check the disk, besides scrubbing the pool?
#4
I'm trying to catch up with the latest release. Two weeks ago I upgraded from v22 to v23. Had to dig out a serial cable so I could (re)configure the interfaces over serial, but except that the upgrade was fine.

Today I initiated the update from 23.1.11_2 to 23.7 through the web UI. It indicated that base, kernel and packages needed update.
The web UI stalled at the kernel update.
An hour later I could still access internet from my devices (through Opnsense). Opnsense still responded to ping. But ssh hanged (no connection timeout, no connection refused, just hanging indefinitely). Same with web interface.
Serial console gave me login prompt, but after providing the password nothing happened.

I pulled the power, waited a bit and powered on again. Got this:

PC Engines apu4
coreboot build 20212402
BIOS version v4.13.0.4
4080 MB ECC DRAM

SeaBIOS (version rel-1.12.1.3-0-g300e8b70)

Press F10 key now for boot menu

Booting from Hard Disk...



            /  __  |/ ___ |/ __  |
            | |  | | |__/ | |  | |___  ___ _ __  ___  ___
            | |  | |  ___/| |  | / __|/ _ \ '_ \/ __|/ _ \
            | |__| | |    | |  | \__ \  __/ | | \__ \  __/
            |_____/|_|    |_| /__|___/\___|_| |_|___/\___|

 +-----------------------------------------+     @@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |  1. Boot Multi user [Enter]             |   @@@@@                    @@@@@
 |  2. Boot Single user                    |       @@@@@            @@@@@
 |  3. Escape to loader prompt             |    @@@@@@@@@@@       @@@@@@@@@@@
 |  4. Reboot                              |         \\\\\         /////
 |  5. Cons: Serial                        |   ))))))))))))       (((((((((((
 |                                         |         /////         \\\\\
 |  Options:                               |    @@@@@@@@@@@       @@@@@@@@@@@
 |  6. Kernel: default/kernel (1 of 2)     |       @@@@@            @@@@@
 |  7. Boot Options                        |   @@@@@                    @@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@
 +-----------------------------------------+
   Autoboot in 0 seconds. [Space] to pause     23.1 ``Quintessential Quail'' |

Loading kernel...
Failed to load kernel 'kernel'
can't load 'kernel'

can't load 'kernel'

I proceeded with boot kernel.old which was successful (except that it was still on 23.1 of course). This error was noted during boot though:
swapon: adding /dev/ada0p3 as swap device
.ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'upgrade'
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Version number mismatch, aborting.
    Kernel: 13.1
    Base:   13.2
>>> Invoking early script 'configd'
Starting configd.

There is 3.3GB available space on zroot. I did a scrub two weeks ago which resulted in 0B repaired and 0 errors.

Any ideas on how to proceed from here?
#5
Thanks @pmhausen, that was it! The remote host was connected to the LAN through the wireless adapter, so all traffic from the remote host was delivered locally, without passing through opnsense. No wonder the state table couldn't keep up.

After turning off the wlan adapter, I can use ssh without keepalives and with Firewall Optimization set to "normal".
#6
I added ClientAliveInterval 15 and ClientAliveCountMax 3 to sshd_config on the remote host and restarted sshd. Keepalives are posted every 15 seconds, but the firewall still expires the connection:
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
client_loop: send disconnect: Connection reset by peer



Edit: same, but with the conservative setting:

Apr 08 17:58:34 debug1: Sending command: while true; do uptime; sleep 60; done
Apr 08 17:58:34  17:58:33 up 29 days, 22:35,  1 user,  load average: 0.57, 0.16, 0.04
Apr 08 17:58:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 17:59:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 17:59:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 17:59:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 17:59:34  17:59:33 up 29 days, 22:36,  1 user,  load average: 0.21, 0.13, 0.04
Apr 08 17:59:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:00:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:00:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:00:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:00:34  18:00:33 up 29 days, 22:37,  1 user,  load average: 0.12, 0.12, 0.04
Apr 08 18:00:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:01:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:01:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:01:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:01:34  18:01:33 up 29 days, 22:38,  1 user,  load average: 0.04, 0.10, 0.03
Apr 08 18:01:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:02:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:02:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:02:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:02:34  18:02:33 up 29 days, 22:39,  1 user,  load average: 0.02, 0.08, 0.03
Apr 08 18:02:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:03:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:03:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:03:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:03:34  18:03:33 up 29 days, 22:40,  1 user,  load average: 0.00, 0.06, 0.02
Apr 08 18:03:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:04:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:04:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:04:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:04:34  18:04:33 up 29 days, 22:41,  1 user,  load average: 0.05, 0.06, 0.02
Apr 08 18:04:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:05:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:05:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:05:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:05:34  18:05:33 up 29 days, 22:42,  1 user,  load average: 0.02, 0.05, 0.02
Apr 08 18:05:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:06:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:06:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:06:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:06:34  18:06:33 up 29 days, 22:43,  1 user,  load average: 0.00, 0.04, 0.01
Apr 08 18:06:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:07:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:07:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:07:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:07:34  18:07:33 up 29 days, 22:44,  1 user,  load average: 0.00, 0.03, 0.00
Apr 08 18:07:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:08:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:08:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:08:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:08:34  18:08:33 up 29 days, 22:45,  1 user,  load average: 0.05, 0.04, 0.00
Apr 08 18:08:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:09:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:09:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:09:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:09:34  18:09:33 up 29 days, 22:46,  1 user,  load average: 0.02, 0.03, 0.00
Apr 08 18:09:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:10:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:10:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:10:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:10:34  18:10:33 up 29 days, 22:47,  1 user,  load average: 0.00, 0.02, 0.00
Apr 08 18:10:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:11:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:11:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:11:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:11:34  18:11:33 up 29 days, 22:48,  1 user,  load average: 0.00, 0.02, 0.00
Apr 08 18:11:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:12:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:12:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:12:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:12:34  18:12:33 up 29 days, 22:49,  1 user,  load average: 0.00, 0.01, 0.00
Apr 08 18:12:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:13:04 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:13:19 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:13:34 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:13:34  18:13:33 up 29 days, 22:50,  1 user,  load average: 0.00, 0.00, 0.00
Apr 08 18:13:49 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Apr 08 18:14:08 client_loop: send disconnect: Connection reset by peer
#7
I have now verified multiple times that the connection is dropped after approximately 15 minutes.

The tcp.opening parameter is 900s, which is 15 minutes. I wonder if that is a coincidence.

If I change back to "normal", tcp.opening becomes 30s, which matches pretty good with the timeouts I had in the beginning.

So it seems the firewall closes the connection after the tcp.opening timer expires, instead of identifying the traffic as established.

I don't think ClientAlive/ServerAlive settings will make any difference, since there is continuous traffic on the ssh connection. Those settings (afaik) are only applicable if you leave the terminal idle. Besides, if I move the host to the same lan, there is no need to tweak the settings.

There are two managed (VLAN-capable) switches in between. But would that make a difference when chainging the timeouts on the firewall change how soon the connection is dropped?
#8
Yup, got kicked off again after 15 minutes.
# pfctl -st
tcp.first                  3600s
tcp.opening                 900s
tcp.established          432000s
tcp.closing                3600s
tcp.finwait                 600s
tcp.closed                  180s
tcp.tsdiff                   60s
udp.first                   300s
udp.single                  150s
udp.multiple                900s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s
#9
Seems like I spoke too soon. The ssh connection was dropped after 15 minutes. Doing another test now to verify if the same happens again.
#10
Thanks! That does seem to do the trick. The ssh connection has been working for 8 minutes now. Before, it never lasted more than about a minute.

I'm a bit curious why this setting works, and how much impact it will have on the firewall performance.
#11
Firewall:Diagnostics:States claims that the connection has been closed. But I can still use the connection when this happens (for a few seconds more). See below image.
#12
I don't think this is specific to ssh connections, but ssh is where I am experiencing the problem.

ssh connection from a host on [LAN] to a host on [Untrusted] is dropped after 30-60 seconds.

In the ssh window, I run `watch uptime` to generate activity every 2 seconds. I can see the terminal updating every 2 seconds for a while, before it freezes. After some more time, my ssh client exits with `client_loop: send disconnect: Connection reset by peer`.

See attached images for data from Firewall: Log Files: Live View.

My setup
internal network [LAN]: 192.168.32.0/24
internal network [Untrusted]: 192.168.33.0/24

Firewall:Rules:LAN has "Default allow LAN to any rule" which allows any IPv4 traffic from LAN to anywhere

If I ssh from LAN to a server on the internet, the connection works fine.

The host on the [Untrusted] lan sees the real IP of the [LAN] host; there is no NAT performed.

Questions:
1. Why does the initial ssh connection hit the "let out anything from the firewall itself" rule? I thought it would hit the "Default allow LAN to any rule" rule.
2. Isn't opnsense supposed to keep track of the allowed connection, until the connection is closed? Why does the connection hit the "Default deny rule" after a while?

#13
General Discussion / Re: CrowdSec
August 24, 2021, 09:30:06 AM
Anyone using Crowdsec yet?
I installed it, but it says

crowdsec is installed.

You need to edit the agent config file /usr/local/etc/crowdsec/crowdsec.yaml and
enable rc via sysrc.

# sysrc crowdsec_enable="YES"
root@OPNsense:~ # sysrc crowdsec_enable="YES"
awk: can't open file /etc/rc.conf

I am not sure what I need to place in the yaml file.