OPNsense 21.1 for ARM self destructs

[dgktkr]

I've been running OPNsense20.7 since August of 2020 without issue and I figured it was time to upgrade to 21.1. So I built OPNsense 21.1 for my ARM device (https://github.com/opnsense/tools) using the same configuration files I used for 20.7.

The storage medium used is an SD card.

After a fresh build according to the instructions  and install on the SD card, OPNsense seems to boot up and run OK, but if all I do is use the web GUI to reboot (that is, make no changes) the second boot fails.

On the serial console the output from u-boot and ubldr seem normal, but after control is passed to the kernel:

Code: [Select]

Kernel entry at 0xc00180...
Kernel args: (null)
nothing happens.

So, it seems that there must have been some changes made to the contents of the SD card. Sure enough, if I compare the SD image before and after (using wxHexEditor), extensive changes are apparent. wxHexEditor stops counting at 250,000 bytes differing.


1) Why is OPNsense making extensive changes to the SD card?

I would've thought it would be good not to make changes to the flash media and do the logs and other frequently written stuff only in DRAM.

Is there a build setting I've missed, but should be using?


2) The next mystery is what is OPNsense overwriting to make the system unbootable. In other words, what all is necessary to get the kernel boot started to the point where it starts emitting output to the serial console?

For the situation I'm describing here, the u-boot area (the first 1MB of the image), the UFS file system (it can be mounted on a running FreeBSD system), the kernel, ubldr.bin and the device tree all appear intact.

[dgktkr]

Well, I did some digging and it looks like loader.conf is the culprit. For a fresh build, the contents are:

Code: [Select]

##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

kern.cam.boot_delay="10000"

after nothing but a reboot, the contents are changed to

Code: [Select]

##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

# dynamically generated console settings follow
#comconsole_speed
#boot_multicons
#boot_serial
#kern.vty
#console

# dynamically generated tunables settings follow
hw.ixl.enable_head_writeback="0"
net.enc.in.ipsec_bpf_mask="2"
net.enc.in.ipsec_filter_mask="2"
net.enc.out.ipsec_bpf_mask="1"
net.enc.out.ipsec_filter_mask="1"
net.inet.icmp.reply_from_interface="1"
net.local.dgram.maxdgram="8192"
vfs.read_max="32"
net.inet.ip.portrange.first="1024"
net.inet.tcp.blackhole="2"
net.inet.udp.blackhole="1"
net.inet.ip.random_id="1"
net.inet.ip.sourceroute="0"
net.inet.ip.accept_sourceroute="0"
net.inet.icmp.log_redirect="0"
net.inet.tcp.drop_synfin="1"
net.inet6.ip6.redirect="1"
net.inet6.ip6.use_tempaddr="0"
net.inet6.ip6.prefer_tempaddr="0"
net.inet.tcp.syncookies="1"
net.inet.tcp.recvspace="65228"
net.inet.tcp.sendspace="65228"
net.inet.tcp.delayed_ack="0"
net.inet.udp.maxdgram="57344"
net.link.bridge.pfil_onlyip="0"
net.link.bridge.pfil_local_phys="0"
net.link.bridge.pfil_member="1"
net.link.bridge.pfil_bridge="0"
net.link.tap.user_open="1"
kern.randompid="347"
net.inet.ip.intr_queue_maxlen="1000"
hw.syscons.kbd_reboot="0"
hw.uart.console="io:0x3f8,br:115200"
net.inet.tcp.log_debug="0"
net.inet.icmp.icmplim="0"
net.inet.tcp.tso="1"
net.inet.udp.checksum="1"
kern.ipc.maxsockbuf="4262144"
vm.pmap.pti="1"
hw.ibrs_disable="0"
security.bsd.see_other_gids="0"
security.bsd.see_other_uids="0"
net.inet.ip.redirect="0"
net.inet.icmp.drop_redirect="1"

Now, if the SD card is removed from the SBC and mounted on a live FreeBSD system, the lines for the tunables can be deleted. After that is done, the SDC can again be booted successfully.

So the problem appears to be the tunables.

It'll take me a while to figure out what to do about this.

[Joonas42]

I think I saw the same reboot issue when I complied my ARM VMWare image and running OPNsense on RPI4. YRZR seems to have found quick fix for the issue. https://www.yrzr.tk/opnsense-images-for-aarch64/#21-reboot-issue-must-read

So before first reboot run in command line:

Code: [Select]

echo 'hw.uart.console=""' > /boot/loader.conf.local
This worked for me and after this I can reboot normally. Maybe something to fix in the source code.

[dgktkr]

I think I saw the same reboot issue when I complied my ARM VMWare image and running OPNsense on RPI4. YRZR seems to have found quick fix for the issue. https://www.yrzr.tk/opnsense-images-for-aarch64/#21-reboot-issue-must-read

So before first reboot run in command line:

Code: [Select]

echo 'hw.uart.console=""' > /boot/loader.conf.local
This worked for me and after this I can reboot normally. Maybe something to fix in the source code.

Hi Joonas42,

I was wondering whether anyone else had encountered this.

Another solution is to go to the Web GUI immediately after the initial boot up to System:Settings:Tunables and delete  the tunable hw.uart.console="io:0x3f8,br:115200" and save.

I also see that if you are running off flash memory, like an SD card, you can go to the Web GUI System:Settings:Miscellaneous and choose to have /var and /tmp on a RAM disk.

[franco]

"Self destruct" is a bit misleading for a console setting change, isn't it? hw.uart.console was added in 21.1 for amd64. It can be stripped from ARM image with no problem. That's what the build tools are for.

Note that we are still not there to fully support ARM.


Cheers,
Franco

[franco]

PS: https://github.com/opnsense/tools/commit/8fa1eafb36

[Joonas42]

Thanks franco for the quick fix.

ARM might not be fully supported but OPNsense works quite well if you just compile it and the packages yourself. Only thing really missing from official/half support is just the package and update repositories are missing. Little bit of hassle to host own repo's for that.

[franco]

> just the package and update repositories are missing

That is the herculean amount of work and energy providing steady package updates really

There's a PR here waiting for more feedback to be processed https://github.com/opnsense/tools/pull/222

At least if we could get ARM64 into a self-contained state it would be easier to maintain and fix in the future.


Cheers,
Franco

[dgktkr]

"Self destruct" is a bit misleading for a console setting change, isn't it? hw.uart.console was added in 21.1 for amd64. It can be stripped from ARM image with no problem. That's what the build tools are for.

Note that we are still not there to fully support ARM.


Cheers,
Franco

Hi Franco,

I didn't mean to imply that you or your team officially supported ARM.  I'm appreciative of and thankful for all the work that has been put into OPNsense and that it can be run on ARM with relatively minor tweaks even though your primary target is Intel hardware.

The point of the post was to share the issue with other users of ARM and share any solution(s).

As for "self destruct", the problem doesn't just affect console output. It appears that it causes the system to fail to reboot.

A test that seemed to indicate that booting fails after that one line appears in loader.conf: wait one minute, which is long enough for a full reboot on my device, and try to access the Web GUI and also try to ssh in (which was enabled in the GUI). Both fail. That says to me that it is not just console output that is affected.

I'll concede that "fails to reboot" would have been a better choice for the title.

[franco]

Hi dgktkr,

No worries, I suppose it would be bad if a sysctl halts/crashes the kernel but since we can't look and it quacks like a duck it's fair to assume the situation should be avoided so I hope the patch does just that.


Cheers,
Franco