Messages

I'm trying to make sense of what I see on the graphs show on the Reporting :Traffic page.

So I start playing a Netflix movie on my local machine and look at the OPNsense graphs in the Web GUI.

https://pasteboard.co/JYHGtem.png

First of all -- negative and positive bps? What's up with that?

Then, the x axis scaling is different for the total and top hosts graphs. That makes comparison not so easy.

And shouldn't adding the numbers for the top hosts shown at the bottom of the window add up to the total shown in the graphs at the top?

"Self destruct" is a bit misleading for a console setting change, isn't it? hw.uart.console was added in 21.1 for amd64. It can be stripped from ARM image with no problem. That's what the build tools are for.

Note that we are still not there to fully support ARM.


Cheers,
Franco

Hi Franco,

I didn't mean to imply that you or your team officially supported ARM.  I'm appreciative of and thankful for all the work that has been put into OPNsense and that it can be run on ARM with relatively minor tweaks even though your primary target is Intel hardware.

The point of the post was to share the issue with other users of ARM and share any solution(s).

As for "self destruct", the problem doesn't just affect console output. It appears that it causes the system to fail to reboot.

A test that seemed to indicate that booting fails after that one line appears in loader.conf: wait one minute, which is long enough for a full reboot on my device, and try to access the Web GUI and also try to ssh in (which was enabled in the GUI). Both fail. That says to me that it is not just console output that is affected.

I'll concede that "fails to reboot" would have been a better choice for the title.

I think I saw the same reboot issue when I complied my ARM VMWare image and running OPNsense on RPI4. YRZR seems to have found quick fix for the issue. https://www.yrzr.tk/opnsense-images-for-aarch64/#21-reboot-issue-must-read

So before first reboot run in command line:

Code Select Expand

echo 'hw.uart.console=""' > /boot/loader.conf.local

This worked for me and after this I can reboot normally. Maybe something to fix in the source code.

Hi Joonas42,

I was wondering whether anyone else had encountered this.

Another solution is to go to the Web GUI immediately after the initial boot up to System:Settings:Tunables and delete  the tunable hw.uart.console="io:0x3f8,br:115200" and save.

I also see that if you are running off flash memory, like an SD card, you can go to the Web GUI System:Settings:Miscellaneous and choose to have /var and /tmp on a RAM disk.

Well, I did some digging and it looks like loader.conf is the culprit. For a fresh build, the contents are:

Code Select Expand

##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

kern.cam.boot_delay="10000"


after nothing but a reboot, the contents are changed to

Code Select Expand

##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

# dynamically generated console settings follow
#comconsole_speed
#boot_multicons
#boot_serial
#kern.vty
#console

# dynamically generated tunables settings follow
hw.ixl.enable_head_writeback="0"
net.enc.in.ipsec_bpf_mask="2"
net.enc.in.ipsec_filter_mask="2"
net.enc.out.ipsec_bpf_mask="1"
net.enc.out.ipsec_filter_mask="1"
net.inet.icmp.reply_from_interface="1"
net.local.dgram.maxdgram="8192"
vfs.read_max="32"
net.inet.ip.portrange.first="1024"
net.inet.tcp.blackhole="2"
net.inet.udp.blackhole="1"
net.inet.ip.random_id="1"
net.inet.ip.sourceroute="0"
net.inet.ip.accept_sourceroute="0"
net.inet.icmp.log_redirect="0"
net.inet.tcp.drop_synfin="1"
net.inet6.ip6.redirect="1"
net.inet6.ip6.use_tempaddr="0"
net.inet6.ip6.prefer_tempaddr="0"
net.inet.tcp.syncookies="1"
net.inet.tcp.recvspace="65228"
net.inet.tcp.sendspace="65228"
net.inet.tcp.delayed_ack="0"
net.inet.udp.maxdgram="57344"
net.link.bridge.pfil_onlyip="0"
net.link.bridge.pfil_local_phys="0"
net.link.bridge.pfil_member="1"
net.link.bridge.pfil_bridge="0"
net.link.tap.user_open="1"
kern.randompid="347"
net.inet.ip.intr_queue_maxlen="1000"
hw.syscons.kbd_reboot="0"
hw.uart.console="io:0x3f8,br:115200"
net.inet.tcp.log_debug="0"
net.inet.icmp.icmplim="0"
net.inet.tcp.tso="1"
net.inet.udp.checksum="1"
kern.ipc.maxsockbuf="4262144"
vm.pmap.pti="1"
hw.ibrs_disable="0"
security.bsd.see_other_gids="0"
security.bsd.see_other_uids="0"
net.inet.ip.redirect="0"
net.inet.icmp.drop_redirect="1"


Now, if the SD card is removed from the SBC and mounted on a live FreeBSD system, the lines for the tunables can be deleted. After that is done, the SDC can again be booted successfully.

So the problem appears to be the tunables.

It'll take me a while to figure out what to do about this.

I've been running OPNsense20.7 since August of 2020 without issue and I figured it was time to upgrade to 21.1. So I built OPNsense 21.1 for my ARM device (https://github.com/opnsense/tools) using the same configuration files I used for 20.7.

The storage medium used is an SD card.

After a fresh build according to the instructions  and install on the SD card, OPNsense seems to boot up and run OK, but if all I do is use the web GUI to reboot (that is, make no changes) the second boot fails.

On the serial console the output from u-boot and ubldr seem normal, but after control is passed to the kernel:

Code Select Expand

Kernel entry at 0xc00180...
Kernel args: (null)

nothing happens.

So, it seems that there must have been some changes made to the contents of the SD card. Sure enough, if I compare the SD image before and after (using wxHexEditor), extensive changes are apparent. wxHexEditor stops counting at 250,000 bytes differing.


1) Why is OPNsense making extensive changes to the SD card?

I would've thought it would be good not to make changes to the flash media and do the logs and other frequently written stuff only in DRAM.

Is there a build setting I've missed, but should be using?


2) The next mystery is what is OPNsense overwriting to make the system unbootable. In other words, what all is necessary to get the kernel boot started to the point where it starts emitting output to the serial console?

For the situation I'm describing here, the u-boot area (the first 1MB of the image), the UFS file system (it can be mounted on a running FreeBSD system), the kernel, ubldr.bin and the device tree all appear intact.

Thanks jasonmc, that was it. The behavior occurred because the protocol selected was the default "any". When changed to "TCP/UDP" the buttons under "From:" and "To:" became active. That allowed a choice of a well known port or an alias of my own making.

dgktkr

Does it work when you assign an alias in the port, where that alias consists of that port range in question?
An I assume you have defined as protocol either tcp or udp, right?

Hi,

I've succeeded in bringing up 20.7 on my arm device (Clearfog Base) and so far most everything functions as expected.

An exception is setting a port range for source or destination in a firewall rule in the http GUI. While on a page generated by firewall_rules_edit.php the section "Destination port range" shows up with "From:" and "To:", but neither field will accept input. The fields below those labelled "any" are grayed out and show a "not allowed" symbol (circle with a slash) when the mouse cursor hovers over them.

The same behavior occurs with Safari, Firefox and Chrome. I don't know about Edge or IE since we're an Apple household.

Am I doing something wrong, or is this a bug?

dgktkr

Hi,

Noob here. I'm trying explore syslog capabilities in 20.1 (with FreeBSD 12.1) by editing /var/etc/syslog.conf and restarting the service. The problem is that the restart rewrites syslog.conf and wipes out the modifications.

Where can I put edits so that they survive a restart? I've looked through various /etc rc files and I can't find where rewriting the conf file is implemented.

FWIW plain FreeBSD 12.1 doesn't seem to do that to /etc/syslog.conf.

[/var/log]

Hi,

I'm running OPNsense 20.1 built from source for an arm device and things generally seem to work.

One thing that doesn't seem to work is logging for pf. It can be enabled in the web GUI, but nothing shows up when using the GUI to view the pf logs.

pf seems to be working properly because

Code Select Expand

tcpdump -n -e -ttt -i pflog0 shows the expected information on filtered packets.

On the other hand, the usual logger daemon doesn't seem to be running:

Code Select Expand

# ps -auxww | grep pflogd
root    37336   0.0  0.2  4632  2088  0  S+   12:33    0:00.01 grep pflogd and there isn't any apparent log file for pf in /var/log. There's a bunch of other log files, for instance dhcpd.log, that are 511488B in size that look like circular logs.

Has a circular log for pf been implemented in 20.1 source code yet?

Edit: Further investigation seems to reveal that pf logging in OPNsense doesn't use the usual log daemon or log file. Instead, it looks like /usr/local/sbin/filterlog and /var/log/filter.log are used:

Code Select Expand

# ps -auxww | grep filterlog
root    73740   0.0  0.2  5152  2104  -  Ss   12:14     0:02.73 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root    64192   0.0  0.2  4640  2096  0  S+   16:06     0:00.01 grep filterlog

But no info is shown up in filter.log:

Code Select Expand

root@OPNsense:/var/log # clog filter.log
root@OPNsense:/var/log # even thought tcpdump shows pflog0 pushing out info at a good rate.

So, is filterlog not working as expected? If not, why not?

Do OPNsense builds for arm devices have logging disabled somehow? When using OPNsense as a gateway, "Inspect" shows large number of packets being blocked for some rules, but nothing shows up in the logs even though logging has been selected for those rules.

rene_,

It looks like you have gotten further in the boot sequence than I have. For ClearFog Base, I've gotten OPNsense base, kernel, packages and arm-3G to build without error. u-boot and ubldr.bin work as expected, but as soon as control is passed to the kernel, the serial console goes non-responsive. I've seen behavior like this before, which turned out to be a device tree/driver problem. The configuration for the serial console in the device tree needed to be changed. That modification works for FreeBSD 12.0 and 12.1, but, so far, I haven't been able to get the console to work with FreeBSD 11.2.

Currently, I'm trying to figure out how to get FreeBSD 11.2 to work with ClearFog Base's console. I may just have to wait a couple of months until OPNsense is migrated to 12.1.

dgktkr

[packages]

After the suggestions by nekoprog and franco were followed, the packages target was successfully built.

Now the problem is when building the arm image

Code Select Expand

make arm-3G DEVICE=CLEARFOG

the build errors out complaining

Code Select Expand

ln: /usr/obj/usr/tools/config/19.7/OpenSSL:armv6/boot/dtb/clearfog.dtb: No such file or directory

even though I put a valid clearfog.dtb there before the build. So where do I put a valid dtb file (or valid dts file if that's what is required) so that the build finds it?

dgktkr

Hi rene_,

It does seem strange that your SG3100 can output characters to your terminal emulator, but it can't read input from it. I'm wondering if an appropriate dtb file is being used. Can you (repeatedly) press a key during early boot to get to the u-boot shell? If so, can you type "bootcmd" and get the OS to boot? That should verify if serial communication is working. For proper functioning of the OS, the u-boot code and the dtb file for the OS have to have precise customizations for your particular device.

Hopefully, someone with more knowledge can give you more detailed advice.

dgktkr

[/usr/local/opnsense/build/19.7/armv6/sets]

Hi nekoprog,

Base and kernel for ClearFog have been built successfully thanks to your tips. The corresponding tgz files show up in /usr/local/opnsense/build/19.7/armv6/sets.

The packages build

Code Select Expand

make -j4 packages DEVICE=CLEARFOG errors out after many hours. If the command is repeated, it errors out in a few seconds. Taking out the -j4 option gets farther, but it runs into an error I've made (leaving out suricata as well as suricata-devel). I'll fix that and try again.

dgktkr

Hi,

When using FreeBSD 11.2 as the host, make base succeeds, but I'm wondering about the compiles. The build output to the console shows the message

Code Select Expand

SYSTEM_COMPILER: Determined that CC=cc matches the source tree. Not bootstrapping a cross-compiler.

Is code being cross-compiled? If not, is there some environment variable that has to be set so that cross-compiling is done?

dgktkr