[Work In Progress] OPNsense Ported into ARM Devices

[nekoprog]

Hi,

I have a ClearFog Base which is based on an armv7 SOC, but it can run armv6. I've gotten FreeBSD 11.2, 12.0 and 12.1 to boot and run on it with both ethernet ports working properly.

I've followed nekoprog's instructions https://github.com/nekoprog/pieSense/blob/master/README.md with appropriate modifications.

In the /usr/tools/Makefile the line

Code Select Expand

ARCH?= ${_ARCH}

was replaced by

Code Select Expand

ARCH?= arm:armv6

and the line

Code Select Expand

DEVICE?= A10

was replaced by

Code Select Expand

DEVICE?= CLEARFOG.


The file /usr/tools/device/CLEARFOG.conf was created and modeled after BANANAPI.conf:

Code Select Expand


export MAKE_ARGS_DEV="
CROSS_BINUTILS_PREFIX=/usr/local/arm-gnueabi-freebsd11.2/bin
UBLDR_LOADADDR=0x900000
"

export PRODUCT_TARGET=arm
export PRODUCT_ARCH=armv6
export PRODUCT_WANTS="arm-gnueabi-binutils qemu-user-static u-boot-clearfog"

export ARM_FAT_SIZE="32m -b 1m"
export ARM_UBOOT_DIR="/usr/local/share/u-boot/u-boot-clearfog"

arm_install_uboot()
{
arm_unmount
dd if=${ARM_UBOOT_DIR}/u-boot-spl.kwb of=/dev/${DEV} bs=1k seek=8 conv=sync
arm_mount
cp ${ARM_UBOOT_DIR}/boot.scr ${STAGEDIR}/boot/msdos
cp -p ${STAGEDIR}/boot/ubldr.bin ${STAGEDIR}/boot/msdos
ln ${STAGEDIR}/boot/dtb/clearfog.dtb ${STAGEDIR}/boot/dtb/armada-388-clearfog.dtb
}


Code Select Expand

#pkg install git qemu-user-static arm-gnueabi-binutils u-boot-clearfog


was invoked and completed successfully.

When

Code Select Expand

/usr/tools #make base

is executed, the computer works for quite a while until the build fails with a relocation overflow. The first sign of trouble is

Code Select Expand

===> usr.bin/clang/clang (all)
cc1_main.o: In function `cc1_main(llvm::ArrayRef<char const*>, char const*, void*)':
cc1_main.cpp:(.text._Z8cc1_mainN4llvm8ArrayRefIPKcEES2_Pv+0x45c): relocation truncated to fit: R_ARM_CALL against symbol `llvm::install_fatal_error_handler(void (*)(void*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool), void*)' defined in .text._ZN4llvm27install_fatal_error_handlerEPFvPvRKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEbES0_ section in /usr/obj/arm.armv6/usr/src/lib/clang/libllvm/libllvm.a(ErrorHandling.o)


This problem looks like the same issue reported and supposedly solved about 11 months ago https://github.com/opnsense/tools/issues/113

Running

Code Select Expand

/usr/tools #make xtools


before

Code Select Expand

/usr/tools # make base


doesn't solve the problem. And after building xtools, running

Code Select Expand

/usr/tools #make packages


fails in the same place in the same way as

Code Select Expand

/usr/tools #make base.

Was the fix that was implemented in tools last December propagated to the current version at https://github.com/opnsense/tools?

This repo has been archived and only works on OPNsense 18.01 where building for arm is not yet supported.  What version of FreeBSD you're using? This line should be edited too following your FreeBSD version.
CROSS_BINUTILS_PREFIX=/usr/local/arm-gnueabi-freebsd11.2/bin

Instead of editing Makefile, you should just create new device config and call it like this

Code Select Expand

make STEP DEVICE=CLEARFOG

[dgktkr]

Hi nekoprog,

Thanks for the prompt reply. The version of the host FreeBSD was 12.0. I'm downloading 11.2 right now so it can be installed and used to cross build OPNsense for arm.

This repo has been archived and only works on OPNsense 18.01 where building for arm is not yet supported.

Could you elaborate on that? Where should tools be downloaded from to enable a successful build for armv6? From the console output, it looks like it was trying to build OPNsense 19.7.

dgktkr

[dgktkr]

Another question: Where does one specify KERNCONF for the kernel build of the target SOC, in my case: /usr/src/sys/arm/conf/ARMADA38X? For bananapi it seemed to be specified in older versions of BANANAPI.conf as KERNEL=ALLWINNER https://github.com/opnsense/src/blob/8d1a9170bd4d9d50ffeecc2c8366ec34b8ebdb02/release/arm/BANANAPI.conf#L6-L18.

dgktkr

[nekoprog]

Hi nekoprog,

Thanks for the prompt reply. The version of the host FreeBSD was 12.0. I'm downloading 11.2 right now so it can be installed and used to cross build OPNsense for arm.

This repo has been archived and only works on OPNsense 18.01 where building for arm is not yet supported.

Could you elaborate on that? Where should tools be downloaded from to enable a successful build for armv6? From the console output, it looks like it was trying to build OPNsense 19.7.

dgktkr

pieSense repo has been archived, it's not associated to OPNsense and I use it in the past to test for RPI2 build. Current OPNsense/tools config for arm is more robust, just need to add kernel and device configs.

Another question: Where does one specify KERNCONF for the kernel build of the target SOC, in my case: /usr/src/sys/arm/conf/ARMADA38X? For bananapi it seemed to be specified in older versions of BANANAPI.conf as KERNEL=ALLWINNER https://github.com/opnsense/src/blob/8d1a9170bd4d9d50ffeecc2c8366ec34b8ebdb02/release/arm/BANANAPI.conf#L6-L18.

dgktkr

Can you confirm the kernel type? This should be change at https://github.com/opnsense/tools/tree/master/config/19.7 and you need to create new kernel config file, use SMP-BANANAPI for example and rename it to SMP-CLEARFOG and change include ALLWINNER to include ARMADA38X maybe? And ident SMP-BANANAPI to ident SMP-CLEARFOG.

After you have a successful results for BASE KERNEL and PACKAGES, we need to make a minor adjustment to device config for .img creation to make sure it boots into OPNsense later. Here my sample CLEARFOG.conf if you like to test, I have commented out the parts that needed some modifications. Hope you can help with this.

[dgktkr]

Hi,

When using FreeBSD 11.2 as the host, make base succeeds, but I'm wondering about the compiles. The build output to the console shows the message

Code Select Expand

SYSTEM_COMPILER: Determined that CC=cc matches the source tree. Not bootstrapping a cross-compiler.

Is code being cross-compiled? If not, is there some environment variable that has to be set so that cross-compiling is done?

dgktkr

[nekoprog]

Hi,

When using FreeBSD 11.2 as the host, make base succeeds, but I'm wondering about the compiles. The build output to the console shows the message

Code Select Expand

SYSTEM_COMPILER: Determined that CC=cc matches the source tree. Not bootstrapping a cross-compiler.

Is code being cross-compiled? If not, is there some environment variable that has to be set so that cross-compiling is done?

dgktkr

Just ignore it because we set the target arch on device config. If sets are listed here "ls /usr/local/opnsense/build/19.7/armv6/sets", then it's a success. Proceed with next make step.

Don't forget to include DEVICE=CLEARFOG arguments when running "make". Else you will build i386/amd64 arch images.

Forgot to mention you this before. You need to disable suricata-devel when running "make packages DEVICE=CLEARFOG".

[dgktkr]

Hi nekoprog,

Base and kernel for ClearFog have been built successfully thanks to your tips. The corresponding tgz files show up in /usr/local/opnsense/build/19.7/armv6/sets.

The packages build

Code Select Expand

make -j4 packages DEVICE=CLEARFOG errors out after many hours. If the command is repeated, it errors out in a few seconds. Taking out the -j4 option gets farther, but it runs into an error I've made (leaving out suricata as well as suricata-devel). I'll fix that and try again.

dgktkr

[rene_]

Hi,

i think it better to discuss in this thread.

after some start problems, i got opnsense booting on the armada 385 SoC (Netgate SG3100).

But i am not able to type anything in the serial console, and their are no network ports discoverd :-(

any ideas?

Greets,
René

[dgktkr]

Hi rene_,

It does seem strange that your SG3100 can output characters to your terminal emulator, but it can't read input from it. I'm wondering if an appropriate dtb file is being used. Can you (repeatedly) press a key during early boot to get to the u-boot shell? If so, can you type "bootcmd" and get the OS to boot? That should verify if serial communication is working. For proper functioning of the OS, the u-boot code and the dtb file for the OS have to have precise customizations for your particular device.

Hopefully, someone with more knowledge can give you more detailed advice.

dgktkr

[dgktkr]

After the suggestions by nekoprog and franco were followed, the packages target was successfully built.

Now the problem is when building the arm image

Code Select Expand

make arm-3G DEVICE=CLEARFOG

the build errors out complaining

Code Select Expand

ln: /usr/obj/usr/tools/config/19.7/OpenSSL:armv6/boot/dtb/clearfog.dtb: No such file or directory

even though I put a valid clearfog.dtb there before the build. So where do I put a valid dtb file (or valid dts file if that's what is required) so that the build finds it?

dgktkr

[nekoprog]

Hi nekoprog,

Base and kernel for ClearFog have been built successfully thanks to your tips. The corresponding tgz files show up in /usr/local/opnsense/build/19.7/armv6/sets.

The packages build

Code Select Expand

make -j4 packages DEVICE=CLEARFOG errors out after many hours. If the command is repeated, it errors out in a few seconds. Taking out the -j4 option gets farther, but it runs into an error I've made (leaving out suricata as well as suricata-devel). I'll fix that and try again.

dgktkr

By default, "make" is running with how many CPUs you have from the output of "sysctl kern.smp.cpus".

Hi,

i think it better to discuss in this thread.

after some start problems, i got opnsense booting on the armada 385 SoC (Netgate SG3100).

But i am not able to type anything in the serial console, and their are no network ports discoverd :-(

any ideas?

Greets,
René

Need to look further into defconfig or into kernelconf, maybe the port is not loaded on boot. Got full log?

After the suggestions by nekoprog and franco were followed, the packages target was successfully built.

Now the problem is when building the arm image

Code Select Expand

make arm-3G DEVICE=CLEARFOG

the build errors out complaining

Code Select Expand

ln: /usr/obj/usr/tools/config/19.7/OpenSSL:armv6/boot/dtb/clearfog.dtb: No such file or directory

even though I put a valid clearfog.dtb there before the build. So where do I put a valid dtb file (or valid dts file if that's what is required) so that the build finds it?

dgktkr

You need to copy .dtb into kernel.tgz (look for /boot/dtb, create directory if not exist) in case FreeBSD does not include it during "make kernel". Do not copy into STAGEDIR (/usr/obj/usr/tools/config/19.7/OpenSSL:armv6), files will be replaces with base.tgz and kernel.tgz upon running "make arm".

OR

Edit config/19.7/SMP-CLEARFOG, change include ARMADA38X to include GENERIC, then edit device/CLEARFOG.conf, add this line cp -p ${STAGEDIR}/boot/dtb ${STAGEDIR}/boot/msdos/, this will copy all arm device dtb instead of looking only for clearfog. Hope this will solve both you and rene problem.

[rene_]

Hi Nekoprog,

i attached the output of the boot sequence.

It seems, that the kernel drivers for the Marvell switch and interfaces are missing.

i found them in the freebsd source, but dont know how to add them to the build procedure.

i used the default ARMADA-38X kernel config for my build, and put the prebuilt dtb files from the running pfsense image into my arm image. (still wondering this works ^^)

Kind regards,
René

[nekoprog]

Hi Nekoprog,

i attached the output of the boot sequence.

It seems, that the kernel drivers for the Marvell switch and interfaces are missing.

i found them in the freebsd source, but dont know how to add them to the build procedure.

i used the default ARMADA-38X kernel config for my build, and put the prebuilt dtb files from the running pfsense image into my arm image. (still wondering this works ^^)

Kind regards,
René

Have you tried compiling with GENERIC kernel then copy all DTB into /boot/msdos/dtb?

[dgktkr]

rene_,

It looks like you have gotten further in the boot sequence than I have. For ClearFog Base, I've gotten OPNsense base, kernel, packages and arm-3G to build without error. u-boot and ubldr.bin work as expected, but as soon as control is passed to the kernel, the serial console goes non-responsive. I've seen behavior like this before, which turned out to be a device tree/driver problem. The configuration for the serial console in the device tree needed to be changed. That modification works for FreeBSD 12.0 and 12.1, but, so far, I haven't been able to get the console to work with FreeBSD 11.2.

Currently, I'm trying to figure out how to get FreeBSD 11.2 to work with ClearFog Base's console. I may just have to wait a couple of months until OPNsense is migrated to 12.1.

dgktkr

[orangana]

http://wiki.friendlyarm.com/wiki/index.php/NanoPi_R1S-H5

Was there ever a opnsense device for less than $25 which does the job? This could be the first one!  ;D