nano image slices

[vince]

Hi there!

I am currently testing with OPNsense and the nano image. One of the main reasons I choose this image is the two slices setup that I like very much. The OPNsense website does not hold much information regarding the comparison of full vs nano, however the pfsense docs do (see link below). Now my question is: does that still hold true or did it change since the fork?

Regards, Vince

https://doc.pfsense.org/index.php/Full_Install_and_NanoBSD_Comparison

[franco]

Hi Vince,

The second slice was never operational ever since OPNsense started in 2015, hoping somebody would want to pick up the work. It was removed in 2017 in favour of an "auto-grow" Nano image and to reclaim the unused space.

It did not look like the second slice was a hard requirement for stability in the years in between.


Cheers,
Franco

[vince]

Thanks for your quick reply! That auto-growth seems to be the thing giving us trouble, always corrupting our sd card if we use anything newer than 16.7 for installation. We will probably switch to a different installation type then.

edit: does it make a difference if we install onto the sd card from a different system than the one it will be used with later? (e.g. hw-specific optimizations etc.)

[franco]

The install system doesn't matter.

It's possible to disable the auto-grow upon first boot if you can manage to mount the root partition and delete the following file:

/.probe.for.growfs.nano

Depending on the hardware you are using, the auto-grow may not be the problem but rather FreeBSD 11.0. Some people report the reboot afterwards does not work, auto-grow or not.

Then again, it may make more sense to tailor an image for your SD cards, the build system can do that. e.g. "make nano-30G", or use raw disk images (no auto-grow by default) with "make vm-raw,30G".

https://github.com/opnsense/tools


Cheers,
Franco

[vince]

We are using PC Engines APU1D4 and we did have problems with usb flash drives quickly getting corrupted and using sd cards solved that for one of them somehow. We might have started with a 16.7 install as well, I'm not sure anymore, but we did successfully upgrade it to 17.1 and lately to 17.7. We did have some more problems, however I'm not sure they're version specific.
Now we've started setting up a new set of boxes and they all fail after the first boot. Using the vga installer on another system seems to have solved that, but I dont' have the best feeling about using those boxes in production anymore.
What might be the way to go here? Use 16.7 for now, as I remember it being FreeBSD 10.0 based, and wait for a FreeBSD >11 based image? Order different hardware? And if so is there a vendor you can recommend? (at least 2 Gbit NICs required, 3 would be preferrable) What's the issue with FreeBSD 11.0 here anyway?

[franco]

Hi Vince,

If you have a reproducible setup, can you try the following image?

https://pkg.opnsense.org/snapshots/OPNsense-17.7-test1-OpenSSL-nano-amd64.img.bz2

I've looked at the Nano image changes from 16.7 -> 17.1 and noticed something we haven't tried changing back yet.

The first boot went quite slow, the reboot after that was better. No problem with auto-grow as far as I could see, but we could also try to remove that in another test.

I'm all for fixing this. It's just that building images is hard and very few have helped to analyse the images and provide actionable advice. Any help is highly appreciated!


Thanks,
Franco

[vince]

Hi Franco,

sure, right now we have a few spare boxes around that I can test with.
It's the same thing with this image as well though: first boot works fine and then on reboot I get "USB transmission failed". On our devices the internal sd-card reader is attached via USB.

Regards, Vince

[tillsense]

Hi,

i have today the nano 17.7 image on an ancient alix 2d13 with cf (4GB) installed without any problems with auto-grow.

@vince
How do you write the image to sd?

cheers till

[vince]

It seems to have something to do with the specific hardware we use, as I remember reading quite a few posts about problems with it.

About writing the image, well, dd of course. I think we used bs=1M instead of the bs=16k though.

[franco]

Hi Vince,

I use bs=1M, it shouldn't matter for writing the final image.

I have another test image now after digging through more documentation and build scripts:

https://pkg.opnsense.org/snapshots/OPNsense-17.7-test3-OpenSSL-nano-amd64.img.bz2


Cheers,
Franco

[vince]

Hi Franco

thank you for putting in all that effort! I just tested your image and it appears to be the same error, shown over and over again:

Code: [Select]

WARNING - Timeout at ehci_wait_td:517!
ehci pipe=0x000eee80 cur=000efdc0 tok=801f0c81 next=1 td=0x0000fdc0 status=1f0c80
USB transmission failed

I don't know if that helps, I just thought I might post it since I do not remember that warning from my earlier tests.

Regards, Vince

[franco]

Can you try this one more time? But before reboot add the following to /boot/loader.conf.local

kern.cam.boot_delay="10000"

This really annoys me.


Thanks,
Franco

[vince]

Yields the exact same result.

Regards,
Vince

[franco]

Hi Vince,

Thank you for your testing. It is really appreciated.

Ok, last overhaul here, I have completely changed the assembly to copy the Serial images 1 to 1 for the Nano image... If this doesn't work it is very likely a kernel issue on FreeBSD 11.0 as we've emptied the whole range of image assembly tools available to us. :/

https://pkg.opnsense.org/snapshots/OPNsense-17.7.1-OpenSSL-nano-amd64.img.bz2


Cheers,
Franco

[vince]

You're welcome. Same thing again though.

What does work though is to use an installer on another system, write to an sd card and use that card in the APU1D4.