Resizing embeded version on first boot - ARM64 custom images

[none]

Hi,

I build images using the line:

Code Select Expand

make -C /usr/tools/ VERSION=25.1.4 DEVICE=R4S arm-8G

and then I get my 8GB image ready for dd'ing. It works but the disk is not resized.

I have 32GB and 64GB microSD in use and it keeps the 8GB partition size.

My research took me to the need for the .probe.for.growfs file on /, and the opnsense scripts do create it as is written in extras.conf. The extras.conf I use (https://github.com/matheusber/opnsense/blob/main/arm64-opnsense-build/opnsense-confs/extras.conf) just add the lines to load if_re from Realtek kmod version 1.98 from ports, that I build here.

The disk layout is what opnsense scripts creates. Have anyone delt with this?

Would really help create 1GB image files that would expand to the microSD size after. Will help too much here in storage space and other people to download.

Thanks,

none

[franco]

https://github.com/opnsense/tools/blob/3966b4b63b4a740631ba68c20898e716613ef7b3/config/25.1/extras.conf#L55

I'm relatively sure 1GB image files are too small. The Nano image intentionally uses 3GB as its default.

[none]

Hi Franco,

I tried before using the default image. But that lead me to really small systems :(

You are right, 1GB images should not be enough, but I tried using 3GB and after fisrt boot it doesn't expand. Do I need anything else for it?

As far as I remember, the images have that file you pointed, but nevertheless there is a message on boot telling the fs doesn't need growing. I will try again at home and see if I can get that message and post here.

Thanks again for the help.

none

[none]

Hi Franco, here are my logs:

The message is:

Code Select Expand

mmcsd0 recovering is not needed
mmcsd0s1 resized
growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB
** /dev/ufs/OPNsense

The microSD is just what opnsense creates and the file is there:

Code Select Expand

root@xxx:/mnt # glabel status
                                      Name  Status  Components
                  diskid/DISK-121220160204     N/A  da11
                         msdosfs/MSDOSBOOT     N/A  da11s1
                    ufsid/67ec8a03acb330e8     N/A  da11s2
                              ufs/OPNsense     N/A  da11s2

root@xxx:/mnt # mount /dev/ufs/OPNsensea temp/
root@xxx:/mnt # ls temp/
.cshrc COPYRIGHT conf entropy libexec proc sys
.probe.for.growfs bin dev etc media rescue tmp
.profile boot efiboot.img home mnt root usr
.snap boot.config efiboot.img.d lib net sbin var
root@xxx:/mnt # gpart show da11
=>      63  61069249  da11  MBR  (29G)
        63    131009        - free -  (64M)
    131072     65536     1  fat32lba  [active]  (32M)
    196608   6094848     2  freebsd  (2.9G)
   6291456  54777856        - free -  (26G)

root@trocatador:/mnt # umount temp/


full dmesg if needed in the next message.

none

[none]

Code Select Expand


?cLoading kernel...
/boot/kernel/kernel text=0x2a8 text=0xa3c030 text=0x28630c data=0x15e678 data=0x
0+0x2c3000 0x8+0x1634b8+0x8+0x18e7bb
Loading configured modules...
/etc/hostid...can't find '/etc/hostid'
failed!
carp.../boot/kernel/carp.ko text=0x3888 text=0x78b8 data=0xe28+0x48 0x8+0x1b48+0
x8+0x13cc
if_gre.../boot/kernel/if_gre.ko text=0x254e text=0x3f10 data=0x950+0x40 0x8+0x16
f8+0x8+0xfdd
ugen...can't find 'ugen'
failed!
/boot/modules/if_re.ko.../boot/modules/if_re.ko text=0x7988 text=0x7ccf8 data=0x
a38+0x14 0x8+0x1bd8+0x8+0x15f7
if_tun...usb...uhid.../boot/kernel/uhid.ko text=0x1fc0 text=0x1660 data=0x6f8+0x
4 0x8+0xe88+0x8+0xa61
/boot/entropy.../boot/entropy size=0x1000
ukbd...pfsync.../boot/kernel/pfsync.ko text=0x38b9 text=0x8fd4 data=0xd00+0x8 0x
8+0x1ba8+0x8+0x1485
loading required module 'pf'
/boot/kernel/pf.ko text=0x12340 text=0x4a610 data=0x6bc8+0x484 0x8+0x7020+0x8+0x
68a0
pflog.../boot/kernel/pflog.ko text=0xff8 text=0x938 data=0x488 0x8+0xa20+0x8+0x7
30
pf...if_vlan...if_tap...if_lagg.../boot/kernel/if_lagg.ko text=0x3b5e text=0x9f5
0 data=0xea0+0x8 0x8+0x1d88+0x8+0x16a5
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko text=0x1102 text=0x12ac data=0x308+0x8 0x8+0x8a0+0
x8+0x590
if_gif...if_enc.../boot/kernel/if_enc.ko text=0x168a text=0x974 data=0x840 0x8+0
xd08+0x8+0xb96
if_bridge.../boot/kernel/if_bridge.ko text=0x38cf text=0x6bb0 data=0xee8+0x8 0x8
+0x1c50+0x8+0x1695
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko text=0x12cb text=0x4ca4 data=0x2d0+0x28 0x8+0xb28+0x8+
0x6af
umass...acpi...
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by EFI at 0xef03c000.
����"�+#*�������� ш�1������)�$�� ��Y����r��x�)Չ����u�x�l��y9������-j���˫m���Kt���ݪ�����k����Ot���t��ű8�S���
                                                                                                           �h
                                                                                                             ������������qp���i*����+���e�Je�J����ճ��Nqu�vy}V[]N�����:Y���e������y�ѫ�����Ż����k�ѵ��I[ͭQ������Ũ�����J�����ʊ��딣��uQ<�J����8�i����魾���l������K��$"��
                    $����j�Qq���᪩��}���+��!���+����}<���I�����-���*����p�����u�����)��Q��/ռ<�>�&Չ�W�Q�2�J�����S�]]����<�����+�+*���*����<ɮz��ɩ���"���-�J�����������.�����������,��٩�
                                                                                                                                                                                    ��<������<�*��"d������$�����b&�x��zu8��*��"�e�(ɹ�q��ы�   �V��و(��4��ͣ����W2av:��Jm����,)�
                              (�8���W������y���W������*��*��rq:c��r��������թf�
                                                                              ���5�)�bj�i�?1Ẋՠ��ᭉj驩��Q�u�������(���fr���*ѤY�Q1QA�$��,(��������W�(����"�������z������2��?�ܯ�k
                                                                                                                                                                             �i�,����ű�����,(��+���A[YǴ�,(��+�����"����/��+�?�(�������˲Q)?�Qb�鑬��}��|�S����*��<��`�����񱁙�Yu<�`h(���x��k������!���i(���Nu<���ݪ�+��e�髴qq���
                                                                             Q|�Yq}�W��|��(�h���0|�|7�"���$���
                                                                                                              ����2��!��������.�頭�;����$����i���e#�������x�9y�Y>����   j���!��j�����!'���žከ���::I����ʱ��[��骥����!::�����*�Uu2|�w��z�uhub3: 1 port with 1 removable, self powered
uhub0: 1 port with 1 removable, self powered
uhub5: 2 ports with 2 removable, self powered
Mounting filesysuhub2: 2 ports with 2 removable, self powered
tems...
tunefs: soft updates remains unchanged as enabled
uhub4: 1 port with 1 removable, self powered
uhub1: 1 port with 1 removable, self powered
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mmcsd0 doesn't exist
mmcsd0 recovering is not needed
mmcsd0s1 resized
growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB
** /dev/ufs/OPNsense
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 200966 free (126 frags, 25105 blocks, 0.0% fragmentation)
Setting hostuuid: 00000000-0000-0000-eb8b-ff6f21811d36.
Setting hostid: 0x5f653895.
>>> Invoking import script 'importer'
Press any key to start the configuration importer: .......
Bootstrapping config.xml...done.
Setting up /var/log memory disk...done.
Configuring crash dump device: /dev/null
.ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <ADATA LEGEND 710 x x>
nda0: Serial Number x
nda0: nvme version 1.4
nda0: 244198MB (500118192 512 byte sectors)
>>> Invoking early script 'upgrade'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: ERR
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing..........done.
Starting device manager...hym8563_rtc0: <HYM8694 RTC> at addr 0xa2 irq 90 on iicbus1
hym8563_rtc0: registered as a time-of-day clock, resolution 1.000000s
done.
Configuring login behaviour...done.

Default interfaces not found -- Running interface assignment option.

Press any key to start the manual interface assignment: 1
Do you want to configure LAGGs now? [y/N]: n
Do you want to configure VLANs now? [y/N]: n

Valid interfaces are:

re0              8e:04:f8:20:xx:xx Realtek PCIe 2.5GbE Family Controller
re1              56:f0:8d:5d:xx:xx Realtek PCIe 2.5GbE Family Controller
eqos0            22:3c:1b:bc:xx:xx DesignWare EQOS Gigabit ethernet

If you do not know the names of your interfaces, you may choose to use
auto-detection. In that case, disconnect all interfaces now before
hitting 'a' to initiate auto detection.

Enter the WAN interface name or 'a' for auto-detection: re1

Enter the LAN interface name or 'a' for auto-detection
NOTE: this enables full Firewalling/NAT mode.
(or nothing if finished): re0

Enter the Optional interface 1 name or 'a' for auto-detection
(or nothing if finished):

The interfaces will be assigned as follows:

WAN  -> re1
LAN  -> re0

Do you want to proceed? [y/N]: y

Writing configuration...done.
Configuring loopback interface...lo0: link state changed to UP
done.
Configuring kernel modules...done.
Setting up extended sysctls...done.
Setting timezone: Etc/UTC
Writing firmware settings: FreeBSD OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: Modified 206 trust store links.
Writing trust bundles...done.
Setting hostname: OPNsense.localdomain
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring firewall.......done.
Configuring hardware interfaces...done.
Configuring loopback interface...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring WAN interface...done.
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring firewall.......done.
Starting web GUI...done.
Setting up routes...done.
Starting DHCPv4 service...done.
Starting DHCPv6 service...done.
Starting router advertisement service...done.
Starting Unbound DNS...done.
Configuring firewall.......done.
Setting up gateway monitor...done.
Syncing OpenVPN settings...done.
Starting NTP service...done.
Starting Unbound DNS...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
>>> Invoking start script 'syslog'
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'openvpn'
>>> Invoking start script 'sysctl'
Service `sysctl' has been restarted.
>>> Invoking start script 'beep'
>>> Error in start script '95-beep'
Root file system: /dev/ufs/OPNsense
Wed Apr  2 02:49:09 UTC 2025

*** OPNsense.localdomain: OPNsense 25.1.3_33 (aarch64) ***

 LAN (re0)       -> v4: 192.168.1.1/24
 WAN (re1)       ->

 HTTPS: sha256 70 84 EC 39 83 90 BA 72 B7 6C 74 99 84 8E 8A 87
               2D 10 81 1D D7 27 33 3D 4C 98 C5 CF CD 17 AA FA

FreeBSD/arm64 (OPNsense.localdomain) (ttyu1)

login:



[none]

the part of dmesg that is not readable is where the kernel changes the serial to 1500000 baud rate. Other times, its 115200 and is readable.

Thanks again,

none

[franco]

> mmcsd0s1 resized
> growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB

This definitely works for everyone else and you can see it going through the motions, but your particular hardware doesn't notice the SD card is bigger.

The way this works is very simple. Take image, put on bigger flash storage or even HDD/SSD and it expands and nobody reported an issue in years as well as QA testing works fine. But granted none use ARM custom images so this could also be a driver or hw issue ;)


Cheers,
Franco

[none]

Hi,

thanks again for the help.

So you think:

Code Select Expand

camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mmcsd0 doesn't exist
mmcsd0 recovering is not needed

this error could be the culprit?

If I got it right, the scripting system detects the size of the media, and then compares to the size of the partition.

I will try again using usb drive and will boot again the microSD to figure out the line the kernel notices the microSD hardware and size. May that be on the 1500000 part of the log (I will try to change the image to that speed).

Together will try it on R4S, a more stable board that opnsense has a config for building. My goal is to submit a conf for R5S, but I want to have it tested as much as I can. And I don't know if opnsense can have EDK2 bytes on the code. This working image need it. The U-Boot is not working for both 2.5Gbps nics.

Again thanks, I will try to dig further here.

none

[none]

I started the R5S using microSD and got this:

mmcsd0: 31GB <SDHC SD32G 6.1 SN 77400AE2 MFG 10/2020 by 159 TI> at mmc0 50.0MHz/4bit/1016-block

At least now it can figure all microSD size. Tried create again the file, no good.

Will try USB stick and that image.

none

[none]

Very same thing happens when using USB to boot from, and this time its u-boot firmware:

da0 recovering is not needed
da0s1 resized
growfs: requested size 3.0GB is equal to the current filesystem size 3.0GB

I used the same build files for both images.

none

[franco]

Sorry, something is broken there but I'm unable to reproduce.


Cheers,
Franco