Hi,
I build images using the line:
make -C /usr/tools/ VERSION=25.1.4 DEVICE=R4S arm-8G
and then I get my 8GB image ready for dd'ing. It works but the disk is not resized.
I have 32GB and 64GB microSD in use and it keeps the 8GB partition size.
My research took me to the need for the .probe.for.growfs file on /, and the opnsense scripts do create it as is written in extras.conf. The extras.conf I use (https://github.com/matheusber/opnsense/blob/main/arm64-opnsense-build/opnsense-confs/extras.conf) just add the lines to load if_re from Realtek kmod version 1.98 from ports, that I build here.
The disk layout is what opnsense scripts creates. Have anyone delt with this?
Would really help create 1GB image files that would expand to the microSD size after. Will help too much here in storage space and other people to download.
Thanks,
none
https://github.com/opnsense/tools/blob/3966b4b63b4a740631ba68c20898e716613ef7b3/config/25.1/extras.conf#L55
I'm relatively sure 1GB image files are too small. The Nano image intentionally uses 3GB as its default.
Hi Franco,
I tried before using the default image. But that lead me to really small systems :(
You are right, 1GB images should not be enough, but I tried using 3GB and after fisrt boot it doesn't expand. Do I need anything else for it?
As far as I remember, the images have that file you pointed, but nevertheless there is a message on boot telling the fs doesn't need growing. I will try again at home and see if I can get that message and post here.
Thanks again for the help.
none
Hi Franco, here are my logs:
The message is:
mmcsd0 recovering is not needed
mmcsd0s1 resized
growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB
** /dev/ufs/OPNsense
The microSD is just what opnsense creates and the file is there:
root@xxx:/mnt # glabel status
Name Status Components
diskid/DISK-121220160204 N/A da11
msdosfs/MSDOSBOOT N/A da11s1
ufsid/67ec8a03acb330e8 N/A da11s2
ufs/OPNsense N/A da11s2
root@xxx:/mnt # mount /dev/ufs/OPNsensea temp/
root@xxx:/mnt # ls temp/
.cshrc COPYRIGHT conf entropy libexec proc sys
.probe.for.growfs bin dev etc media rescue tmp
.profile boot efiboot.img home mnt root usr
.snap boot.config efiboot.img.d lib net sbin var
root@xxx:/mnt # gpart show da11
=> 63 61069249 da11 MBR (29G)
63 131009 - free - (64M)
131072 65536 1 fat32lba [active] (32M)
196608 6094848 2 freebsd (2.9G)
6291456 54777856 - free - (26G)
root@trocatador:/mnt # umount temp/
full dmesg if needed in the next message.
none
?cLoading kernel...
/boot/kernel/kernel text=0x2a8 text=0xa3c030 text=0x28630c data=0x15e678 data=0x
0+0x2c3000 0x8+0x1634b8+0x8+0x18e7bb
Loading configured modules...
/etc/hostid...can't find '/etc/hostid'
failed!
carp.../boot/kernel/carp.ko text=0x3888 text=0x78b8 data=0xe28+0x48 0x8+0x1b48+0
x8+0x13cc
if_gre.../boot/kernel/if_gre.ko text=0x254e text=0x3f10 data=0x950+0x40 0x8+0x16
f8+0x8+0xfdd
ugen...can't find 'ugen'
failed!
/boot/modules/if_re.ko.../boot/modules/if_re.ko text=0x7988 text=0x7ccf8 data=0x
a38+0x14 0x8+0x1bd8+0x8+0x15f7
if_tun...usb...uhid.../boot/kernel/uhid.ko text=0x1fc0 text=0x1660 data=0x6f8+0x
4 0x8+0xe88+0x8+0xa61
/boot/entropy.../boot/entropy size=0x1000
ukbd...pfsync.../boot/kernel/pfsync.ko text=0x38b9 text=0x8fd4 data=0xd00+0x8 0x
8+0x1ba8+0x8+0x1485
loading required module 'pf'
/boot/kernel/pf.ko text=0x12340 text=0x4a610 data=0x6bc8+0x484 0x8+0x7020+0x8+0x
68a0
pflog.../boot/kernel/pflog.ko text=0xff8 text=0x938 data=0x488 0x8+0xa20+0x8+0x7
30
pf...if_vlan...if_tap...if_lagg.../boot/kernel/if_lagg.ko text=0x3b5e text=0x9f5
0 data=0xea0+0x8 0x8+0x1d88+0x8+0x16a5
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko text=0x1102 text=0x12ac data=0x308+0x8 0x8+0x8a0+0
x8+0x590
if_gif...if_enc.../boot/kernel/if_enc.ko text=0x168a text=0x974 data=0x840 0x8+0
xd08+0x8+0xb96
if_bridge.../boot/kernel/if_bridge.ko text=0x38cf text=0x6bb0 data=0xee8+0x8 0x8
+0x1c50+0x8+0x1695
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko text=0x12cb text=0x4ca4 data=0x2d0+0x28 0x8+0xb28+0x8+
0x6af
umass...acpi...
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
Using DTB provided by EFI at 0xef03c000.
����"�+#*�������� ш�1������)�$�� ��Y����r��x�)Չ����u�x�l��y9������-j���˫m���Kt���ݪ�����k����Ot���t��ű8�S���
�h
������������qp���i*����+���e�Je�J����ճ��Nqu�vy}V[]N�����:Y���e������y�ѫ�����Ż����k�ѵ��I[ͭQ������Ũ�����J�����ʊ��딣��uQ<�J����8�i����魾���l������K��$"��
$����j�Qq���᪩��}���+��!���+����}<���I�����-���*����p�����u�����)��Q��/ռ<�>�&Չ�W�Q�2�J�����S�]]����<�����+�+*���*����<ɮz��ɩ���"���-�J�����������.�����������,��٩�
��<������<�*��"d������$�����b&�x��zu8��*��"�e�(ɹ�q��ы� �V��و(��4��ͣ����W2av:��Jm����,)�
(�8���W������y���W������*��*��rq:c��r��������թf�
���5�)�bj�i�?1Ẋՠ��ᭉj驩��Q�u�������(���fr���*ѤY�Q1QA�$��,(��������W�(����"�������z������2��?�ܯ�k
�i�,����ű�����,(��+���A[YǴ�,(��+�����"����/��+�?�(�������˲Q)?�Qb�鑬��}��|�S����*��<��`������Yu<�`h(���x��k������!���i(���Nu<���ݪ�+��e�髴qq���
Q|�Yq}�W��|��(�h���0|�|7�"���$���
����2��!��������.�頭�;����$����i���e#�������x�9y�Y>���� j���!��j�����!'���žከ���::I����ʱ��[��骥����!::�����*�Uu2|�w��z�uhub3: 1 port with 1 removable, self powered
uhub0: 1 port with 1 removable, self powered
uhub5: 2 ports with 2 removable, self powered
Mounting filesysuhub2: 2 ports with 2 removable, self powered
tems...
tunefs: soft updates remains unchanged as enabled
uhub4: 1 port with 1 removable, self powered
uhub1: 1 port with 1 removable, self powered
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mmcsd0 doesn't exist
mmcsd0 recovering is not needed
mmcsd0s1 resized
growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB
** /dev/ufs/OPNsense
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 200966 free (126 frags, 25105 blocks, 0.0% fragmentation)
Setting hostuuid: 00000000-0000-0000-eb8b-ff6f21811d36.
Setting hostid: 0x5f653895.
>>> Invoking import script 'importer'
Press any key to start the configuration importer: .......
Bootstrapping config.xml...done.
Setting up /var/log memory disk...done.
Configuring crash dump device: /dev/null
.ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <ADATA LEGEND 710 x x>
nda0: Serial Number x
nda0: nvme version 1.4
nda0: 244198MB (500118192 512 byte sectors)
>>> Invoking early script 'upgrade'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: ERR
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing..........done.
Starting device manager...hym8563_rtc0: <HYM8694 RTC> at addr 0xa2 irq 90 on iicbus1
hym8563_rtc0: registered as a time-of-day clock, resolution 1.000000s
done.
Configuring login behaviour...done.
Default interfaces not found -- Running interface assignment option.
Press any key to start the manual interface assignment: 1
Do you want to configure LAGGs now? [y/N]: n
Do you want to configure VLANs now? [y/N]: n
Valid interfaces are:
re0 8e:04:f8:20:xx:xx Realtek PCIe 2.5GbE Family Controller
re1 56:f0:8d:5d:xx:xx Realtek PCIe 2.5GbE Family Controller
eqos0 22:3c:1b:bc:xx:xx DesignWare EQOS Gigabit ethernet
If you do not know the names of your interfaces, you may choose to use
auto-detection. In that case, disconnect all interfaces now before
hitting 'a' to initiate auto detection.
Enter the WAN interface name or 'a' for auto-detection: re1
Enter the LAN interface name or 'a' for auto-detection
NOTE: this enables full Firewalling/NAT mode.
(or nothing if finished): re0
Enter the Optional interface 1 name or 'a' for auto-detection
(or nothing if finished):
The interfaces will be assigned as follows:
WAN -> re1
LAN -> re0
Do you want to proceed? [y/N]: y
Writing configuration...done.
Configuring loopback interface...lo0: link state changed to UP
done.
Configuring kernel modules...done.
Setting up extended sysctls...done.
Setting timezone: Etc/UTC
Writing firmware settings: FreeBSD OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: Modified 206 trust store links.
Writing trust bundles...done.
Setting hostname: OPNsense.localdomain
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring firewall.......done.
Configuring hardware interfaces...done.
Configuring loopback interface...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring WAN interface...done.
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring firewall.......done.
Starting web GUI...done.
Setting up routes...done.
Starting DHCPv4 service...done.
Starting DHCPv6 service...done.
Starting router advertisement service...done.
Starting Unbound DNS...done.
Configuring firewall.......done.
Setting up gateway monitor...done.
Syncing OpenVPN settings...done.
Starting NTP service...done.
Starting Unbound DNS...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
>>> Invoking start script 'syslog'
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'openvpn'
>>> Invoking start script 'sysctl'
Service `sysctl' has been restarted.
>>> Invoking start script 'beep'
>>> Error in start script '95-beep'
Root file system: /dev/ufs/OPNsense
Wed Apr 2 02:49:09 UTC 2025
*** OPNsense.localdomain: OPNsense 25.1.3_33 (aarch64) ***
LAN (re0) -> v4: 192.168.1.1/24
WAN (re1) ->
HTTPS: sha256 70 84 EC 39 83 90 BA 72 B7 6C 74 99 84 8E 8A 87
2D 10 81 1D D7 27 33 3D 4C 98 C5 CF CD 17 AA FA
FreeBSD/arm64 (OPNsense.localdomain) (ttyu1)
login:
the part of dmesg that is not readable is where the kernel changes the serial to 1500000 baud rate. Other times, its 115200 and is readable.
Thanks again,
none
> mmcsd0s1 resized
> growfs: requested size 2.9GB is equal to the current filesystem size 2.9GB
This definitely works for everyone else and you can see it going through the motions, but your particular hardware doesn't notice the SD card is bigger.
The way this works is very simple. Take image, put on bigger flash storage or even HDD/SSD and it expands and nobody reported an issue in years as well as QA testing works fine. But granted none use ARM custom images so this could also be a driver or hw issue ;)
Cheers,
Franco
Hi,
thanks again for the help.
So you think:
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mmcsd0 doesn't exist
mmcsd0 recovering is not needed
this error could be the culprit?
If I got it right, the scripting system detects the size of the media, and then compares to the size of the partition.
I will try again using usb drive and will boot again the microSD to figure out the line the kernel notices the microSD hardware and size. May that be on the 1500000 part of the log (I will try to change the image to that speed).
Together will try it on R4S, a more stable board that opnsense has a config for building. My goal is to submit a conf for R5S, but I want to have it tested as much as I can. And I don't know if opnsense can have EDK2 bytes on the code. This working image need it. The U-Boot is not working for both 2.5Gbps nics.
Again thanks, I will try to dig further here.
none
I started the R5S using microSD and got this:
mmcsd0: 31GB <SDHC SD32G 6.1 SN 77400AE2 MFG 10/2020 by 159 TI> at mmc0 50.0MHz/4bit/1016-block
At least now it can figure all microSD size. Tried create again the file, no good.
Will try USB stick and that image.
none
Very same thing happens when using USB to boot from, and this time its u-boot firmware:
da0 recovering is not needed
da0s1 resized
growfs: requested size 3.0GB is equal to the current filesystem size 3.0GB
I used the same build files for both images.
none
Sorry, something is broken there but I'm unable to reproduce.
Cheers,
Franco
Hi Franco, can you point me out a case that should work so I can try to look it and reproduce here? What image to test?
I want to know what disk partitioning there should be so the growfs will work.
thanks,
none
You can try this with the prebuilt amd64 Nano image... debugging this is going to be hard as the tools do what they should but the result is wrong.
Cheers,
Franco
Hi Franco,
so even in the nano image the the resizing is not working?
I will fix a pc for testing this and see what I can help.
Thanks again.
none
> so even in the nano image the the resizing is not working?
Says you? Last time I checked it worked, which was 25.1...
Quote from: franco on April 04, 2025, 02:42:57 PM> so even in the nano image the the resizing is not working?
Says you? Last time I checked it worked, which was 25.1...
No, please don't get me wrong. I misunderstood your post, so I asked to be sure. I thought you were saying that was not working anymore, and now I see I got it wrong.
You said earlier it worked for you on amd64 and as I read it I got confused, as if you had tested again and said was not working anymore. Sorry for the noise :(
I just was trying to get something to compare and have a starting point. And now I have it. Thanks again.
none
Ok, thanks for clearing that up. I will test Nano for business 25.4 images again early next week anyway and promise to report back.
Cheers,
Franco
Hi Franco,
I got to walk a few steps here, here is the result.
I found that the line:
local IDX=${1##"${DEV}p"}
in /usr/local/etc/rc was returning a value that would make the next "if" always be true:
if [ "${IDX}" = "${1}" ]; then
# assume schema ada0a used by nano images
IDX=1
fi
This way when disks configured as MBR, as is the case here, would always default to IDX=1 and make it to the next "if":
if [ -n "${DEV}" -a "${DEV}" != "${1}" -a \
-n "${IDX}" -a "${IDX}" != "${1}" ]; then
gpart recover ${DEV}
gpart resize -i ${IDX} ${DEV}
fi
issuing gpart resize on a partition that has no place to grow (and not the root one, also). The partitions here are organized like this:
gpart show da11
=> 63 61069249 da11 MBR (29G)
63 131009 - free - (64M)
131072 65536 1 fat32lba [active] (32M)
196608 6094848 2 freebsd (2.9G)
6291456 54777856 - free - (26G)
Just as the default opnsense scripts build. Running on index 1 would not meet the premise for the growfs step that follows, and then get that message saying it already has the desired size.
For the Nano image that would do fine, as it has only 1 partition, the BSD scheme is written directly on the device, no MBR or GPT is created.
To address this I created a test to know if it is a MBR scheme, and then act accordingly. That way it finds the index for the root file system and the gpart resize step works.
Now I got a message from gpart tellimg me I need to do a commit to the UFS label (my case was ufs/OPNsense. Not doing that would ruin the boot:
mmcsd0 recovering is not needed
GEOM_PART: ufs/OPNsense was automatically resized.
Use `gpart commit ufs/OPNsense` to save changes or `gpart undo ufs/OPNsense` to revert them.
mmcsd0s2 resized
That step is outside the grow_partition(), and later I found was not welcome on AMD64 Nano image (may be arch related?). I then created another test to run only on MBR schemes.
My skills here are far from yours and the opnsense team. Here is the file I ended with, I hope my humble time here can help anyhow.
My tests here are also limited. Nano images works as is now and keeps working after the modifications. ARM64 images now work in the first boot. I just need to learn how to commit changes to the local repo, so images will have it out-of-the-box either.
Thanks for the help, as usual :)
The dmesg from that part is:
Booting [/boot/kernel/kernel]...
Using DTB provided by EFI at 0xef03c000.
����#U}�qqC��� ����!����*���mu��������k�饭�9� ���5������j�j�����������)��y��i*���Ovq4�y���,�J�uQ�����4����+��������몋���
���8�q����i*���
+�v����J��:u�Ѩ��V[]����U8��x��y>k�8�ռy�Ѫ��i�d���i�����.�����8;�u�+����+�ꪨ�,���+���ʊ�����ʊ���+��v�����+��)�ź����*�����l�"�)��᪩��0���uW�S�I+��4��{qɩ��J����+���0��9},����e����Q���}����..��Y��0�ټ�.�g�����+�}���*�����(��k��W;�+*���<�ܑ����������y�J�h������"�(����(��J��$��b*��,��٩���9��ը�����3 �����q�WS[�mq=1>Q2ը��.��z����>A�Y�q5��q��W�騤�ɪ�`�ѤY�Q1QA��8��m+�+��,���m�����)������"�������ъ(�#��������r�*�u�u���������f+����U1��*�b���0�������������+���q�t�9���%��Z�ᩴu�]T���ԼQWI�QS����]�8<������2��*�S�*��j�r��������n�*�j�*[TWS�>��<�,����K
�++��Y)����,(�誶��$�+���uNJ�ي+k����;z���J�������;��գ��ű��
����Y�?�����՜u������q���������Y8���������(�����a����ɛ�Q:��(���������**�ͣ�����f���Q�Z��銊�����9x��+��e.������ ~qq���z��!��**+���$����(��8x�ዧ������}��(���Ț�Z�)���������动���::I�����1::I�Ŋ�:����!::�����*��8
������uhub0: 1 port with 1 removable, self powered
uhub3: 1 port with 1 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
uhub1: 1 port with 1 removable, self powered
uhub4: 1 port with 1 removable, self powered
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mmcsd0 doesn't exist
mmcsd0 recovering is not needed
GEOM_PART: ufs/OPNsense was automatically resized.
Use `gpart commit ufs/OPNsense` to save changes or `gpart undo ufs/OPNsense` to revert them.
mmcsd0s2 resized
super-block backups (for fsck_ffs -b #) at:
6402432, 7682880, 8963328, 10243776, 11524224, 12804672, 14085120, 15365568,
16646016, 17926464, 19206912, 20487360, 21767808, 23048256, 24328704,
25609152, 26889600, 28170048, 29450496, 30730944, 32011392, 33291840,
34572288, 35852736, 37133184, 38413632, 39694080, 40974528, 42254976,
43535424, 44815872, 46096320, 47376768, 48657216, 49937664, 51218112,
52498560, 53779008, 55059456, 56339904, 57620352, 58900800, 60181248
** /dev/ufs/OPNsense
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 6831398 free (94 frags, 853913 blocks, 0.0% fragmentation)
Setting hostuuid: 00000000-0000-0000-eb8b-ff6f21811d36.
Setting hostid: 0x5f653895.
>>> Invoking import script 'importer'
Press any key to start the configuration importer: .......
Bootstrapping config.xml...done.
FreeBSD/arm64 (OPNsense.localdomain) (ttyu1)
login: root
Password:
------------------------------------------------
| Hello, this is OPNsense 25.1 | :::::::.
| | :::::::::.
| Website: https://opnsense.org/ | ::: :::
| Handbook: https://docs.opnsense.org/ | ::: :::
| Forums: https://forum.opnsense.org/ | ::: :::
| Code: https://github.com/opnsense | `:::::::::
| Reddit: https://reddit.com/r/opnsense | `:::::::
------------------------------------------------
*** OPNsense.localdomain: OPNsense 25.1.3_33 (aarch64) ***
LAN (re0) -> v4: 192.168.1.1/24
WAN (re1) ->
HTTPS: sha256 2B 47 72 39 88 B9 32 31 2F 65 59 19 61 F6 85 76
08 19 F7 E9 39 34 18 F8 97 70 A1 58 50 46 2C 41
0) Logout 7) Ping host
1) Assign interfaces 8) Shell
2) Set interface IP address 9) pfTop
3) Reset the root password 10) Firewall log
4) Reset to factory defaults 11) Reload all services
5) Power off system 12) Update from console
6) Reboot system 13) Restore a backup
Enter an option: 8
root@OPNsense:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/OPNsense 28G 2.0G 24G 8% /
devfs 1.0K 0B 1.0K 0% /dev
/dev/msdosfs/MSDOSBOOT 32M 832K 31M 3% /boot/msdos
devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev
devfs 1.0K 0B 1.0K 0% /var/unbound/dev
/usr/local/lib/python3.11 28G 2.0G 24G 8% /var/unbound/usr/local/lib/python3.11
/lib 28G 2.0G 24G 8% /var/unbound/lib