OPNsense Forum
English Forums => General Discussion => Topic started by: franco on May 27, 2016, 07:31:03 pm
-
Hi all,
So I'd just like to share this piece of info because I think that it's worth mentioning how far FreeBSD has evolved in terms of a flexible distribution and package management platform. The background:
Back in 2015 we've faced the challenge of not being able to easily move OPNsense to cloud deployments, because either no images were allowed or because vendor signup wasn't as easy as expected. Instead, we've explored ways to bring OPNsense to native FreeBSD installations and thus "opnsense-bootstrap" was born as a light but powerful shell script.
https://github.com/opnsense/update#opnsense-bootstrap
What it would do is transform the installation into an OPNsense system by securely fetching the latest packages with the weakest link being trusting the GitHub SSL certificate using the commonly available CA package. The benefit was that any FreeBSD 10 could be changed into the latest version of OPNsense without an issue.
It was later thought that we may be able to use it as a tool to move OPNsense to HardenedBSD/OPNsense or back as well. Since HardenedBSD/OPNsense runs on 11-CURRENT we've not attempted such a transformation, but I found a viable candidate... Our predecessor pfSense in the latest version 2.3.x.
Running two simple commands will execute the bootstrap process, I've highlighted them and dumped the full output in the hopes that anyone finds it useful.
Since the config.xml layout constantly changes with pfSense, the default way of settings passwords for users was removed with 2.3, but fear not! You can recover a password from the console using:
# /usr/local/etc/rc.initial.password
After one more reboot the system is ready for login. :)
opnsense-bootstrap can also reset the configuration, which would be preferred in such cases by using the "-f" option. In theory this also works the other way around but may need further tweaking. All in all, I thank the pfSense developers for doing such a quality step in firmware adaption in the scope of FreeBSD. It will make all of our lives easier in the future.
Cheers,
Franco
PS: Don't try this at home. ;)
*** Welcome to pfSense 2.3.1-RELEASE (amd64 full-install) on pfSense ***
WAN (wan) -> em0 -> v4/DHCP4: 10.0.100.17/16
0) Logout (SSH only) 9) pfTop
1) Assign Interfaces 10) Filter Logs
2) Set interface(s) IP address 11) Restart webConfigurator
3) Reset webConfigurator password 12) pfSense Developer Shell
4) Reset to factory defaults 13) Update from console
5) Reboot system 14) Disable Secure Shell (sshd)
6) Halt system 15) Restore recent configuration
7) Ping host 16) Restart PHP-FPM
8) Shell
Enter an option: 8
[2.3.1-RELEASE][admin@pfSense.localdomain]/root: fetch https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh
opnsense-bootstrap.sh 100% of 3200 B 33 MBps 00m00s
[2.3.1-RELEASE][admin@pfSense.localdomain]/root: sh ./opnsense-bootstrap.sh
This utility will attempt to turn this installation into the latest
OPNsense 16.1 release. All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.
Proceed with this action? [y/N]: y
pkg already bootstrapped at /usr/local/sbin/pkg
Updating pfSense-core repository catalogue...
pfSense-core repository is up-to-date.
Updating pfSense repository catalogue...
pfSense repository is up-to-date.
All repositories are up-to-date.
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
The most recent version of packages are already installed
fetch: https://github.com/opnsense/core/archive/stable/16.1.zip: size of remote file is not known
16.1.zip 12 MB 2340 kBps 00m06s
Archive: 16.1.zip
d core-stable-16.1
ataidle-2.7.2: already unlocked
beep-1.0_1: already unlocked
bind-tools-9.10.3P4: already unlocked
bsnmp-regex-0.6_1: already unlocked
bsnmp-ucd-0.4.2: already unlocked
bwi-firmware-kmod-3.130.20: already unlocked
ca_root_nss-3.22.2: already unlocked
check_reload_status-0.0.7: already unlocked
choparp-20150613: already unlocked
clog-1.0.1: already unlocked
cpdup-1.18: already unlocked
cpustats-0.1_1: already unlocked
curl-7.48.0_1: already unlocked
dhcp6-20080615_7: already unlocked
dhcpleases-0.3_1: already unlocked
dhcpleases6-0.1_2: already unlocked
dmidecode-3.0: already unlocked
dnsmasq-devel-2.76.0test12: already unlocked
dpinger-2.0: already unlocked
expat-2.1.0_3: already unlocked
expiretable-0.6_1: already unlocked
filterdns-1.0_9: already unlocked
filterlog-0.1_2: already unlocked
gettext-runtime-0.19.7: already unlocked
glib-2.46.2: already unlocked
gmp-5.1.3_3: already unlocked
gogoc-1.2_1: already unlocked
idnkit-1.0_5: already unlocked
igmpproxy-0.1_3,1: already unlocked
indexinfo-0.2.4: already unlocked
ipmitool-1.8.15_1: already unlocked
isc-dhcp43-client-4.3.3P1_1: already unlocked
isc-dhcp43-relay-4.3.3P1_1: already unlocked
isc-dhcp43-server-4.3.3P1_1: already unlocked
ldns-1.6.17_5: already unlocked
libdaemon-0.14_1: already unlocked
libedit-3.1.20150325_2: already unlocked
libevent2-2.0.22_1: already unlocked
libffi-3.2.1: already unlocked
libiconv-1.14_9: already unlocked
libidn-1.31: already unlocked
libltdl-2.4.6: already unlocked
libmcrypt-2.5.8_3: already unlocked
libpdel-0.5.3_6: already unlocked
libsodium-1.0.8: already unlocked
libssh2-1.6.0_1,2: already unlocked
libucl-0.8.0: already unlocked
libxml2-2.9.3: already unlocked
libzmq4-4.1.4_1: already unlocked
links-2.9,1: already unlocked
lzo2-2.09: already unlocked
minicron-0.0.2: already unlocked
miniupnpd-1.9.20160113,1: already unlocked
mpd4-4.4.1_1: already unlocked
mpd5-5.8: already unlocked
nettle-3.2: already unlocked
nginx-1.8.1,2: already unlocked
ntp-4.2.8p7: already unlocked
oniguruma5-5.9.6_1: already unlocked
openldap-client-2.4.44: already unlocked
openvpn-2.3.11: already unlocked
pcre-8.38_1: already unlocked
pecl-radius-1.2.7: already unlocked
pecl-rrd-1.1.3_2: already unlocked
pecl-ssh2-0.12: already unlocked
pecl-zmq-1.1.3_1: already unlocked
perl5-5.20.3_12: already unlocked
pfSense-2.3.1: already unlocked
pfSense-Status_Monitoring-1.3_1: already unlocked
pfSense-base-2.3.1: already unlocked
pfSense-default-config-2.3.1: already unlocked
Unlocking pfSense-kernel-pfSense-2.3.1
pfSense-rc-2.3.1: already unlocked
pfSense-repo-2.3.1: already unlocked
pftop-0.7_6: already unlocked
php-suhosin-0.9.38: already unlocked
php-xdebug-2.2.5: already unlocked
php56-5.6.21: already unlocked
php56-bcmath-5.6.21: already unlocked
php56-bz2-5.6.21: already unlocked
php56-ctype-5.6.21: already unlocked
php56-curl-5.6.21: already unlocked
php56-dom-5.6.21: already unlocked
php56-filter-5.6.21: already unlocked
php56-gettext-5.6.21: already unlocked
php56-hash-5.6.21: already unlocked
php56-json-5.6.21: already unlocked
php56-ldap-5.6.21: already unlocked
php56-mbstring-5.6.21: already unlocked
php56-mcrypt-5.6.21: already unlocked
php56-opcache-5.6.21: already unlocked
php56-openssl-5.6.21: already unlocked
php56-pcntl-5.6.21: already unlocked
php56-pdo-5.6.21: already unlocked
php56-pdo_sqlite-5.6.21: already unlocked
php56-pfSense-module-0.12: already unlocked
php56-posix-5.6.21: already unlocked
php56-readline-5.6.21: already unlocked
php56-session-5.6.21: already unlocked
php56-shmop-5.6.21: already unlocked
php56-simplexml-5.6.21: already unlocked
php56-sockets-5.6.21: already unlocked
php56-sqlite3-5.6.21: already unlocked
php56-sysvmsg-5.6.21: already unlocked
php56-sysvsem-5.6.21: already unlocked
php56-sysvshm-5.6.21: already unlocked
php56-tokenizer-5.6.21: already unlocked
php56-xml-5.6.21: already unlocked
php56-xmlreader-5.6.21: already unlocked
php56-xmlwriter-5.6.21: already unlocked
php56-zlib-5.6.21: already unlocked
pkg-1.7.2_2: already unlocked
python27-2.7.11_2: already unlocked
qstats-0.1_1: already unlocked
radvd-1.9.1: already unlocked
rate-0.9_1: already unlocked
relayd-5.5.20140810_1: already unlocked
rrdtool-1.5.5_1: already unlocked
scponly-4.8.20110526_2: already unlocked
smartmontools-6.4_2: already unlocked
sqlite3-3.11.1: already unlocked
ssh_tunnel_shell-0.1: already unlocked
sshlockout_pf-0.0.2: already unlocked
strongswan-5.4.0: already unlocked
uclcmd-0.1: already unlocked
unbound-1.5.8: already unlocked
voucher-0.1_2: already unlocked
vstr-1.0.15_1: already unlocked
wol-0.7.1_2: already unlocked
wrapalixresetbutton-0.0.7: already unlocked
xinetd-2.3.15_1: already unlocked
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 131 packages (of 0 packages in the universe):
Installed packages to be REMOVED:
ataidle-2.7.2
beep-1.0_1
bind-tools-9.10.3P4
bsnmp-regex-0.6_1
bsnmp-ucd-0.4.2
bwi-firmware-kmod-3.130.20
ca_root_nss-3.22.2
check_reload_status-0.0.7
choparp-20150613
clog-1.0.1
cpdup-1.18
cpustats-0.1_1
curl-7.48.0_1
dhcp6-20080615_7
dhcpleases-0.3_1
dhcpleases6-0.1_2
dmidecode-3.0
dnsmasq-devel-2.76.0test12
dpinger-2.0
expat-2.1.0_3
expiretable-0.6_1
filterdns-1.0_9
filterlog-0.1_2
gettext-runtime-0.19.7
glib-2.46.2
gmp-5.1.3_3
gogoc-1.2_1
idnkit-1.0_5
igmpproxy-0.1_3,1
indexinfo-0.2.4
ipmitool-1.8.15_1
isc-dhcp43-client-4.3.3P1_1
isc-dhcp43-relay-4.3.3P1_1
isc-dhcp43-server-4.3.3P1_1
ldns-1.6.17_5
libdaemon-0.14_1
libedit-3.1.20150325_2
libevent2-2.0.22_1
libffi-3.2.1
libiconv-1.14_9
libidn-1.31
libltdl-2.4.6
libmcrypt-2.5.8_3
libpdel-0.5.3_6
libsodium-1.0.8
libssh2-1.6.0_1,2
libucl-0.8.0
libxml2-2.9.3
libzmq4-4.1.4_1
links-2.9,1
lzo2-2.09
minicron-0.0.2
miniupnpd-1.9.20160113,1
mpd4-4.4.1_1
mpd5-5.8
nettle-3.2
nginx-1.8.1,2
ntp-4.2.8p7
oniguruma5-5.9.6_1
openldap-client-2.4.44
openvpn-2.3.11
pcre-8.38_1
pecl-radius-1.2.7
pecl-rrd-1.1.3_2
pecl-ssh2-0.12
pecl-zmq-1.1.3_1
perl5-5.20.3_12
pfSense-2.3.1
pfSense-Status_Monitoring-1.3_1
pfSense-base-2.3.1
pfSense-default-config-2.3.1
pfSense-kernel-pfSense-2.3.1
pfSense-rc-2.3.1
pfSense-repo-2.3.1
pftop-0.7_6
php-suhosin-0.9.38
php-xdebug-2.2.5
php56-5.6.21
php56-bcmath-5.6.21
php56-bz2-5.6.21
php56-ctype-5.6.21
php56-curl-5.6.21
php56-dom-5.6.21
php56-filter-5.6.21
php56-gettext-5.6.21
php56-hash-5.6.21
php56-json-5.6.21
php56-ldap-5.6.21
php56-mbstring-5.6.21
php56-mcrypt-5.6.21
php56-opcache-5.6.21
php56-openssl-5.6.21
php56-pcntl-5.6.21
php56-pdo-5.6.21
php56-pdo_sqlite-5.6.21
php56-pfSense-module-0.12
php56-posix-5.6.21
php56-readline-5.6.21
php56-session-5.6.21
php56-shmop-5.6.21
php56-simplexml-5.6.21
php56-sockets-5.6.21
php56-sqlite3-5.6.21
php56-sysvmsg-5.6.21
php56-sysvsem-5.6.21
php56-sysvshm-5.6.21
php56-tokenizer-5.6.21
php56-xml-5.6.21
php56-xmlreader-5.6.21
php56-xmlwriter-5.6.21
php56-zlib-5.6.21
pkg-1.7.2_2
python27-2.7.11_2
qstats-0.1_1
radvd-1.9.1
rate-0.9_1
relayd-5.5.20140810_1
rrdtool-1.5.5_1
scponly-4.8.20110526_2
smartmontools-6.4_2
sqlite3-3.11.1
ssh_tunnel_shell-0.1
sshlockout_pf-0.0.2
strongswan-5.4.0
uclcmd-0.1
unbound-1.5.8
voucher-0.1_2
vstr-1.0.15_1
wol-0.7.1_2
wrapalixresetbutton-0.0.7
xinetd-2.3.15_1
The operation will free 309 MiB.
[1/131] Deinstalling pfSense-2.3.1...
[1/131] Deleting files for pfSense-2.3.1: 100%
[2/131] Deinstalling pfSense-Status_Monitoring-1.3_1...
[2/131] Deleting files for pfSense-Status_Monitoring-1.3_1: 100%
[... lots more of uninstalling]
etc/pkg -> /usr/local/etc/pkg
etc/pkg/fingerprints -> /usr/local/etc/pkg/fingerprints
etc/pkg/fingerprints/OPNsense -> /usr/local/etc/pkg/fingerprints/OPNsense
etc/pkg/fingerprints/OPNsense/revoked -> /usr/local/etc/pkg/fingerprints/OPNsense/revoked
etc/pkg/fingerprints/OPNsense/revoked/pkg.opnsense.org.20150402 -> /usr/local/etc/pkg/fingerprints/OPNsense/revoked/pkg.opnsense.org.20150402
etc/pkg/fingerprints/OPNsense/trusted -> /usr/local/etc/pkg/fingerprints/OPNsense/trusted
etc/pkg/fingerprints/OPNsense/trusted/pkg.opnsense.org.20160104 -> /usr/local/etc/pkg/fingerprints/OPNsense/trusted/pkg.opnsense.org.20160104
etc/pkg/repos -> /usr/local/etc/pkg/repos
etc/pkg/repos/FreeBSD.conf -> /usr/local/etc/pkg/repos/FreeBSD.conf
etc/pkg/repos/origin.conf.sample.in -> /usr/local/etc/pkg/repos/origin.conf.sample.in
/usr/local/etc/pkg/repos/origin.conf.sample.in -> /usr/local/etc/pkg/repos/origin.conf.sample
/usr/local/etc/pkg/repos/origin.conf.sample -> /usr/local/etc/pkg/repos/origin.conf
Bootstrapping pkg from pkg+http://pkg.opnsense.org/FreeBSD:10:amd64/16.1/latest, please wait...
Verifying signature with trusted certificate pkg.opnsense.org.20160104... done
Installing pkg-1.7.2...
Extracting pkg-1.7.2: 100%
Updating OPNsense repository catalogue...
Fetching meta.txz: 100% 1 KiB 1.5kB/s 00:01
Fetching packagesite.txz: 100% 87 KiB 89.0kB/s 00:01
Processing entries: 100%
OPNsense repository update completed. 289 packages processed.
Updating database digests format: 100%
The following 116 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
opnsense: 16.1.15
php56-zlib: 5.6.21
php56: 5.6.21
libxml2: 2.9.3
pcre: 8.38_1
php-pfSense: 0.3
radvd: 1.15
libdaemon: 0.14_1
unbound: 1.5.8
expat: 2.1.0_3
openssl: 1.0.2_12
ldns: 1.6.17_5
libevent2: 2.0.22_1
isc-dhcp43-relay: 4.3.3P1_1
dhcpleases: 0.2
php56-dom: 5.6.21
expiretable: 0.6_1
lighttpd: 1.4.39_1
wol: 0.7.1_2
indexinfo: 0.2.4
choparp: 20150613
opnsense-lang: 16.1.14
dnsmasq: 2.75_2,1
nettle: 3.2
gmp: 5.1.3_3
php56-json: 5.6.21
pftop: 0.7_6
pecl-radius: 1.3.0
python27: 2.7.11_2
libffi: 3.2.1
py27-requests: 2.9.1
py27-setuptools27: 20.0
zip: 3.0_1
strongswan: 5.4.0
py27-Jinja2: 2.8
py27-Babel: 2.3.3
py27-pytz: 2016.4,1
py27-MarkupSafe: 0.23
php56-sockets: 5.6.21
php56-filter: 5.6.21
beep: 1.0_1
isc-dhcp43-server: 4.3.3P1_1
filterdns: 0.2
ca_root_nss: 3.22.2
mpd4: 4.4.1_3
libpdel: 0.5.3_6
mpd5: 5.8
squid: 3.5.19
perl5: 5.20.3_13
bsdinstaller: 2.3_5
cpdup: 1.18
isc-dhcp43-client: 4.3.3P1_1
bind910: 9.10.4
idnkit: 1.0_5
libedit: 3.1.20150325_2
ngattach: 1.2
apinger: 0.6.1_9
php56-hash: 5.6.21
php56-pdo: 5.6.21
openssh-portable: 7.2.p2,1
ifinfo: 10.1
filterlog: 0.2
igmpproxy: 0.1_2,1
php56-openssl: 5.6.21
py27-netaddr: 0.7.18
php56-gettext: 5.6.21
gettext-runtime: 0.19.7
samplicator: 1.3.7.b6_2
php56-mcrypt: 5.6.21
libltdl: 2.4.6
libmcrypt: 2.5.8_3
bsnmp-regex: 0.6_1
php56-ldap: 5.6.21
openldap-client: 2.4.44
rrdtool12: 1.2.30_7
libart_lgpl: 2.3.21_2,1
png: 1.6.21
freetype2: 2.6.3
py27-ujson: 1.33
rate: 0.9_1
php56-curl: 5.6.21
curl: 7.48.0_2
php56-session: 5.6.21
php56-xml: 5.6.21
suricata: 3.0.1
libnet: 1.1.6_4,1
GeoIP: 1.6.9
libyaml: 0.1.6_2
jansson: 2.7_3
libhtp: 0.5.18
libiconv: 1.14_9
minicron: 0.0.2
dhcp6: 20080615_7
py27-sqlite3: 2.7.11_7
sqlite3: 3.12.2
php56-ctype: 5.6.21
openvpn: 2.3.11
lzo2: 2.09
easy-rsa: 3.0.1_1
ntp: 4.2.8p7
bsnmp-ucd: 0.4.2
sudo: 1.8.16
syslogd: 10.2
clog: 1.0.1_3
miniupnpd: 1.9.20160113,1
php56-simplexml: 5.6.21
relayd: 5.5.20140810_1
php56-sqlite3: 5.6.21
opnsense-update: 16.1.14
libucl: 0.8.0
flowd: 0.9.1_3
cpustats: 0.1
php-suhosin: 0.9.38
sshlockout_pf: 0.0.2_2
p7zip: 15.14
phalcon: 2.0.10
The process will require 379 MiB more space.
79 MiB to be downloaded.
Fetching opnsense-16.1.15.txz: 100% 9 MiB 3.1MB/s 00:03
Fetching php56-zlib-5.6.21.txz: 100% 14 KiB 14.4kB/s 00:01
[... lots of fetching ...]
Checking integrity... done (0 conflicting)
[1/116] Installing indexinfo-0.2.4...
[1/116] Extracting indexinfo-0.2.4: 100%
[2/116] Installing openssl-1.0.2_12...
[2/116] Extracting openssl-1.0.2_12: 100%
[... lots of installing ...]
Message from opnsense-16.1.15:
ACME delivery for the crafty coyote!
!!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!!!
! A kernel/base upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Fetching kernel-16.1.14-amd64.txz: ............ done
Fetching base-16.1.14-amd64.txz: .................... done
Fetching base-16.1.14-amd64.obsolete: ....... done
Installing kernel-16.1.14-amd64.txz: ... done
Installing base-16.1.14-amd64.txz: ... done
Installing base-16.1.14-amd64.obsolete: ... done
Please reboot.
tar: Removing leading '/' from member names
tar: Removing leading '/' from member names
Shutdown NOW!
shutdown: [pid 29331]
[2.3.1-RELEASE][admin@pfSense.localdomain]/root:
*** FINAL System shutdown message from admin@pfSense.localdomain ***
System going down IMMEDIATELY
-
Nice freature.
Thanks
-
Can I install OPNsense on freeBSD 11 using bootstrap?
Why?: It seems I have some hardware compatibility issues in regards to FreeBSD 10.3, it takes forever to boot from USB but ver. 11 boot just fine.
Board: Asus H110T, chipset Intel H110.
I've tried to use bootstrap but got this:
root@opnsense2:/tmp # sh ./opnsense-bootstrap.sh
Must be a FreeBSD 10.x release.
*UPDATE*
Problem solved....on my board the bootloader is failing unless using UEFI. And to make UEFI work I had to disable the Win10-sercure-bios-boot-whatever-stuff.
Now I can boot FreeBSD 10.3 :-)
No work-around needed anyway :-P
-
Ha - this is the sort of fun that I love. Beating 'the system' :-)
Well done !
-
opnsense-bootstrap reuses compiled sets that we do for our images and firmware updates. As such, you'll get dragged back to whatever OPNsense is currently using in attempt to restore a good working environment. 16.7 is based on FreeBSD 10.3. Only time will tell whether or not FreeBSD 11.0 will be in 17.1.
So far we know that several things don't work there until we start working through them:
o Wifi support as Shawn has reported during his 11-CURRENT HardenedBSD builds
o Firmware updates crossing major ABI boarders (FreeBSD 10 to FreeBSD 11)
o em(1) netmap weirdness with extended descriptor format changes since FreeBSD 10.3
o Forward-porting the stf(4) patch that was inherited from pfSense
o FreeBSD releasing a final 11.0 and maybe one or two errata on the side ;)
o All the still-to-be-uncovered oddities that need ironing out before we can offer a smooth FreeBSD 11.0 experience
Yes, the build system can be coerced to use FreeBSD's vanilla ports tree and with minor adaptions one can easily assemble a booting system, but what lies beyond we do not yet know. :)
Cheers,
Franco
-
Hi Franco,
Apologies for resurrecting a zombie thread, but related to my issues installing OPNSense on a Netgate SG-2440 (thread here (https://forum.opnsense.org/index.php?topic=4596.0)), should this bootstrap script work to convert my current pfSense v2.3.3-RELEASE installation to OPNSense v17.1 (in theory)?
-
Hi vocatus,
I really recommend a clean FreeBSD 11.0 install now. 2.3.3 is still FreeBSD 10.3 and the bootstrap script doesn't support this anymore.
Cheers,
Franco