OPNsense Forum

Archive => 18.1 Legacy Series => Topic started by: franco on August 03, 2016, 08:34:16 am

Title: [OBSOLETE] How to test the development version
Post by: franco on August 03, 2016, 08:34:16 am
Hello everyone,

The development package is available now for the upcoming 17.1 and updated alongside each stable release. You can read about its most prominent changes here:

https://forum.opnsense.org/index.php?topic=3478.0

With 16.7.1 and up, it's possible to switch to the development version by invoking this command on a shell:

# opnsense-update -t opnsense-devel

Switching back to the release version is done by typing the following instead:

# opnsense-update -t opnsense

On top of using the opnsense-devel package, it's now (as of 16.7.10) possible to upgrade to the next FreeBSD 11.0 underneath as well by invoking the console menu item 12, typing "17.1.b" at the prompt.

THERE IS NO SAFE WAY TO UNDO THIS, PLEASE MAKE SURE YOU BACK UP YOUR CONFIGS, SNAPSHOT YOUR VMS AND/OR USE APPROPRIATE TEST SETUPS.

FURTHERMORE, THE UPDATE SETS ARE UPDATED INFREQUENTLY UNTIL THE RELEASE CANDIDATE PHASE IS REACHED.

It's possible to move back to the stock 16.7, but newer files can linger in the file system preventing the downgrade or proper operation thereof.

# opnsense-update -ur 16.7
# /usr/local/etc/rc.reboot

Switching the top package is also required for using the upgrade mechanism from the git repository to get even newer changes:

# opnsense-code core
# cd /usr/core
# git pull
# opnsense-update -t opnsense-devel
# make upgrade CORE_ABI=17.1


Cheers,
Franco
Title: Re: How to test the development version
Post by: tillsense on August 11, 2016, 09:01:48 pm
Quote
# pkg install git
# cd /usr
# git clone https://github.com/opnsense/core
# cd core
# make package-keywords
# opnsense -t opnsense-devel
# make upgrade

small typo

Quote
...
#opnsense-update -t opnsense-devel
...

cheers till
Title: Re: How to test the development version
Post by: franco on August 12, 2016, 07:06:45 am
Nice catch, sorry, fixed now. :)
Title: Re: How to test the development version
Post by: sfn on October 13, 2016, 11:10:52 pm
If we upgrade our opnsense like that will be opnsense beta version of 17 ? Can we use multiwan with Squid Proxy ?
Title: Re: How to test the development version
Post by: franco on October 14, 2016, 09:24:27 am
Not yet, I'm still working on patches with FreeBSD.
Title: Re: How to test the development version
Post by: franco on October 29, 2016, 04:32:23 pm
Hey everyone,

I've put up instructions for upgrading into the FreeBSD 11.0 sets above, but please be aware that 17.1 is currently ALPHA status and anything can happen there.


Cheers,
Franco
Title: Re: How to test the development version
Post by: fabian on October 29, 2016, 05:50:33 pm
@franco: Is the wireless code ready for 11?
Title: Re: How to test the development version
Post by: SOUK on October 29, 2016, 06:01:18 pm
Hey everyone,

I've put up instructions for upgrading into the FreeBSD 11.0 sets above, but please be aware that 17.1 is currently ALPHA status and anything can happen there.


Cheers,
Franco

I will give it a go... try my luck!


Title: Re: How to test the development version
Post by: franco on October 29, 2016, 06:31:29 pm
Wireless code is not adapted, but that may be a thing for the brave to look into. The packages on the mirrors are a full batch, you can install git, vim-lite, php-xdebug and so forth...

Two amd64 images here as well just for fun:

https://pkg.opnsense.org/snapshots/OPNsense-17.1.a-OpenSSL-cdrom-amd64.iso.bz2
https://pkg.opnsense.org/snapshots/OPNsense-17.1.a-OpenSSL-serial-amd64.img.bz2

Cheers,
Franco
Title: Re: How to test the development version
Post by: lattera on October 29, 2016, 07:13:03 pm
One big piece to note is that both base (the underlying operating system itself) and ports (the third-party packages, like Suricata) are all compiled as Position-Independent Executables (PIEs). That means that every application has ASLR fully applied to it.

OPNsense 16.7.7 has PIE applied to base only. PIEified ports will launch with 17.1. Really good stuff to see.

I'm hoping to land SEGVGUARD in time for 17.1. SEGVGUARD provides ASLR bruteforce protection.
Title: Re: How to test the development version
Post by: SOUK on October 29, 2016, 09:14:39 pm
One big piece to note is that both base (the underlying operating system itself) and ports (the third-party packages, like Suricata) are all compiled as Position-Independent Executables (PIEs). That means that every application has ASLR fully applied to it.

OPNsense 16.7.7 has PIE applied to base only. PIEified ports will launch with 17.1. Really good stuff to see.

I'm hoping to land SEGVGUARD in time for 17.1. SEGVGUARD provides ASLR bruteforce protection.


I followed the update procedure, does this look right?

OPNsense 17.1.a_549-amd64?
FreeBSD 11.0-RELEASE-p2?
OpenSSL 1.0.2j 26 Sep 2016?


Also, why is it i can never install Opnsense via USB with any USB installation method via Rufus it always fails.  I'm only ever able to re-install Opnsense via cdrom which is a right pain cos all my cd's are scratched to **** so my install takes like 20minutes via DVD due to read errors..  ;D LOL

USB install works fine with that other sense router software, is their anything i can do about that?  I've got like 2 rusty old DVD's left and im pretty sure they're unusable now..   ;D
Title: Re: How to test the development version
Post by: weust on October 29, 2016, 11:31:59 pm
Are you using a USB 3.0 stick? Those *might* cause problems on some hardware.
Title: Re: How to test the development version
Post by: franco on October 30, 2016, 10:41:50 am
I followed the update procedure, does this look right?

OPNsense 17.1.a_549-amd64?
FreeBSD 11.0-RELEASE-p2?
OpenSSL 1.0.2j 26 Sep 2016?

Yes, looks good.

Also, why is it i can never install Opnsense via USB with any USB installation method via Rufus it always fails.  I'm only ever able to re-install Opnsense via cdrom which is a right pain cos all my cd's are scratched to **** so my install takes like 20minutes via DVD due to read errors..  ;D LOL

Windows has a bug reading GPT, that's why Rufus can/won't always work correctly. https://github.com/pbatard/rufus/wiki/FAQ#problematic-images

Any Unix works here, I really don't know what else to say. GPT has been a standard for over a decade, and just last week I flashed a memstick that would later not work on a windows pc, because it wasn't MBR...

USB install works fine with that other sense router software, is their anything i can do about that?  I've got like 2 rusty old DVD's left and im pretty sure they're unusable now..   ;D

In pfSense, there is no GPT or UEFI. I will be happy to see them switch, which either shows we're not doing the best work we could or that they run into the very same issue. I'm ok with both outcomes. ;)


Cheers,
Franco
Title: Re: How to test the development version
Post by: mais_um on November 04, 2016, 05:08:43 pm
Quote
In pfSense, there is no GPT or UEFI. I will be happy to see them switch, which either shows we're not doing the best work we could or that they run into the very same issue. I'm ok with both outcomes. ;)

This specific forum is about alfa version right? of course pfsense have a GPT UEFI compatible version is alpha and i have installed my home router and works great (on a GPT ), on virtualbox have a GPT and UEHI working to.

But i'm not here to defend pfsense. I like OpnSense interface way better and i'm study if i will make the transition for me and for my clients. Write now i need to see if i have the same services/features (snort/suricata, freeradius, squid, VPNs, traffic shaper - HFSC or equivalent ) and working ok. But i ended with a repository problem after trying de opnsense-devel. Versions    OPNsense 17.1.a_531-amd64 - FreeBSD 11.0-RELEASE-p2 -
OpenSSL 1.0.2j 26 Sep 2016. Can i do something to solve this rep problem?
Title: Re: How to test the development version
Post by: mais_um on November 04, 2016, 05:24:59 pm
Trying redoing this and stuck on "make package-keyords" the output gives "make: don't know how to make package-keyords. stop" sounds missing some tool.

Edit: nothing like installing everything again.
Title: Re: How to test the development version
Post by: franco on November 07, 2016, 06:54:11 am
"make package-keywords" was a workaround for a particular quirk when building packages that really needed the /usr/ports tree, or very few parts of it. Another workaround has been put into place meanwhile, and the step became unnecessary. I've updated the documentation.

I'm not entirely sure how stuck on "make package-keywords" warrants a reinstall. Each step should tell you what's wrong or what command needs to be run to fix it. If there is a particular error from a working command, the output should be posted in order to fix the things that don't do that yet.

17.1 is alpha, as is 2.4. But case in point GPT/UEFI was shipped in July 2016 along with 16.7. ;)


Thanks,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 15, 2016, 04:23:20 am
Finally took the plunge and did the upgrade.

Only issues iv ran into thus far are now I cant SSH and UPnP is broken.

Those 2 services

miniupnpd
sshd

refuse to start. Even after reboots. Changing the gui has no effect.

I also wasnt able to run the commands to allow updates by isntalling GIT, because I cant SSH into the unit. I will need to wait until I can console in.
Title: Re: How to test the development version
Post by: franco on November 15, 2016, 05:04:57 pm
It sounds like the underlying packages haven't been properly reinstalled similar to what we had here:

https://forum.opnsense.org/index.php?topic=3909.0

We're looking into it.

In the meantime, which command sequence did you use to install? Which version was active before changing to 17.1.a?


Thanks,
Franco
Title: Re: How to test the development version
Post by: franco on November 15, 2016, 10:39:12 pm
Was this a LibreSSL install prior to upgrade?
Title: Re: How to test the development version
Post by: Solaris17 on November 16, 2016, 01:38:56 am
Hey Franco, Sorry its taken so long to respond.

I was running the alpha 17.x on BSD 10.3. I dont remember exact versioning. sorry.

It was an openSSL build. Though funny you should mention I could have sworn I had libreSSL installed at one point and I have not formatted.

I used the update command sequence in the OP of this thread.

You wouldnt have any idea how to maybe get this patched up would you? Any way I might be able to just pkg install the missing dependencies? I will format if needed but if there is a chance I can pull up the plane ill do it.

Thanks a bunch let me know if I can be of any assistance Franco!
Title: Re: How to test the development version
Post by: franco on November 16, 2016, 07:31:45 am
It should be fairly easy given internet+dns works:

# opnsense-update -sn "17.1\/latest"
# opnsense-update -fp

It should reinstall all packages, if not please let me know the error encountered.

After successful run, reboot from the GUI and services should be back up afterwards.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 16, 2016, 07:37:20 pm
Hey thanks for taking the time out of your day to give me the commands. I dont know the repos well enough to be able to guess. Im at work ATM but till give it a try tonight.
Title: Re: How to test the development version
Post by: tillsense on November 16, 2016, 07:41:52 pm
that's really interesting - a new install the repo is 16.7 after this is 17.1 but DOWNGRADED from 17.1.a_673 -> 17.1.a_539

Code: [Select]
*** test.loca: OPNsense 17.1.a_673 (amd64/OpenSSL) ***

 LAN (re0)       -> v4: 192.168.1.1/24
 WAN (pppoe0)    -> v4/PPPoE: *.*.*.*

 0) Logout                             7) Ping host
 1) Assign Interfaces                  8) Shell
 2) Set interface(s) IP address        9) pfTop
 3) Reset the root password           10) Filter Logs
 4) Reset to factory defaults         11) Restart web interface
 5) Power off system                  12) Upgrade from console
 6) Reboot system                     13) Restore a configuration

Enter an option: 8       

root@test:~ # cat /usr/local/etc/pkg/repos/origin.conf
OPNsense: {
  fingerprints: "/usr/local/etc/pkg/fingerprints/OPNsense",
  url: "pkg+http://pkg.opnsense.org/${ABI}/16.7/latest",
  signature_type: "fingerprints",
  mirror_type: "srv",
  priority: 11,
  enabled: yes
}
root@test:~ # opnsense-update -sn "17.1\/latest"
root@test:~ # opnsense-update -fp
Updating OPNsense repository catalogue...
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01   
Fetching packagesite.txz: 100%   94 KiB  95.8kB/s    00:01   
Processing entries: 100%
OPNsense repository update completed. 307 packages processed.
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Updating database digests format: 100%
Checking for upgrades (121 candidates): 100%
Processing candidates (121 candidates): 100%
The following 121 package(s) will be affected (of 0 checked):

Installed packages to be DOWNGRADED:
opnsense-devel: 17.1.a_673 -> 17.1.a_539

Installed packages to be REINSTALLED:
pkg-1.9.1
zip-3.0_1
wol-0.7.1_2
unbound-1.5.10
syslogd-10.3
suricata-3.1.2
sudo-1.8.18
strongswan-5.5.0
sshlockout_pf-0.0.2_2

---------------------------------------------------------------------------------snip

/var/cache/pkg/beep-1.0_1-b2d78e37a3.txz
/var/cache/pkg/bind910-9.10.4P3.txz
/var/cache/pkg/beep-1.0_1.txz
/var/cache/pkg/apinger-0.7-38946bab0c.txz
/var/cache/pkg/apinger-0.7.txz
/var/cache/pkg/GeoIP-1.6.9-3fe84bba19.txz
/var/cache/pkg/GeoIP-1.6.9.txz
The cleanup will free 76 MiB
Deleting files: 100%
All done
root@test:~ #                                                                               
*** FINAL System shutdown message from root@test.loca ***             

System going down IMMEDIATELY                                                 

                                                                               
Connection to 192.168.1.1 closed by remote host.
Connection to 192.168.1.1 closed.
root@test:~$ ssh root@192.168.1.1
Password for root@test.loca:
Last login: Wed Nov 16 17:49:50 2016 from 192.168.1.101
FreeBSD 11.0-RELEASE-p2 (SMP) #0 850e1e9(master): Fri Oct 28 17:09:55 CEST 2016

----------------------------------------------
| Hello, this is OPNsense 16.7!              |       @@@@@@@@@@@@@@@
|                                            |      @@@@         @@@@
| Website: https://opnsense.org/        |       @@@\\\   ///@@@
| Handbook: https://docs.opnsense.org/   |     ))))))))   ((((((((
| Forums: https://forums.opnsense.org/ |       @@@///   \\\@@@
| Lists: https://lists.opnsense.org/  |      @@@@         @@@@
| Code: https://github.com/opnsense  |       @@@@@@@@@@@@@@@
----------------------------------------------

 0) Logout                             7) Ping host
 1) Assign Interfaces                  8) Shell
 2) Set interface(s) IP address        9) pfTop
 3) Reset the root password           10) Filter Logs
 4) Reset to factory defaults         11) Restart web interface
 5) Power off system                  12) Upgrade from console
 6) Reboot system                     13) Restore a configuration

Enter an option: 8

root@test:~ # cat /usr/local/etc/pkg/repos/origin.conf
OPNsense: {
  fingerprints: "/usr/local/etc/pkg/fingerprints/OPNsense",
  url: "pkg+http://pkg.opnsense.org/${ABI}/17.1/latest",
  signature_type: "fingerprints",
  mirror_type: "srv",
  priority: 11,
  enabled: yes
}
Title: Re: How to test the development version
Post by: franco on November 16, 2016, 08:10:51 pm
Hi Till,

That's normal as the 17.1.a upgrade installs an older snapshot--it hasn't received an update in a few weeks. But anyway, it looked like it works as expected there.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 17, 2016, 02:57:51 am
ran the updates via console and everything is fine now on the following.

OPNsense 17.1.a_539-amd64
FreeBSD 11.0-RELEASE-p2
OpenSSL 1.0.2j 26 Sep 2016
Title: Re: How to test the development version
Post by: franco on November 24, 2016, 12:36:10 pm
Neat, pkg 1.9.x has a bug that prevents installation of packages because it tries to drop privileges and thus can't read the directory owned by root.

EDIT: OTOH, I'm relieved that the upgrades do work and we can explain the missing packages transition reported recently.
Title: Re: How to test the development version
Post by: Solaris17 on November 24, 2016, 04:45:36 pm
Neat, pkg 1.9.x has a bug that prevents installation of packages because it tries to drop privileges and thus can't read the directory owned by root.

EDIT: OTOH, I'm relieved that the upgrades do work and we can explain the missing packages transition reported recently.

Wow thats odd. Does BSD know about this yet?
Title: Re: How to test the development version
Post by: franco on November 24, 2016, 05:08:28 pm
Considering that the fix was backported 13 days ago, a week after pkg 1.9.3 came out and the commit message reads "Stop dropping privileges when fetching as it causes more issues than it solves", they know about it but 1.9.4 isn't out so the impact in FreeBSD itself is likely low.

https://github.com/freebsd/pkg/commit/504cff94

Various project use pkg in different ways, report bugs upstream, have different side-effects from changes. I'm just glad that wasn't injected into a production environment as it only affects upgrading to 17.1-ALPHA at this point.

Packages are rebuilt, I'm fixing the upgrade instructions now.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 24, 2016, 05:12:04 pm
On that topic Were the commands at the end of the OP supposed to bring people upto the GIT version? Because the base install only currently goes upto 539. After the mishap with upgrading those commands don't really work, stating that the directory already exists etc. I would remove it to attempt it again but i'm uncertain what was touched in the process originally.
Title: Re: How to test the development version
Post by: franco on November 24, 2016, 05:17:47 pm
17.1.a_539 is still recent, mostly because each build requires changes that are done out of tree to avoid breaking 16.7 progress. The exercise is testing 11.0 and the upgrades more than the GUI code in this case.

I'm not sure if the installation can be unbroken easily. Which command is failing for you particularly?
Title: Re: How to test the development version
Post by: Solaris17 on November 24, 2016, 05:20:38 pm
hm, here ya go. Maybe its a misinterpretation on my part on what they are supposed to do. However I was under the impression it would allow updates to the _7xx builds

Code: [Select]
root@TDE-Core-Router:~ # opnsense-code core
fatal: destination path '/usr/core' already exists and is not an empty directory.
root@TDE-Core-Router:~ # cd /usr/core
root@TDE-Core-Router:/usr/core # opnsense-update -t opnsense-devel
The package type 'opnsense-devel' is already installed.
root@TDE-Core-Router:/usr/core # make upgrade
/usr/core/+POST_INSTALL -> /usr/core/work/src/+POST_INSTALL
/usr/core/+PRE_DEINSTALL -> /usr/core/work/src/+PRE_DEINSTALL
>>> Missing dependency: ngattach
*** Error code 1

Stop.
make[1]: stopped in /usr/core
*** Error code 1

Stop.
make: stopped in /usr/core
root@TDE-Core-Router:/usr/core # pkg install ngattach
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
pkg: No packages available to install matching 'ngattach' have been found in the repositories
root@TDE-Core-Router:/usr/core #
Title: Re: How to test the development version
Post by: franco on November 24, 2016, 05:26:59 pm
>>> Missing dependency: ngattach

That hasn't been on master since September 16, but you can simply delete the one line from /usr/core/Makefile

opnsense-code is for getting the code, it's already there so error is ok.
opnsense-update -t is for running the development version, already installed :)


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 24, 2016, 05:32:00 pm
Has anyone else reported this? I figured the other two made sense, but I was unable to run any of that initially because it failed at ngattach from the very beginning even when the other two stages completed fine initially.
Title: Re: How to test the development version
Post by: franco on November 24, 2016, 05:34:15 pm
It must have been an early time in 17.1-ALPHA when you set up this machine.

Are the packages fine now? If you have connectivity back, run this:

# cd /usr/core
# git pull
# make upgrade


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on November 24, 2016, 05:42:21 pm
It must have been an early time in 17.1-ALPHA when you set up this machine.

Are the packages fine now? If you have connectivity back, run this:

# cd /usr/core
# git pull
# make upgrade


Cheers,
Franco

Very early, I have been running the betas and upgrading into them steadily since early 16.x. The packages were fine and I was able to correct SSH with your help from before. The above commands worked with no issue. I did have to git reset --hard for my modification to Makefile in removing the dependency, wouldnt have worked anyway because I was also missing p7zip.
Title: Re: How to test the development version
Post by: franco on November 30, 2016, 09:14:19 am
Service annoucement: 17.1.a is being removed from the mirrors in preparation for 17.1.b. ETA 1-2 weeks. Before Christmas in any case. :D
Title: Re: How to test the development version
Post by: Solaris17 on December 04, 2016, 06:54:58 am
Exciting! Hopefully we will have an upgrade path?  :-X
Title: Re: How to test the development version
Post by: franco on December 08, 2016, 11:38:47 pm
Yes, builds have started, then were a affected by FreeBSDs security advisories and security advisory corrections :)
Title: Re: How to test the development version
Post by: tillsense on December 10, 2016, 09:32:10 am
Hi Franco,

do you have a current amd64 image?

chers till
Title: Re: How to test the development version
Post by: franco on December 10, 2016, 09:00:07 pm
Hi Till,

Images will be out next week. The kerne/base sets are done, only package builds and last preparations (changelogs and image testing) remain. :)


Cheers,
Franco
Title: Re: How to test the development version
Post by: franco on December 14, 2016, 03:11:56 pm
17.1.b beta for OpenSSL (amd64 and i386) online upgrades are already up and running. With 16.7.11 installed, you can do:

# opnsense-update -t opnsense-devel

(if you don't use the opnsense-devel already)

Then just run console option 12) typing "17.1.b" at the prompt.

Images and release annoucement follow in the next days.


Cheers,
Franco
Title: Re: How to test the development version
Post by: lattera on December 14, 2016, 06:36:02 pm
I did a two-stage update remotely (I'm at work, my OPNsense firewall is at home) successfully.

First, I updated from 16.7.10 to 16.7.11. Then I used your instructions to update from 16.7.11 to 17.1-beta.

Here is a nice screenshot: https://goo.gl/photos/rqu3ncheSFDaKigm7
Title: Re: How to test the development version
Post by: tillsense on December 14, 2016, 07:27:19 pm
great work! post from beta  :)
Title: Re: How to test the development version
Post by: Solaris17 on December 15, 2016, 03:35:51 am
My machine cant find the repository. This happened after I did the git pull several posts ago. I have just continued to do them since I cannot update normally.

EDIT:: was able to back up to 17.1.b from 17.1.b_9 by

# opnsense-update -sn "17.1\/latest"
# opnsense-update -fp

Title: Re: How to test the development version
Post by: franco on December 15, 2016, 06:40:16 am
@Solaris17

The command for upgrading from the source repo is:

# make upgrade CORE_ABI=17.1


Cheers,
Franco
Title: Re: How to test the development version
Post by: franco on December 16, 2016, 11:47:53 am
Instructions in the original post have been updated.
Title: Re: How to test the development version
Post by: Dean E. Weimer on December 16, 2016, 04:25:08 pm
My system doesn't seem to want to update, if I do the make upgrade CORE_ABI=17.1 step I get this.

Code: [Select]
#make upgrade CORE_ABI=17.1
/usr/core/+POST_INSTALL -> /usr/core/work/src/+POST_INSTALL
/usr/core/+PRE_DEINSTALL -> /usr/core/work/src/+PRE_DEINSTALL
>>> Missing dependency: bind911
*** Error code 1

when I did the opnsense-update -t opnsense-devel it did install some updates, the system reports its up-to-date and even shows 17.1.b14 as the version, but its still running 10.3, and still has PHP 5.6 packages instead of the php 7.0
Title: Re: How to test the development version
Post by: franco on December 16, 2016, 04:30:59 pm
Delete bind911 dependency line from Makefile and rerun. The system will fix itself, although it may be that more steps are needed since it's not on 11.0 yet.

What does this do:

# opnsense-verify -a


Cheers,
Franco
Title: Re: How to test the development version
Post by: Dean E. Weimer on December 16, 2016, 04:40:13 pm
root@opnsense:/usr/core # opnsense-verify -a
FreeBSD:10:amd64
Title: Re: How to test the development version
Post by: Dean E. Weimer on December 16, 2016, 04:44:16 pm
It appears to be stopping on everything uses the variable ${CORE...} that is missing, so the bigger question is why isn't it updating the base OS when switching to the devel branch?
Title: Re: How to test the development version
Post by: franco on December 16, 2016, 04:54:34 pm
Because the vital step is missing? :)

Console Option 12, type "17.1.b".

Make sure you have pkg 1.9.3_1 installed, pkg 1.9.3 is buggy:

# pkg query %v pkg


Cheers,
Franco
Title: Re: How to test the development version
Post by: Dean E. Weimer on December 16, 2016, 06:10:01 pm
OK, when doing option 12 it had 17.1.b listed, but apparently typing yes didn't suffice, I typed 17.1.b instead of y and it downloaded and installed the rest of the updates. so its running normally now after the reboot.
Title: Re: How to test the development version
Post by: franco on December 16, 2016, 08:38:49 pm
Perfect :)
Title: Re: How to test the development version
Post by: tillsense on December 19, 2016, 08:23:24 pm
hi franco,
Instructions in the original post have been updated.

from the second time the howto should be supplemented: (if this is your preferred way)
Code: [Select]
# cd /usr/core/
# git pull
# make upgrade CORE_ABI=17.1

cheers till
Title: Re: How to test the development version
Post by: franco on December 20, 2016, 03:55:50 pm
I added the git pull in there, thanks!
Title: Re: How to test the development version
Post by: Julien on December 30, 2016, 12:34:15 am
is the 17.1 will fix the multi wan issue?
hopefully it will.
Title: Re: How to test the development version
Post by: franco on December 30, 2016, 07:49:16 am
Yes, we have started a new test round yesterday for IPv4 only. We need to see if it is reliable enough for 17.1 in the next week.

The same work is going into FreeBSD as well, but covers more use cases so that work progresses a lot slower (hopefully FreeBSD 12.0).
Title: Re: How to test the development version
Post by: franco on December 30, 2016, 06:05:53 pm
Here is a test kernel, participation welcome.... https://forum.opnsense.org/index.php?topic=4170
Title: Re: How to test the development version
Post by: mmorev on January 14, 2017, 09:01:34 am
Hi! I want to try develop some additions to opnsense Web Proxy module and append sssd package/plugin. How should I build current 17.1 images from sources for testing? There are instructions for building 16.7 on github.com/opnsense/tools page, when I follow them, make cdrom says:
Code: [Select]
>>> Running build step: ports
>>> /usr/src does not match expected branch: stable/16.7
>>> To contunue anyway set SRCBRANCH=master
*** Error code 1
And if I set SRCBRANCH to master, build fails on ngattach, which afaik shouldn't be there for 17.1 (link (https://forum.opnsense.org/index.php?topic=3912.msg14145#msg14145)).

Btw, 2 more questions:
- Should building vm have exactly 6 GB of RAM as said on tools page, or I can assign, for example, 5,5? My pc has only 8 and goes slow.
- For building 17.1 should I use FreeBSD 10.3 or 11.0?
Title: Re: How to test the development version
Post by: franco on January 14, 2017, 09:39:52 am
# make cdrom SETTINGS=17.1

Or edit tools.git/Makefile to default to 17.1 there.

16.7 is the release branch, so the build tools still build it by default. I recently added safeguards to see whether branches match because things like ngattach clashed from time to time with users. As for ngattach itself, it was a very ancient workaround that has since been removed, restoring binary compatibility with FreeBSD. :)

You can try with less RAM. 6 GB was for building "make nano" in RAM, it needed 4 GB for the image. These days it doesn't require RAM anymore, I will change the documentation to say "at least 4 GB" right now.

For 17.1 you can use FreeBSD 10.3 or 11.0, but since 17.1 is based on 11.0 building the initial FreeBSD sets (base and kernel) is faster on 11.0. I recommend 11.0 as it should also build 16.7 if need be.


Cheers,
Franco

Title: Re: How to test the development version
Post by: mmorev on January 15, 2017, 06:28:52 am
Thanks!
I have a new issue when building:
Code: [Select]
>>> Setting up /usr/core copy in /usr/obj/usr/tools/config/17.1/OpenSSL:amd64
fatal: ambiguous argument 'stable/17.1': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
*** Error code 128

Stop.
make: stopped in /usr/tools

There is core folder mentioned, i have forked its master branch to my own repo (https://github.com/mmorev/opnsense) and made some minor changes in code. Could that be a cause of tools build failure?
Title: Re: How to test the development version
Post by: franco on January 15, 2017, 05:42:02 pm
core.git's stable/17.1 branch was added a few days ago, simply do:

# cd /usr/core && git pull


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on January 15, 2017, 05:43:07 pm
core.git's stable/17.1 branch was added a few days ago, simply do:

# cd /usr/core && git pull


Cheers,
Franco

Thanks I am also going to give this a shot because my beta install is having issues updating every way thats posted.

EDIT:: Bah my main fear is not being able to upgrade into the RC or stable editions. currently checking for firmware updates yeilds no results (Im assuming this is intended) and git pull / make install requests are now failing. Bind910 among other dependencies it says it needs.

Should I really be going line by line and deleting these? Can I download a fresh copy of the file without the old entries.
Title: Re: How to test the development version
Post by: franco on January 15, 2017, 06:00:26 pm
Hey Solaris,

Some packages changed, some got renamed in FreeBSD (openvpn -> openvpn23), which can only be tied together in points where we release a version like 17.1.b or the upcoming 17.1.r1 next week.

I think most of your trouble can be fixed with:

# make upgrade CORE_ABI=17.1

I expect the shifting of build bits to stop once 17.1.r1 (17.1-RC1) is out, because then it will be a full build like 16.7 has been for the last couple of months.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on January 15, 2017, 06:07:22 pm
Hey Solaris,

Some packages changed, some got renamed in FreeBSD (openvpn -> openvpn23), which can only be tied together in points where we release a version like 17.1.b or the upcoming 17.1.r1 next week.

I think most of your trouble can be fixed with:

# make upgrade CORE_ABI=17.1

I expect the shifting of build bits to stop once 17.1.r1 (17.1-RC1) is out, because then it will be a full build like 16.7 has been for the last couple of months.


Cheers,
Franco

Hey Frano,

I appreciate the advice and I have seen that in the OP. Unfortunetely I have the same issues even with core ABI updates. IE

Code: [Select]
root@TDE-Core-Router:/usr/core # make upgrade CORE_ABI=17.1
/usr/core/+POST_INSTALL -> /usr/core/work/src/+POST_INSTALL
/usr/core/+PRE_DEINSTALL -> /usr/core/work/src/+PRE_DEINSTALL
>>> Missing dependency: openvpn23
*** Error code 1

Stop.
make[1]: stopped in /usr/core
*** Error code 1

Stop.
make: stopped in /usr/core
root@TDE-Core-Router:/usr/core #

I run into the same issues via

git pull
make install

I also cant (obviously) update via the firmware section in th GUI itself. What concerns me is that both of the GIT methodes fail and im concerned I won't be able to dig myself out to get into the RC and Stable builds.

EDIT:: I have even tried updating from console and reinstalling 17.1_b in the hopes it would fix whatever issues I was having.
Title: Re: How to test the development version
Post by: franco on January 15, 2017, 06:13:35 pm
It's really not an issue to fix this...

The package updates we provide will work when they are published.

In the meantime there cannot be a guarantee that all repositories, in this case ports and core are always 100% in sync. You are currently relying on an older ports repository, but core wants a newer ports repository. In fact, it prevents you from upgrading into a yet unknown state, which is good.

I said it just then, openvpn got renamed to openvpn23, because openvpn package is now version 2.4 but we need to stay on 2.3 a bit longer:

https://github.com/opnsense/core/commit/1d694b2f29db6

In anything we try to do, we try not to brick installs for users. Errors like these are a healthy sign and with a bit of context they do make sense. :)


Cheers,
Franco
Title: Re: How to test the development version
Post by: Solaris17 on January 15, 2017, 06:35:46 pm
Awesome that makes much more sense thank you!
Title: Re: How to test the development version
Post by: Julien on October 18, 2017, 11:36:32 pm
Done the update from17.7 however the Suricada is not starting.
this update is on a home box and not a productions
OPNsense 18.1.a_203-amd64
FreeBSD 11.0-RELEASE-p12
OpenSSL 1.0.2l 25 May 2017
Title: Re: How to test the development version
Post by: DanMc85 on October 19, 2017, 12:44:53 am
I keep getting a Missing dependency: hostapd error when trying to run the make upgrade command.
Title: Re: How to test the development version
Post by: DanMc85 on October 19, 2017, 01:36:10 am
I edited the Makefile to bypass the missing files (wpa_supplicant) was also missing.

OPNsense 18.1.a_261-amd64
FreeBSD 11.0-RELEASE-p12
OpenSSL 1.0.2l 25 May 2017

I thought it was supposed to include FreeBSD 11.1 ?
Title: Re: How to test the development version
Post by: franco on October 19, 2017, 07:23:07 am
Hi Dan,

We haven't fully updated the instructions yet. Normally, going through core.git upgrades the core UI/API package, nothing more.

The preferred method to do this now is simply:

# opnsense-update -t opnsense-devel

Which gives you no trouble with missing dependencies. We update the opnsense-devel package with each release.

This time around, the operating system will be shipped as a separate component to be installed at will. Since we can easily interchange 11.0 for 11.1, you can either choose the opnsense-devel package with it, or the normal release version.

Switching back to the release version:

# opnsense-update -t opnsense

We're still waiting for 17.7.6 to hit (tomorrow) because of an opnsense-update tweak that allows to lock the operating system updates in case people want to make sure they stay on the 18.1-BETA code during subsequent firmware updates. After that is out we will publish the remaining info as an official CFT.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Julien on October 19, 2017, 11:10:07 am
Hi Franco,
any suggestions why the IDS won't start after the update?
Title: Re: How to test the development version
Post by: franco on October 19, 2017, 02:38:45 pm
Which update do you speak of?

Maybe simply this bug?

https://github.com/opnsense/core/issues/1848

Fix via:

# rm /var/run/suricata.pid

We are fixing this in FreeBSD very soon. Suricata also needs updating to 4.0.1.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Julien on October 20, 2017, 04:25:58 pm
Which update do you speak of?

Maybe simply this bug?

https://github.com/opnsense/core/issues/1848

Fix via:

# rm /var/run/suricata.pid

We are fixing this in FreeBSD very soon. Suricata also needs updating to 4.0.1.


Cheers,
Franco
Thank you Franco, yes updating to V18.
just want to inform you about the Suricata it won't start.
Can we update Suricata to 4.0.1 or its not on yet ?
thank you for the great job.
Title: Re: How to test the development version
Post by: franco on October 20, 2017, 04:37:15 pm
Did you delete the PID file and try to start it? ;)
Title: Re: How to test the development version
Post by: Julien on October 20, 2017, 05:25:50 pm
Did you delete the PID file and try to start it? ;)
yes already did but it didn't start, somehow I needed to reboot the box to get stuff up and running.
its working now.
is the latest version or Suricada included on the release?
Title: Re: How to test the development version
Post by: franco on October 20, 2017, 09:10:59 pm
Suricata 4.0.1 is not included. There was no time for testing. I'll issue a test package on Monday.


Cheers,
Franco
Title: Re: How to test the development version
Post by: Julien on October 21, 2017, 12:27:04 pm
Suricata 4.0.1 is not included. There was no time for testing. I'll issue a test package on Monday.


Cheers,
Franco
Thank you Franco,
I have a hardware OPNsense to test this with you guys.
thank you for the effort.
let me know when i can install the new package.
Title: Re: How to test the development version
Post by: franco on October 23, 2017, 06:36:30 pm
Here you go, my friend: https://forum.opnsense.org/index.php?topic=6220.0


Cheers,
Franco
Title: Re: How to test the development version
Post by: cardins2u on January 15, 2018, 09:08:51 pm
@franco

How do you change stable to 18.1.r~ version

I tried

 opnsense-update -t opnsense-devel in SSH

after that I go to web GUI to run the upgrade. It switch it back to opnsense stable version.

What am I missing
Title: Re: How to test the development version
Post by: franco on January 15, 2018, 09:25:10 pm
Hi there,

It will work in 17.7.12... we'll post instructions on Thursday.


Cheers,
Franco
Title: Re: How to test the development version
Post by: cardins2u on January 15, 2018, 09:46:23 pm
@Franco

Awww common, I have to wait till Thursday to test! thats 3 more days!

any work around beside reinstalling?
Title: Re: How to test the development version
Post by: franco on January 15, 2018, 10:14:47 pm
I sent a PM....