[SOLVED] How do I set the development environment?

[iosense]

Hi, I'm reading the documents a few days and trying to find the most appropriate method for development.

I love to use Vagrant(https://www.vagrantup.com/), and convenience that it provides.

I've set up a simple way with FreeBSD 10.1 Vagrant ( FreeBSD 10.1 i386 (UFS, VirtualBox Guest Additions 4.3.20, Python 2.7, Perl 5.18) https://github.com/arkadijs/vagrant-freebsd ) and clone whole OPNsense git repo into opnsense folder.( see folder structure: https://dl.dropboxusercontent.com/u/37848856/vagrant-opnsense.jpg ) These folders synced to the system as follows:

Code: [Select]

config.vm.synced_folder "./opnsense/core", "/usr/core", :nfs => true, id: "opnsense-core"
config.vm.synced_folder "./opnsense/plugins", "/usr/plugins", :nfs => true, id: "opnsense-plugins"
config.vm.synced_folder "./opnsense/ports", "/usr/ports", :nfs => true, id: "opnsense-ports"
config.vm.synced_folder "./opnsense/src", "/usr/src", :nfs => true, id: "opnsense-src"
config.vm.synced_folder "./opnsense/tools", "/usr/tools", :nfs => true, id: "opnsense-tools"

so, we are able to access these folders from a shell screen as shown below:

Code: [Select]

$ cd /usr/core
$ cd /usr/plugins
$ cd /usr/ports
$ cd /usr/src
$ cd /usr/tools

I used the following command as specified in the document:

Code: [Select]

$ cd /usr/core
$ sudo make mount

warning: refname '15.7.12' is ambiguous.
warning: refname '15.7.12' is ambiguous.
mount_unionfs /usr/core/src /usr/local

but I do not think that this is enough, so it seems like the opnsense is not working.

and the following question is; how can i make it work ?

My aim is simply to make some adjustments on the GUI

Thank you.

[AdSchellevis]

Hi,

We have never tried building with Vagrant, but if Vagrant just prepares a standard freebsd environment it should probably work.
The mount command you try to use is mend for an OPNsense machine running the core package from git, more on how this works can be found here:

http://lastsummer.de/development-workflow-in-opnsense/

Which is probably the easiest method to create / alter code some GUI related parts (just install an OPNsense image and start coding).

Information about the build system (create packages/images etc), can be found here:
https://github.com/opnsense/tools

And there's our wiki for more information about our development methods:
https://wiki.opnsense.org/index.php/Developers

Cheers,

Ad

[iosense]

Hey Ad,

Thank you for your reply,

Eventually OPNsense began working on vagrant, but I think I had a problem with the cache system.

I cloned the git core repo to the /root/core folder and add a new theme under the /root/core/src/opnsense/www/themes/opnsense/custom folder. Well, custom theme appears in the settings section but the opnsense is not updated when I select and save it.

I also edit a file for testing purposes:  /root/core/src/opnsense/mvc/app/views/layouts/default.volt
Added some text into the default.volt file but appearance did not change.

How to disable the caching mechanism for GUI development.

[AdSchellevis]

You're welcome, nice to hear it started working on Vagrant as well.

Phalcon saves it's page cache in /usr/local/opnsense/mvc/app/cache/, but keep in mind that the volt templates are only used for the new style components like trafficshaper and proxy.

The theme css is used in both the old and the new code, but if you want to make structural changes (not doable with css), you need to look at the old pages as well. (/usr/local/www/fbegin.inc for example)
Although this is from a maintenance point probably not the best approach.

The advised way of changing the theme is to alter the sass files and compile the css file, the ones for our standard theme can be found in /usr/local/opnsense/www/themes/opnsense/assets/stylesheets/

[iosense]

Hey Ad,

Thank you for your guidance,now I understand the existing structure. I thought that the it was converted completely MVC format but I see that this is only the case for new modules and extended features.

I also would like to report a bug: $g['theme'] = get_current_theme(); is not working correctly for newly added themes.
$g['theme'] returns only opnsense,  theme value can not be saved. see this screencast:  https://db.tt/QDVUgxIJ

Thank you.

[AdSchellevis]

Nice to hear the structure is clear now, eventually we want to migrate all parts to the new structure but this will certainly take quite some time before all old code is gone.

We have chosen to set a new development method, clean-up old code and migrate step by step... it's a lot of work given the complexity of the (unstructured) old code, but with the chosen path we are able to improve gradually.
(a lot of the work is under the hood)

Thanks for the bug report, we probably need to get rid of the global variable as well, I have registered it on our GitHub page:
https://github.com/opnsense/core/issues/388

 

[franco]

Hi klezer,

I think you found an issue that I introduced in March.  The fix is online:

https://github.com/opnsense/core/commit/d89614ab472a59f7372e1dc9a7e4534fa9a164cf

Great to see you're making progress!


Cheers,
Franco

[iosense]

Hey Franco,

Thank you for hotfix global setting works just fine now.

Now I prepare a module by following the instructions, and now I want to build this plugin, but I get an error as follows.

Code: [Select]

$ make plugins
.MAKE.LEVEL.ENV=MAKELEVEL
ARCH=i386
BLOCKSIZE=K
CDROM=/tmp/images/OPNsense-201509240328-OpenSSL-cdrom-i386.iso
CONFIGDIR=/vagrant/opnsense/tools/config/15.7
CONFIG_PKG=/usr/local/etc/pkg/repos/origin.conf
CONFIG_XML=/usr/local/etc/config.xml
COREDIR=/root/core
CPUS=1
EDITOR=vi
GROUP=wheel
HOME=/root
HOST=OPNsense.localhost
HOSTTYPE=FreeBSD
IMAGESDIR=/tmp/images
LABEL=OPNsense
LOGNAME=root
MACHTYPE=i386
MAIL=/var/mail/root
MAKEFLAGS= .MAKE.LEVEL.ENV=MAKELEVEL
MAKELEVEL=1
NANOIMG=/tmp/images/OPNsense-201509240328-OpenSSL-nano-i386.img
OLDPWD=/vagrant/opnsense/tools
OSTYPE=FreeBSD
PACKAGESDIR=/.pkg
PAGER=less
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
PLUGINSDIR=/vagrant/opnsense/plugins
PORTSDIR=/vagrant/opnsense/ports
PORTSREFDIR=/usr/freebsd-ports
PRODUCT_FLAVOUR=OpenSSL
PRODUCT_NAME=OPNsense
PRODUCT_RELEASE=OPNsense-201509240328-OpenSSL
PRODUCT_SETTINGS=15.7
PRODUCT_TYPE=OPNsense-devel
PRODUCT_VERSION=201509240328
PWD=/vagrant/opnsense/tools/build
REMOTEHOST=10.0.1.1
SERIALIMG=/tmp/images/OPNsense-201509240328-OpenSSL-serial-i386.img
SETSDIR=/tmp/sets
SHELL=/usr/local/etc/rc.initial
SHLVL=1
SRCDIR=/vagrant/opnsense/src
SSH_CLIENT=10.0.1.1 54887 22
SSH_CONNECTION=10.0.1.1 54887 10.0.1.10 22
SSH_TTY=/dev/pts/0
STAGEDIR=/usr/local/stage
TARGETARCH=i386
TARGET_ARCH=i386
TERM=xterm
TOOLSDIR=/vagrant/opnsense/tools
USER=root
VENDOR=intel
VGAIMG=/tmp/images/OPNsense-201509240328-OpenSSL-vga-i386.img
>>> Setting up stage in /usr/local/stage
>>> Setting up world in /usr/local/stage
tar: Option -f requires an argument
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
*** Error code 1

Stop.
make: stopped in /vagrant/opnsense/tools

How can I solve this issue?
Could this be due to a missing pkg on FreeBSD ?

Thank you.

[franco]

I think you'll need a base and kernel set, as well as ports afterwards:

# make src ports

(please update all repos, if "src" doesn't work your tools.git is behind)

After that you can run and rerun

# make plugin

as often as you like.

[iosense]

Hey Franco,

Thank you for advice, yeah i just execute only  "make plugin" command,  other deployed operations takes quite long, so I just skipped them but apparently I need them too.

I still had some problems at this stage; make src command worked properly but Unfortunately, make ports command did not work: See the following results

Code: [Select]

root@localhost:/vagrant/opnsense/tools # make src
...
a ./boot/kernel/wlan_wep.ko
a ./boot/kernel/wlan_xauth.ko
a ./boot/kernel/if_wpi.ko
a ./boot/kernel/wpifw.ko
a ./boot/kernel/x86bios.ko
a ./boot/kernel/if_xe.ko
a ./boot/kernel/if_xl.ko
a ./boot/kernel/zfs.ko
a ./boot/kernel/zlib.ko
a ./boot/kernel/iscsi.ko
a ./boot/kernel/iscsi_initiator.ko
a ./boot/kernel/linker.hints
mv dist/kernel*.txz .

root@localhost:/vagrant/opnsense/tools # make ports
...
>>> Setting up stage in /usr/local/stage
>>> Setting up world in /usr/local/stage
>>> Setting up /vagrant/opnsense/ports clone in /usr/local/stage
>>> Setting up /vagrant/opnsense/src clone in /usr/local/stage
>>> Setting up chroot in /usr/local/stage
ps: /dev/null: No such file or directory
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
>>> Setting up distfiles in /usr/local/stage
>>> Extracting packages in /usr/local/stage
>>> Removing packages in /usr/local/stage
>>> Installing packages in /usr/local/stage...
pkg: No packages installed.  Nothing to do!
pkg: No packages installed
>>> Building packages...
pkg: pkg is not installed
make[1]: chdir /vagrant/opnsense/ports/ports-mgmt/pkg: No such file or directory
>>> Creating binary packages...
The package management tool is not yet installed on your system.
Please set ASSUME_ALWAYS_YES=yes environment variable to be able to bootstrap in non-interactive (stdin not being a tty)
>>> The ports build did not finish properly :(
*** Error code 1

Stop.
make: stopped in /vagrant/opnsense/tools

root@localhost:/vagrant/opnsense/ports # git status -uno
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit (use -u to show untracked files)

root@localhost:/vagrant/opnsense/tools # pkg -v
1.5.6


[franco]

It looks like vagrant clobbers the environment by overriding shell variables. Look at:

make[1]: chdir /vagrant/opnsense/ports/ports-mgmt/pkg: No such file or directory

It tries to go to the newly set up ports tree, but this is not the right directory. The ports tree is probably under /usr/local/stage/usr/ports/ports-mgmt/pkg for pkg in this case. Not sure how to debug this. But I know it's just  minor quirk that can be fixed up in your tools.git for now. Although I would expect other issues later on as well.

Can you provide your output of the following:

# env

[iosense]

Hey Franco,

below you can see the system environments.

Code: [Select]

root@OPNsense:~ # env
SSH_CLIENT=10.0.1.1 50044 22
LOGNAME=root
MAIL=/var/mail/root
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
PWD=/root
TERM=xterm
SSH_TTY=/dev/pts/0
HOME=/root
USER=root
SSH_CONNECTION=10.0.1.1 50044 10.0.1.10 22
SHELL=/usr/local/etc/rc.initial
BLOCKSIZE=K
HOSTTYPE=FreeBSD
VENDOR=intel
OSTYPE=FreeBSD
MACHTYPE=i386
SHLVL=1
GROUP=wheel
HOST=OPNsense.localhost
REMOTEHOST=10.0.1.1
EDITOR=vi
PAGER=less

and also tools.git Makefile is edited as follows:

Code: [Select]

PORTSREFDIR?= /usr/freebsd-ports
PLUGINSDIR?=  /vagrant/opnsense/plugins
TOOLSDIR?=  /vagrant/opnsense/tools
PORTSDIR?=  /vagrant/opnsense/ports
COREDIR?= /root/core
SRCDIR?=  /vagrant/opnsense/src

and lastly, all make ports command output as follows:

Code: [Select]

root@OPNsense:/vagrant/opnsense/tools # make ports
.MAKE.LEVEL.ENV=MAKELEVEL
ARCH=i386
BLOCKSIZE=K
CDROM=/tmp/images/OPNsense-201509280956-OpenSSL-cdrom-i386.iso
CONFIGDIR=/vagrant/opnsense/tools/config/15.7
CONFIG_PKG=/usr/local/etc/pkg/repos/origin.conf
CONFIG_XML=/usr/local/etc/config.xml
COREDIR=/root/core
CPUS=1
EDITOR=vi
GROUP=wheel
HOME=/root
HOST=OPNsense.localhost
HOSTTYPE=FreeBSD
IMAGESDIR=/tmp/images
LABEL=OPNsense
LOGNAME=root
MACHTYPE=i386
MAIL=/var/mail/root
MAKEFLAGS= .MAKE.LEVEL.ENV=MAKELEVEL
MAKELEVEL=1
NANOIMG=/tmp/images/OPNsense-201509280956-OpenSSL-nano-i386.img
OLDPWD=/vagrant/opnsense/tools
OSTYPE=FreeBSD
PACKAGESDIR=/.pkg
PAGER=less
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
PLUGINSDIR=/vagrant/opnsense/plugins
PORTSDIR=/vagrant/opnsense/ports
PORTSREFDIR=/usr/freebsd-ports
PRODUCT_FLAVOUR=OpenSSL
PRODUCT_NAME=OPNsense
PRODUCT_RELEASE=OPNsense-201509280956-OpenSSL
PRODUCT_SETTINGS=15.7
PRODUCT_TYPE=opnsense-devel
PRODUCT_VERSION=201509280956
PWD=/vagrant/opnsense/tools/build
REMOTEHOST=10.0.1.1
SERIALIMG=/tmp/images/OPNsense-201509280956-OpenSSL-serial-i386.img
SETSDIR=/tmp/sets
SHELL=/usr/local/etc/rc.initial
SHLVL=1
SRCDIR=/vagrant/opnsense/src
SSH_CLIENT=10.0.1.1 50044 22
SSH_CONNECTION=10.0.1.1 50044 10.0.1.10 22
SSH_TTY=/dev/pts/0
STAGEDIR=/usr/local/stage
TARGETARCH=i386
TARGET_ARCH=i386
TERM=xterm
TOOLSDIR=/vagrant/opnsense/tools
USER=root
VENDOR=intel
VGAIMG=/tmp/images/OPNsense-201509280956-OpenSSL-vga-i386.img
>>> Setting up stage in /usr/local/stage
>>> Setting up world in /usr/local/stage
>>> Setting up /vagrant/opnsense/ports clone in /usr/local/stage
>>> Setting up /vagrant/opnsense/src clone in /usr/local/stage
>>> Setting up chroot in /usr/local/stage
ps: /dev/null: No such file or directory
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
>>> Setting up distfiles in /usr/local/stage
>>> Extracting packages in /usr/local/stage
>>> Removing packages in /usr/local/stage
>>> Installing packages in /usr/local/stage...
pkg: No packages installed.  Nothing to do!
pkg: No packages installed
>>> Building packages...
pkg: pkg is not installed
make[1]: chdir /vagrant/opnsense/ports/ports-mgmt/pkg: No such file or directory
>>> Creating binary packages...
The package management tool is not yet installed on your system.
Please set ASSUME_ALWAYS_YES=yes environment variable to be able to bootstrap in non-interactive (stdin not being a tty)
>>> The ports build did not finish properly :(
*** Error code 1

Stop.
make: stopped in /vagrant/opnsense/tools

Do I need to make changes to another file, as in tools.git Makefile ?  I would be very glad if you inform me about it.

Thank you.

[franco]

Still not sure what's happening, maybe you can override stage dir with:

STAGEDIR=/vagrant/opnsense/stage

(it's in common.sh)