OPNsense Forum
English Forums => Development and Code Review => Topic started by: utkonos on April 22, 2022, 04:05:36 pm
-
I've written a kludgy Jupyter notebook that takes a template config.xml and performs a set of replacements and additions based on the contents of an INI file. I'm using it for deployment of a project, but I figured that I would share it here if anyone needs something like this, at least it may be a place to start. There are a number of optimizations that are just for my project, so you may need to bend it to your will to make it useful.
https://gist.github.com/utkonos/57c79f1a0b68dd6a79cbf2de68db995a
-
Thanks for sharing this!
I've been looking for something sorta similar myself:
https://forum.opnsense.org/index.php?topic=25901
but maybe I'm just crazy. :)
-
If you find this useful, that's great.
I have made a major update to the notebook. It was initially using a very kludgey way of modifying the config XML. Just jamming new XML in Python text formatting. This is all now totally replaced by code that operates natively on XML using Python's xml.etree.ElementTree library. There are only two tiny cosmetic differences between the output of this script and the output of OPNsense's own config manupulators: Python adds an extra space in tags like this:
<sometag/>
So, they look like this:
<sometag />
This is not configurable that I can see in Python. Another person in a Github issue has identified which XML libraries add this space and which ones don't:
https://github.com/zeux/pugixml/issues/87#issuecomment-188621862
OPNsense can read config XML with these spaces, so it's irrelevant.
The only other difference is that the XML declaration at the very start has an encoding, but the one generated in OPNsense does not. Again, this has no effect on OPNsense's ability to import the config file.
Here is what it looks like from the script:
<?xml version='1.0' encoding='us-ascii'?>
Here is the native OPNsense:
<?xml version="1.0"?>
I may try to figure out how to get this output correct.
-
off topic, as it is not a jupyter Notebook.
But the ansible playbook from https://github.com/Rosa-Luxemburgstiftung-Berlin/ansible-opnsense uses a (fetched) config.xml and ensures some settings based on configurations made in yaml files before provisioning them back to the device ...
So if you skip the config.xml fetch and push step, you have the same effect ... validating could be surely be implemented if someone is willing to contribute this ...
-
Thanks, but that Ansible playbook doesn't scratch my itch. I am deploying a new OPNsense instance rather than configuring an already existing instance.
I have updated the notebook again today and released another overhaul. This new version now has a WireGuard bootstrap. If the INI file contains a section "WGB", then it will add all the config sections needed for a WireGuard server to exist in the OPNsense instance from the very start via OPNsense importer. The INI has a few options. If a server private key is missing, the notebook generates a new keypair and inserts them into config XML. If the client private key is missing, it generates the whole WireGuard configuration along with keypair and writes that for you. The end result is a config.xml that can be used with OPNsense importer along with a WireGuard client configuration file. All with fresh keypairs if none were provided in the INI. There are a couple nuances where I deviated from how OPNsense would have created this WireGuard server. First, the interface is in "OPT0" rather than the last OPT number after others are created. This way it lands in the config XML in the order I want without knowing in advance how many OPTs there will be. Second, the WireGuard server is instance 1 and interface is wg1. The reason for this is that the bootstrap should be replaced by one that is generated in the OPNsense instance and the bootstrap one then deleted. By numbering as I have, the permanent WireGuard server and interface will be wg0 and instance 0.
This notebook is feature complete for what I need, so the next task is to look at unattended install in OPNsense. The goal is to have a fully automated and WireGuard bootstrapped install process. As it stands now, there are still three steps that require human interaction: starting OPNsense importer and selecting drive with config; running the installer; and finally running the first updates from console.
NOTE: I know that all three keys in the example INI are malformed. This is on purpose so someone doesn't go using the example keys in their deployment even inadvertently.
-
I did some cleanup and refactoring of code to use tag names rather than positions in the XML. It can also handle hostname and domain now. Here is a screenshot of an initial INI with the WireGuard bootstrap keypairs missing so that it generates them dynamically.
(https://i.imgur.com/zecJxpk.png)
This results in a WireGuard config that can be imported immediately. Here is one:
(https://i.imgur.com/C2uRIN3.png)
And all that matches up with the WireGuard bootstrap in the OPNsense config.xml:
(https://i.imgur.com/esU3yLN.png)
-
I have converted the notebook to a formal Python project and modules. I have a few unit tests to complete before I release it, but I researched how WireGuard keys are properly generated. I had used subprocess and called the WireGuard command line program in the same way that OPNsense does. However, I have now learned that the keys are from Daniel J. Bernstein's NaCl.
https://nacl.cr.yp.to/
So, I have gone ahead and replaced the subprocess code with nacl's PrivateKey() functions in the notebook.
In case you want to use it elsewhere, here is the boiled down code to just the stuff you may need.
$ pip install pynacl
import base64
import nacl.public
def wgkeys():
"""Generate a WireGuard keypair."""
private = nacl.public.PrivateKey.generate()
privkey = base64.b64encode(bytes(private)).decode()
pubkey = base64.b64encode(bytes(private.public_key)).decode()
return privkey, pubkey
-
Sounds interesting ...
btw. the ansible playbook I linked surely can be used to create a initial config:
https://github.com/Rosa-Luxemburgstiftung-Berlin/ansible-opnsense-playbook/blob/main/firewalls.yml#L24:L31
and we use it right this way: initial deploy and continuous maintenance based on the same tool and configuration ... but that is another story, don't mind ...
Is the wireguard cfg you generate working if you apply it on a virgin install?
In the env where we use the generated cfg from scratch we use IPSec VPNs, and these work out of the box: (generate cfg, install opnsense, copy generated config.xml in place, reboot, works)
Just as a test I tried once a device with additional wireguard config from scratch, but no wg instance was started ... as it is not our prod env and not the main problem at the moment, I did not dive deeper into it. (was at least more then a year ago, maybe the updates since then changed behavior ...)
Just interested if your approach creates a working wg config or the approach of just injecting a valid wg config in xml format is not sufficient ...
So some notes on the deploy experience would be appreciated ...
-
I just finished writing unit tests for the package version except for the cli.py file. That's my next task. Once I have 100% unit test coverage for every line of code in the whole package, I will push to Github and you can take a look. I have added the ability to write the config.xml to the correct location in an ISO using "pycdlib". There is a new feature in OPNsense that allows the importer to find CDs, and this fits perfectly.
I think I understand what your difficulty is with regards to no WireGuard instance starting as opposed to IPsec working fine. IPsec is part of OPNsense's base install. There is no plugin required for it to work. Therefore you can just add what you think you need to config.xml and you're good to go. For WireGuard, because it is a plugin, there needs to be a triggering event for the packages for the plugin to be installed. Just having the plugin listed in config.xml is not enough on its own. After the packages corresponfing to the plugin are installed, the WireGuard configuration in the config.xml is picked up and works as one would expect (except for an installer bug that removes the WireGuard Group interface so you have to do without that feature no matter what).
The triggering event that I use is a version upgrade. I start the install and then go to the OPNsense importer. I select the disk with the generated config.xml on it, and then the rest of the first boot occurs. I then run the installer. After the installer is done and the instance has rebooted, I login as root and run the update from console feature. During that update process, the wireguard plugin is detected in the config.xml and rather than updating the plugin packages, they are installed for the first time. After the update process is complete and it has rebooted again, you have a fully working WireGuard bootstrapped OPNsense instance.
-
... I will push to Github and you can take a look.
Great! Always interested and curious
There is a new feature in OPNsense that allows the importer to find CDs, and this fits perfectly.
Yes, I have followed the thread and seen the new feature with interest too ... The ideal for me would be a similar process like we do for linux since ages: build a customized preseeded cd image that will bring up the device with a working basic configuration and pass it over to the config management tool in use for finishing ...
For WireGuard, because it is a plugin, there needs to be a triggering event for the packages for the plugin to be installed.
Thanks for the hint, but I am aware of this ...
Using ansible we install packages and plugins too (https://github.com/Rosa-Luxemburgstiftung-Berlin/ansible-opnsense-plugpack), but maybe something went wrong in the order ...
The trick with the update process sounds quite interesting ... will notice this, thanks.
Thanks for the feedback and looking forward for your github repo..
-
Here is the new package version:
https://github.com/malwarology/opnsense-confgen
I was actually able to ring the 100% unit test coverage bell :)
% coverage report -m
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------
src/oscg/__init__.py 0 0 0 0 100%
src/oscg/cli.py 105 0 36 0 100%
src/oscg/core.py 171 0 32 0 100%
src/oscg/example.py 11 0 0 0 100%
src/oscg/utils.py 14 0 2 0 100%
------------------------------------------------------------------
TOTAL 301 0 70 0 100%
I have built a fresh OPNsense install ISO of 22.1.7 via the instructions here: https://github.com/opnsense/tools
The ISO format output for the config.xml works as expected with the same process that I used with the mounted drive and the official OPNsense image. The issue with the WireGuard interface group being deleted during the install is still there, so I noted that.
I will kick it out to PyPI tomorrow so the pip install in the README actually works, but it is late and I'm tired.
Let me know what you think.
-
It's on PyPI now.
pip install opnsense-confgen