OPNsense Forum
English Forums => Development and Code Review => Topic started by: rvalle on September 22, 2019, 10:56:21 am
-
Hi!
I would like to adopt an OpenSource router/firewall that can be managed from Ansible, as we already manage everything in our network that way.
Ansible has made a lot of work on the Network device configuration front but still there seems to be no support for a product like OPNSense.
I have found 2 modules in gitlab which configure OPNSense with Ansible. The ansible mofule for HAProxy, developed by @mj84 which was announced in this forum and another module by @fpieters that I found in Github.
They both follow different strategies. one seems to use Ansible XML plugins to compose an XML file that is eventually installed in OPNSense. The other uses the REST api to modify objects in OPNSense, but unfortunately is limited to the HAProxy functionality.
I was wondering if there is a way to provide complete support to OPNSese configuration from Ansible in a generic, and easy to implement way. Perhaps using the previous 2 projects as reference.
I am not familiar with OPNSense development but I am familiar with Ansible development as I did some bindings and ansible module work for OpenNebula.
Looking at the intro in the OPNSense REST API there seems to be generic enough, with calls following the format:
https://opnsense.local/api/<module>/<controller>/<command>/[<param1>/[<param2>/...]]
Ansible has also introduced NETCONF which also seems to be generic client for network devices configuration, with SOAP envelopes but generic get-set statements inside.
The question I am wondering is then weather a generic bridge from NETCONF to OPNSense API could be (easily) developed and whether that would work well enough to develop a good base that covers most OPNSense configuration options.
Before trying to put together a small prototype, I was wondering a few things:
I was wondering if anybody else is working, or interested in working on this.
I am assuming that a NETCONF-RESTAPI bridge would be installed in OPNSense, perhaps as an optional module.
I am not sure with Language should be used for this, but looking at the Development Documentation I see some mentions to Python that should be easy and fit for the job.
I guess integration for the authentication system would also be required, but perhaps there is already python code capable of this.
I was also wondering if there are mockups for the RESTAPI that would facilitate the development.
Any commends or ideas regarding OPNSense and Ansible?
-
Writing short (mobile):
2 problems, some functions work via API (like HAProxy, Alias, QoS), legacy is xml only. API calls also generate xml so the generic approach is xml. But as more and more sections getting migrated to API Code base, the xml container will move. Means you have to checkpoint versions to define when is which container location used.
-
Hi there,
while developing my Ansible modules for OPNsense's HAProxy, I have implemented a few generic helper functions which in theory could manipulate every kind of object via API.
These are for example:
- low-level functions for HTTP GET and POST requests to the API
- CRUD operations (create, read, update, delete)
- checking the existence of an object
- find selected item in a list
- compare two lists
However, as of now the functions / objects available through the API of OPNsense are indeed somewhat limited.
If there is more functionality to come to the API, I would gladly extend my Ansible modules to manage other datatypes as well (e.g. network objects and rules), or assist others doing so.
What @mimugmail states is definitely something that needs to be taken care of.
An API should (must) guarantee some sort of stability, so that stuff does not get broken.
That might be implemented by introducing API versions, which could be bound to a specific major version of the product, and then the API is guaranteed to stay stable for that specific version.
Cheers,
Markus
-
Thanks for your comments @mj84 and @mimugmail
So far we have that:
- there is a legacy XML API and a new Rest API
- Rest API is far from complete. implementation seems to translate to XML anyway.
- I also noticed in @mj84 comments that all config objects are referenced by UUID, which is not very Ansible playbook friendly.
- There is generic code that can be used form @mj84 work.
In an ideal world we would go with the Rest API which would be 100% complete.
Having to deal with 2 modules interfaces is not the end of the world, provided that those 2 interfaces are generic enough. If the problem extends to the point in which each module has its own proprietary interface then viability is at stake. If OPNSense is pushing for REST API, then I guess we should follow suit, and deal with XML config as "Legacy" type of config.
I am not sure I understand the XML API. By looking at the @fpieters XML based Ansible module, he seems to compose an XML configuration file that it is then sent at once to the OPNSense router. I wonder if there is API to update that XML configuration perhaps by sending XML fragments or how would it work. Or are we thinking about touching the configuration file directly? How would legacy XML module configuraiton be implemented?
I did not get any comments on whether NETCONF seems like a good architecture way to go with. I assumed it is OK based on Ansible modules and Industry buzz around it only. I this option would involve:
- A NETCONF to REST API/XML Config proxy module would be provided (Python?)
- Ansible generic NETCONF client module can be used to start with
- An specialization writen on top of the generic NETCONF Ansible client Module can be written afterwards if that improves friendliness (I notice some vendors do but not sure why)
- The NETCONF proxy module could take care of issues such as Name to UUID translation, REST/Legacy conf etc.
There are other things that NETCONF imposes, such as the existence of multiple configuration files and been capable to go back to a previous one, etc. I haven't really checked that much, but does not seem like a lot of issues here. I am not sure either if OPNSense has this kind of functionality anyway.
-
I'm wondering who has so much free time to do this :)
-
:)
Depending on how generic the XML system and the Rest API are, it might be quite easy to get something up and running quickly.
The other ansible projects that I mention above involved also a significant effort, and now we can reuse some code.
With a good architecture anybody should be able to help making this project complete. I mean, getting something up and running that it is easy to extend/maintain should be the priority.
I am happy to contribute. I really want an opensource router that it is ansible manageable. And I hope someone else can too.
But I still have some questions with regards to the XML Config API, hope someone can provide some directions.
-
Just copy /conf/config.xml and do some changes, then just diff. For API code you need uuid which can be generated via uuidgen command. Thats it :)
When you create a GitHub project with some stuff I'm happy to help.
-
@mimugmail there is also naturalis opensense ansible role
It is also a XML file composition, config file upload. I don't now how did I miss it when looking at the available work.
I have also been reviewing the Ansible-NETCONF module that there is and the supported transfer is SSH only.
Other network modules use the REST api directly in line with the OPNSense strategy.
I am going to test the naturalis module. which seems pretty complete. Perhaps this is the way to go until the REST api is more complete.
-
OK, I forked the Naturalis repo here: https://github.com/privazio/ansible-opnsense/tree/dev
I manage add support for bridges interface quite quickly.
I think for the current status of OPNSense, for short term solution, this is definitely a good way to go. I am going to be using it for my current project and will add what I need.
I have also reviewed the Ansible documentation and the Networking Development documentation, thinking on something more long term.
Looks like using the HTTP REST API to implement a Connection plugin as documented:
https://docs.ansible.com/ansible/latest/network/dev_guide/developing_plugins_network.html#developing-plugins-network
... seems to be the way to go. Then the rest of the feature modules use this one for basic connectivity. It seems to be the recommended architecture or at least the only one documented.
With the work already made by @mj84 that should be much easier to achieve.
Since the REST API is fairly generic, it should be possible to create a base class so that the individual functionalities use so that they are easy to implement, i.e. each module could create Dictionary that can be exported to JSON, and the base module knows how to turn it into API calls.... etc.
I will try to put together a prototype. One of these weekends....
it would be good to know if HTTP Rest API is coming to the basic modules (interfaces, aliases, fw rules, etc) anytime soon/or if this is something easy to add.
-
It's already implemented for Alias, there's no plan for anythin in 20.1 yet. Quite busy migrating from 11.2 to 12.1 :)
-
ok, good to know alias are supported too.
meanwhile naturalis is merging my pull requests, so the legacy ansible way of doing is progressing.
-
Hi!
I have added support to this ansible role for: bridges, static dhcp maps, CAs, certificates, auth servers, tunnables, port forwarding and VPN. Made Aliases compatible with the new implementation.
This on top of the provided support of all the original components provided: interfaces, users, groups, vlans, routes, etc.
The role is far form perfect but a good start. It is easy to extend with support for new objects and there is support for most already.
-
Do they have a documentation with some real life examples?
-
Yes, naturalis already provided a whole project as documentation.
Also, it allows for creating a virtual networking environment with vagrant for testing.
As for my pull requests, I have added documentation on each of them, with samples.
I have also deployed the role in a lab firewall with most of this functionalities in production.
-
I am afraid that this may never materialize because these distributions were designed to be click-interface configurable from day one, with no API in mind. That is one of the reasons why I am still looking for a replacement router operating system, one that is still open-source but that is also designed to be configured with configuration-management tools (nope downloading entire config and reuploading it is not what I would call CM, at least not good-CM). Sadly nothing at the horizon.
This reminds me that is still impossible to deploy a cron user script job that persists reboots or updates. The only cron activities possible are plugin exposed commands, not possible to write your own.
-
Hello @rvalle and others. Good to see that our PoC and provision repo on Naturalis is being used.
@Naturalis we are using ansible to automate the first (xml rewrite) deployment of OPNsense. After this our module is far from complete to do config changes. So thanks @rvalle for the help on improving it. ;)
Unfortunately lacking time the automation effort is stalled, also waiting for a more complete REST api.
So far we deployed a new campus network and are starting to do the same for our datacenter. In this network (and most other components) we use ansible to automate all of it. Except for the firewalls. So if there could be some joint effort on using the REST api.. great!
-
Hi All, @fpieters
I have been runnig this automation for some time. and the result is very satisfying.
Using ansible to compile an XMl cofiguration is a workable strategy while the REST API matures.
OPNSense is mature in the wasy that the configuration file describes pretty well the desired configuration state.
After having run this for some time, I am planing on a next major version that overcomes some issues found, and will improve it a bit further.
https://github.com/naturalis/ansible-opnsense/issues/19 (https://github.com/naturalis/ansible-opnsense/issues/19)
I am also trying to implement Continiuous Integration on Gitlab so that I can test this roles against new updates in OPNSense. H
ere I need to use packer to generate appliances or something.
-
Greetings everyone,
I am reviving this thread as we are considering assessing OPNsense on our network. Since we are managing a lot with Ansible, I am coming here.
I was wondering if there has been some more observations concerning the naturalis role (in good or bad, I see it's still maintained).
I have also found https://github.com/opoplawski/ansible-pfsense/ whose seems pretty active. @rvalle, did you evaluate it (I had not found it into your messages here)?
Thanks much!
-
Greetings!
I just want to mention the Ansible Collection I am developing: https://github.com/ansibleguy/collection_opnsense
It utilizes the REST APIs provided by OPNSense - therefor it has some limitations, but it is applicable for many common use-cases.
- AnsibleGuy