Messages

1

Development and Code Review / Re: Ansible Support for OPNSense?

« on: September 22, 2019, 03:05:41 pm »

Hi there,

while developing my Ansible modules for OPNsense's HAProxy, I have implemented a few generic helper functions which in theory could manipulate every kind of object via API.

These are for example:

• low-level functions for HTTP GET and POST requests to the API
• CRUD operations (create, read, update, delete)
• checking the existence of an object
• find selected item in a list
• compare two lists

However, as of now the functions / objects available through the API of OPNsense are indeed somewhat limited.
If there is more functionality to come to the API, I would gladly extend my Ansible modules to manage other datatypes as well (e.g. network objects and rules), or assist others doing so.

What @mimugmail states is definitely something that needs to be taken care of.
An API should (must) guarantee some sort of stability, so that stuff does not get broken.
That might be implemented by introducing API versions, which could be bound to a specific major version of the product, and then the API is guaranteed to stay stable for that specific version.

Cheers,
Markus

2

German - Deutsch / Re: HAproxy Anfragende Ip durchreichen (Nextcloud)

« on: September 20, 2019, 03:11:06 pm »

Hi,

das Aktivieren des X-Forwarded-For Header im HTTP-Request ist nur die halbe Miete.
Damit das in den Webserver Logs vom Nextcloud auch so protokolliert wird, muss das Logformat entsprechend angepasst werden, damit die IP-Adresse aus dem X-Forwarded-For protokolliert wird.

Für Apache könnte das dann z.B. so aussehen:

Code: [Select]

LogFormat "%h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" Fwd-For:%{X-Forwarded-For}i" revproxy
CustomLog ${APACHE_LOG_DIR}/access.log revproxy

Viele Grüße,
Markus

3

German - Deutsch / Re: API-Aufruf {"result":"failed"}

« on: September 01, 2019, 03:47:22 pm »

Hi,

der Curl Aufruf sieht für mich schon in Ordnung aus, da hier ja nur "an" oder "aus" geschaltet werden soll, darf der Post-Body trotzdem leer bleiben.

Ich würde laut der Doku verstehen, dass man $enabled durch "true" oder "false" oder auch "0" oder "1" ersetzen muss, so ganz kann man das leider der Doku nicht entnehmen..

Grüße,
Markus

4

Development and Code Review / Re: Ansible role + modules for HAProxy management through API

« on: August 27, 2019, 06:50:17 pm »

Changelog:
2019-08-14: Initial post
2019-08-14: Servers can now be managed
2019-08-14: Lua scripts can now be managed
2019-08-15: Groups and Mapfiles can now be managed
2019-08-27: Frontends can now be managed
2019-09-01: Backends can now be managed
2019-09-03: CPUs and Healthchecks can now be managed, we are feature-complete
                   Also, i have completed the tasks file, so everything can be managed through Ansible variables
2019-09-04: Complete rewrite of ACL module

5

Development and Code Review / Ansible role + modules for HAProxy management through API

« on: August 27, 2019, 06:50:05 pm »

Hi folks,

as my original post got somewhat lost in the 19.7 section, and I figured that this topic rather belongs here, I decided to re-create a thread here.

I have started using OPNsense a few months ago and I'm quite impressed by almost every aspect of the product so far!
My OPNsense cluster is mainly serving as a Reverse Proxy / Loadbalancer using the HAProxy module, so I wanted to search for a way for easier configuration management.

As I am a heavy user of Ansible, I have started developing Ansible modules and a corresponding Ansible role definition for managing OPNsense's HAProxy configuration more easily.
My goal is for this project to be feature-complete, so every aspect of HAProxy on OPNsense shall be reflected in these Ansible modules.

This means that these OPNsense HAProxy object types can be managed:

• ACLs (Conditions)
• Actions (Rules)
• Backend Pools
• CPUs (CPU Affinity Rules)
• Errorfiles (Error Messages)
• Frontends (Public Services)
• Groups
• Healthchecks (Health Monitors)
• LUA services
• Maps (Map Files)
• Servers
• Users

I have created a GitHub repo for this project here: https://github.com/mj84/ansible-opnsense-haproxy

Any questions or input are very welcome

As you can see, most of the data types are already implemented, but
As of 2019-09-03 all datatypes are implemented and seem to be working.
There is still some refactoring to be done, so variable names etc. might change but variable names should be stable. Also, the documentation is still missing for the most parts.
As soon as I have implemented the remaining datatypes, I will provide a fully documented task file and examples for a playbook.

I will try to keep this post updated as I implement the remaining datatypes and update the documentation.

Cheers,
Markus

6

19.7 Legacy Series / Re: Ansible role + modules for HAProxy management through API

« on: August 16, 2019, 02:28:14 pm »

I just moved the changelog to the first reply for better overview.

Changelog:

• 2019-08-14: Initial post
• 2019-08-14: Servers can now be managed
• 2019-08-14: Lua scripts can now be managed
• 2019-08-15: Groups and Mapfiles can now be managed

Can a mod please move this topic in the "Development and Code Review" sub-forum?
Thanks!

7

19.7 Legacy Series / Re: Reverse proxy using HAProxy

« on: August 16, 2019, 12:36:08 pm »

I have the listen addresses configured as "IP address:Port", e.g. "192.168.1.254:80" since HAProxy does not do implicit name based virtual hosting as Apache or nginx would.

In order to achieve name based backend selection, you have to create the rules (and conditions) as mentioned and reference the appropriate rule in the public services.

So the workflow is this:

• HTTP request arrives on your public service at 192.168.1.254:80
• HAProxy evaluates all rules assigned to the public service
• If you have a rule that says "if hostname == host1.domain, then use backend pool 1", backend pool 1 will be used
• If you have a rule that says "if hostname == host2.domain, then use backend pool 2", backend pool 2 will be used

To clarify: You only need multiple public services if you want to listen on different IPs (although you also could have one public service listen on different IPs) or have two completely separate frontends.
If you want to have multiple hostnames served behind the same IP address, you would assign multiple rules to your public service as mentioned in my example.

8

19.7 Legacy Series / Re: Reverse proxy using HAProxy

« on: August 16, 2019, 09:49:40 am »

Does this scheme roughly match your setup?

:80 public service1 ----> backend pool1 ----> server1:80
:80 public service2 ----> backend pool2 ----> server2:80

Within a public service, you can define a default backend pool. If your backend pool only contains one server, that is how you can define a default server.

What I had to learn the hard way, is that you have to assign rules to either a public service or a backend pool.
Otherwise they won't get evaluated.

So you need to have these rules:

1.
condition: hostname1.domain
action: use backend pool1

2.
condition: hostname2.domain
action: use backend pool2

Rule 1 needs to be attached to public service1, Rule 2 to public service2.

HTH

9

19.7 Legacy Series / Re: HAProxy - Why won't my rules work...

« on: August 14, 2019, 11:23:33 pm »

I see, so you are basically using HAProxy in TCP mode and the SSL sessions terminate on your backend servers.
From your first question I was assuming that you are using HTTP mode.

I agree, SSL pass-through can be a bit tricky, that's why I try to avoid it where possible

I doubt you are missing some option in the GUI, I guess there is just no automatism in OPNsense which tells you "Hey, you are doing SNI detection on a TCP session, please be advised to also configure X, Y, and Z..."

You don't need to set these options via passthrough though, you can configure 'tcp-request inspect-delay' and  'tcp-request content accept' via rules but you will have to attach them to your backend or frontend, which should make no difference in the end.

Regards,
Markus

10

19.7 Legacy Series / Re: Feature request

« on: August 14, 2019, 08:34:29 pm »

I think you can build something similar already:

• create an alias which will contain your hosts (Type = Hosts)
• block internet access for that alias
• add/remove hosts from that alias as needed

Regards,
Markus

11

19.7 Legacy Series / Re: HAProxy - Why won't my rules work...

« on: August 14, 2019, 09:21:03 am »

To me that sounds that your server is not reachable for a short period of time..
Did you check your server stats on HAProxy Statistics -> Status?

12

19.7 Legacy Series / Ansible role + modules for HAProxy management through API

« on: August 14, 2019, 12:06:28 am »

Hi folks,

I have started using OPNsense a few months ago and I'm quite impressed by almost every aspect of the product so far!
My OPNsense cluster is mainly serving as a Reverse Proxy / Loadbalancer using the HAProxy module, so I wanted to search for a way for easier configuration management.

As I am a heavy user of Ansible, I have started developing Ansible modules and a corresponding Ansible role definition for managing OPNsense's HAProxy configuration more easily.
My goal is for this project to be feature-complete, so every aspect of HAProxy on OPNsense shall be reflected in these Ansible modules.

For now, the following OPNsense HAProxy object types can be managed:

• ACLs (Conditions)
• Actions (Rules)
• Errorfiles (Error Messages)
• Groups
• LUA services
• Maps (Map Files)
• Servers
• Users

These datatypes are still to come:

• Backend Pools
• CPUs (CPU Affinity Rules)
• Frontends (Public Services)
• Healthchecks (Health Monitors)

I have created a GitHub repo for this project here: https://github.com/mj84/ansible-opnsense-haproxy

Any questions or input are very welcome

Please note that this is still under heavy development, and there is quite a bit of refactoring to be done, so variable names etc. will change for sure.

I will try to keep this post updated as I implement the remaining datatypes and update the documentation.

Cheers,
Markus

13

19.7 Legacy Series / Re: HAProxy - Why won't my rules work...

« on: August 13, 2019, 12:04:06 pm »

Have you assigned your configured rule to the public service?
If not, it won't get evaluated..

Cheers,
Markus