Kubernetes best load balancer setup ?

[Voodoo]

Has anyone a kubernetes setup with opnsense ?

There is barely content about it so I did some research and testing myself.


Opnsense does not come with load balancing by default but offers 3 plugins: os-relayd, os-nginx and os-haproxy

os-relayd was deprecated in the past, solely for load balancing.

os-nginx and os-haproxy are mainly for http reverse proxying, os-haproxy has some more load balancing options but lacks udp load balancing if required.

os-nginx lacks load balancing algorithms, round robin is the only one but that's a limitation of the opnsense gui.

Best option for future support seems to be os-haproxy.

[SFC]

What exactly would you be using the proxy for?  OPNsense would need to live outside of your k8s cluster which is why you likely won't find anyone doing it.  It can't be spun up as a pod, k8s is linux/windows only, there is no bsd support.

[wallaby501]

I actually somehow got this going after a lot of trial and error (still not sure it's correct.) I will update this post when I'm back at my computer with details on what to configure and how.

[wallaby501]

So it was a bit different coming from pfsense but this is working so far. I'm sure there are some tweaks needing to be made but it does load balance properly.

1) Make a VIP as the load balancer under Interfaces-Virtual IPs (IP alias)
2) In Services-haproxy-Real Servers set up your real servers as the actual control plane nodes- IP, port, and do not check SSL
3) Go to Rules and Checks and make a healthcheck- http, GET, healthz, HTTP1.1 version, and the FQDN of your VIP host (load balancer FQDN). Do click "Force SSL" as you need that for the check.
3) Click on virtual services and make your backend pool comprising those nodes from above. TCP, Source-IP hash seems fine, your servers, enable health checking and select your check you just made, and set stick table persistence to source-ip.
4) Click virtual services dropdown arrow and then Public Service. This is the frontend. Select your listen addresses (VIP FQDN and IP), TCP, select the backend pool you just made. I didn't do anything else to this.
5) Settings-Service. Save and test the syntax and check the boxes to enable haproxy and then hit apply.

Should be good to go then.

[mimugmail]

Not sure if this is relevant here, I added traefik and caddy2 pkg's to my community repo.
If you can work with this I can also try adding a plugin.
https://www.routerperformance.net/opnsense-repo/

[Voodoo]

@SFC opnsense is the upstream gateway for the cluster and ha requires load balancing.

@mimugmail traefik plugin sounds nice, I will set it up for ingress cli is more then fine, no eta yet though.