OPNsense Forum
Archive => 20.1 Legacy Series => Topic started by: thoth on February 06, 2020, 03:09:21 am
-
I am having issues again with metallb and opnsense. Everything seems to be working on the kubernetes side,
k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.110.93.202 10.20.16.88 80:30420/TCP,443:30472/TCP 12m
However, from any machine on the same 10.20.16.0/24 network I cannot reach that address:
sudo arping -I br0 10.20.16.88
ARPING 10.20.16.88
Timeout
Timeout
Timeout
Timeout
^C
--- 10.20.16.88 statistics ---
5 packets transmitted, 0 packets received, 100% unanswered (0 extra)
curl 10.20.16.88/echo -v -H Host:echo1.yourdomain.com
* Trying 10.20.16.88...
* TCP_NODELAY set
* connect to 10.20.16.88 port 80 failed: No route to host
* Failed to connect to 10.20.16.88 port 80: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.20.16.88 port 80: No route to host
Of note, I had this exact same issue last year:
https://forum.opnsense.org/index.php?topic=12878.msg59918#msg59918
I got around this ar one point by switching back to layer2 and not using BGP. However, even then only one address worked, not all addresses (otherwise I would've posted back success in that issue).
I'm willing to try both methods again, or anything else that might work.
Also, another similar issue I'd like to point out:
https://forum.opnsense.org/index.php?topic=14654.msg67006#msg67006
Any suggestion on what I might do to solve this issue?
-
Let me add the metallb configs:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: mainips
protocol: layer2
addresses:
- 10.20.16.86-10.20.16.89
auto-assign: false
- name: cheaplyips
protocol: layer2
addresses:
- 10.20.80.91-10.20.80.249
auto-assign: true
- name: prodweb
protocol: layer2
addresses:
- 10.20.80.3-10.20.80.85
auto-assign: false
or for BGP
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.20.16.1
peer-asn: 64501
my-asn: 64500
address-pools:
- name: mainips
protocol: bgp
addresses:
- 10.20.16.86-10.20.16.89
auto-assign: false
- name: cheaplyips
protocol: bgp
addresses:
- 10.20.80.91-10.20.80.249
auto-assign: true
- name: prodweb
protocol: bgp
addresses:
- 10.20.80.80-10.20.80.85
auto-assign: false
I can add that there is still a dhcp server serving 10.27.16.0/24 on the 111-240 block on that network (opnsense is the dhcp server). So I tried moving the loadbalancer to 10.20.80.x, but those addresses have yet to respond at all. The only address that works every now and then is the 88 address.
-
Hi all,
Here's how I got Metallb working with OpnSense. I'm running a home lab with Kubernetes 1.19 on a single Ubuntu node based on Intel NUC hardware. OpnSense 20.1 is running on fitlet2. I have separate VLAN for Metallb but I think the process is similar if you want to use LAN ip range.
The Metallb side of things is pretty much "standard" and similar to what thoth is using so I won't dive into that in much detail. On OpnSense I'm running the frr-plugin to enable BGP.
Steps to enable BGP based routing between Metallb and OpnSense:
0. Have Metallb running with desired configuration
1. Install ffr-plugin on OpnSense
2. Enable both Routing and BGPv4 from OpnSense admin gui
3. Configure BGPv4 as follows (edit AS number as needed):
(https://i.imgur.com/Ul7pJ5wl.png)
4. Add Neighbor to BGPv4 as follows (edit host ip and AS number as needed):
(https://i.imgur.com/h28NOq2l.png)
After setup the Routing -> Diagnostics -> BGPv4 page should be populated with the routed ip:s from Metallb which is a good sign that everything is working as expected.
I also tried Metallb with Layer2 but it wasn't reliable enough for me. The BGP approach works much better.
-
Hi @kya
I'm just curious, which CNI are you using with Metallb?
(I was just looking to set this up - thanks for your post!)
-
I'm using Calico for CNI.
-
I'm using Calico for CNI.
Awesome. That’s what I was planning on using. Thanks again for the info!