I am having issues again with metallb and opnsense. Everything seems to be working on the kubernetes side,
k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.110.93.202 10.20.16.88 80:30420/TCP,443:30472/TCP 12m
However, from any machine on the same 10.20.16.0/24 network I cannot reach that address:
sudo arping -I br0 10.20.16.88
ARPING 10.20.16.88
Timeout
Timeout
Timeout
Timeout
^C
--- 10.20.16.88 statistics ---
5 packets transmitted, 0 packets received, 100% unanswered (0 extra)
curl 10.20.16.88/echo -v -H Host:echo1.yourdomain.com
* Trying 10.20.16.88...
* TCP_NODELAY set
* connect to 10.20.16.88 port 80 failed: No route to host
* Failed to connect to 10.20.16.88 port 80: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.20.16.88 port 80: No route to host
Of note, I had this exact same issue last year:
https://forum.opnsense.org/index.php?topic=12878.msg59918#msg59918
I got around this ar one point by switching back to layer2 and not using BGP. However, even then only one address worked, not all addresses (otherwise I would've posted back success in that issue).
I'm willing to try both methods again, or anything else that might work.
Also, another similar issue I'd like to point out:
https://forum.opnsense.org/index.php?topic=14654.msg67006#msg67006
Any suggestion on what I might do to solve this issue?
Let me add the metallb configs:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: mainips
protocol: layer2
addresses:
- 10.20.16.86-10.20.16.89
auto-assign: false
- name: cheaplyips
protocol: layer2
addresses:
- 10.20.80.91-10.20.80.249
auto-assign: true
- name: prodweb
protocol: layer2
addresses:
- 10.20.80.3-10.20.80.85
auto-assign: false
or for BGP
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.20.16.1
peer-asn: 64501
my-asn: 64500
address-pools:
- name: mainips
protocol: bgp
addresses:
- 10.20.16.86-10.20.16.89
auto-assign: false
- name: cheaplyips
protocol: bgp
addresses:
- 10.20.80.91-10.20.80.249
auto-assign: true
- name: prodweb
protocol: bgp
addresses:
- 10.20.80.80-10.20.80.85
auto-assign: false
I can add that there is still a dhcp server serving 10.27.16.0/24 on the 111-240 block on that network (opnsense is the dhcp server). So I tried moving the loadbalancer to 10.20.80.x, but those addresses have yet to respond at all. The only address that works every now and then is the 88 address.
Hi all,
Here's how I got Metallb working with OpnSense. I'm running a home lab with Kubernetes 1.19 on a single Ubuntu node based on Intel NUC hardware. OpnSense 20.1 is running on fitlet2. I have separate VLAN for Metallb but I think the process is similar if you want to use LAN ip range.
The Metallb side of things is pretty much "standard" and similar to what thoth is using so I won't dive into that in much detail. On OpnSense I'm running the frr-plugin to enable BGP.
Steps to enable BGP based routing between Metallb and OpnSense:
0. Have Metallb running with desired configuration
1. Install ffr-plugin on OpnSense
2. Enable both Routing and BGPv4 from OpnSense admin gui
3. Configure BGPv4 as follows (edit AS number as needed):
(https://i.imgur.com/Ul7pJ5wl.png)
4. Add Neighbor to BGPv4 as follows (edit host ip and AS number as needed):
(https://i.imgur.com/h28NOq2l.png)
After setup the Routing -> Diagnostics -> BGPv4 page should be populated with the routed ip:s from Metallb which is a good sign that everything is working as expected.
I also tried Metallb with Layer2 but it wasn't reliable enough for me. The BGP approach works much better.
Hi @kya
I'm just curious, which CNI are you using with Metallb?
(I was just looking to set this up - thanks for your post!)
I'm using Calico for CNI.
Quote from: kya on November 29, 2020, 12:03:44 PM
I'm using Calico for CNI.
Awesome. That's what I was planning on using. Thanks again for the info!