Metallb and Kubernetes

[thoth]

I am having issues again with metallb and opnsense.  Everything seems to be working on the kubernetes side,

Code: [Select]

k get svc -n ingress-nginx
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.110.93.202   10.20.16.88   80:30420/TCP,443:30472/TCP   12m

However, from any machine on the same 10.20.16.0/24 network I cannot reach that address:

Code: [Select]

sudo arping -I br0 10.20.16.88
ARPING 10.20.16.88
Timeout
Timeout
Timeout
Timeout
^C
--- 10.20.16.88 statistics ---
5 packets transmitted, 0 packets received, 100% unanswered (0 extra)

curl 10.20.16.88/echo -v -H Host:echo1.yourdomain.com
*   Trying 10.20.16.88...
* TCP_NODELAY set
* connect to 10.20.16.88 port 80 failed: No route to host
* Failed to connect to 10.20.16.88 port 80: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.20.16.88 port 80: No route to host


Of note, I had this exact same issue last year:
https://forum.opnsense.org/index.php?topic=12878.msg59918#msg59918

I got around this ar one point by switching back to layer2 and not using BGP.  However, even then only one address worked, not all addresses (otherwise I would've posted back success in that issue).

I'm willing to try both methods again, or anything else that might work.

Also, another similar issue I'd like to point out:
https://forum.opnsense.org/index.php?topic=14654.msg67006#msg67006

Any suggestion on what I might do to solve this issue?

[thoth]

Let me add the metallb configs:

Code: [Select]

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: mainips
      protocol: layer2
      addresses:
      - 10.20.16.86-10.20.16.89
      auto-assign: false
    - name: cheaplyips
      protocol: layer2
      addresses:
      - 10.20.80.91-10.20.80.249
      auto-assign: true
    - name: prodweb
      protocol: layer2
      addresses:
      - 10.20.80.3-10.20.80.85
      auto-assign: false

or for BGP

Code: [Select]

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 10.20.16.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: mainips
      protocol: bgp
      addresses:
      - 10.20.16.86-10.20.16.89
      auto-assign: false
    - name: cheaplyips
      protocol: bgp
      addresses:
      - 10.20.80.91-10.20.80.249
      auto-assign: true
    - name: prodweb
      protocol: bgp
      addresses:
      - 10.20.80.80-10.20.80.85
      auto-assign: false
I can add that there is still a dhcp server serving 10.27.16.0/24 on the 111-240 block on that network (opnsense is the dhcp server).  So I tried moving the loadbalancer to 10.20.80.x, but those addresses have yet to respond at all.  The only address that works every now and then is the 88 address.

[kya]

Hi all,

Here's how I got Metallb working with OpnSense. I'm running a home lab with Kubernetes 1.19 on a single Ubuntu node based on Intel NUC hardware. OpnSense 20.1 is running on fitlet2. I have separate VLAN for Metallb but I think the process is similar if you want to use LAN ip range.

The Metallb side of things is pretty much "standard" and similar to what thoth is using so I won't dive into that in much detail. On OpnSense I'm running the frr-plugin to enable BGP.

Steps to enable BGP based routing between Metallb and OpnSense:

0. Have Metallb running with desired configuration
1. Install ffr-plugin on OpnSense
2. Enable both Routing and BGPv4 from OpnSense admin gui
3. Configure BGPv4 as follows (edit AS number as needed):



4. Add Neighbor to BGPv4 as follows (edit host ip and AS number as needed):



After setup the Routing -> Diagnostics -> BGPv4 page should be populated with the routed ip:s from Metallb which is a good sign that everything is working as expected.

I also tried Metallb with Layer2 but it wasn't reliable enough for me. The BGP approach works much better.

[ejprice]

Hi @kya

I'm just curious, which CNI are you using with Metallb?

(I was just looking to set this up - thanks for your post!)

[kya]

I'm using Calico for CNI.

[ejprice]

I'm using Calico for CNI.

Awesome. That’s what I was planning on using. Thanks again for the info!