Messages

I think I can confirm that realtek is terrible at present under any BSD kernel.  Particularly from the beelink G35 series, as these do seem to be older hardware sold at discount prices (i.e. cheap).  But I have seen this behavior, under many different manufacturers many of the models are very new.   This also happens under PFsense (though I would never use their crap under production, I do test it), and generic FreeBSD.  However, when putting debian in charge of the hardware and then handing opnsense virtio networking everything seems to have stabilized.  I will note that it is best to make debian use static IP addresses.  If you want to avoid these complications overall I suggest finding hardware that does not have realtek networking hardware, intel hardware in particular seems to work better than anything else, I have heard from friends in the biz that they pay for all the patents of various things.  In particular, this is drastic amongst wifi hardware.  I have recently taken to replacing the wifi chips in any laptops that I can, and I have noticed dramatic improvements in not only general reception, but now I can connect in busy public wifi spots that I have never had decent reception before.  Here is an example of a modestly priced computer that runs OPNsense great and has intel hardware:   https://amzn.to/3PBh0wB

and here is an example of a replacement wifi chip for laptops:
https://amzn.to/44PPCze

TLDR realtek is terrible and should be avoided at all costs, otherwise put debian or some other linux on it and then give an opnsense VM kvm's virtio networking.

To complicate matters even more, for some odd reason I thought I might virtualize the OPNsense instance on the very hardware it was crashing on?!?  Why would I do this?  This does not sound like a good idea.....

Well for some reason this works great.

1. Install debian bookworm
2. bridge the network connections
3. install kvm etc, install OPNsense as a VM
4. give the OPNsense VM any bridge you want, ensuring debian has at least created the bridge.

I have even given such a VM access to my direct fiber connection, in this case you'll want to assign static IP addresses to all the debian interfaces. 

Strangely debian seems to handle the realtek interfaces better than OPNsense and I have not had any failures so far.

Any of you ever think to yourselves I am going to write this simple shell script to do one task, and then bazorkin skynet evolves

https://github.com/joshuacox/testnet

you know, its just a ping why would that be complicated....

I have a few OPNsense HA installs on very cheap hardware to test HA on my homelab  (some beelinks and other small cheap computers with at minimum two network ports).  At some point on all of them, some more often than others, they go null route or something peculiar.  If I log in to the machine directly with a keyboard and monitor the thing seems to be just fine but the networking is dead.  The one thing that seems to be common amongst them all is the realtek hardware. Plus I have not seen this on more expensive hardware, though I would be interested if anyone has seen these sort of issues on any other hardware.  To mitigate this I have made a small script that tries to ping out to 3 different ips and reboot the machine if all three fail three times in a row.

Code Select Expand

#!/usr/bin/env bash
ips='ip1 ip2 ip3'

test () {
  ping -t 5 -c 1 $1
  if [[ $? -eq 0 ]]; then
    echo good
    # any successful ping means networking is still working
    exit 0
  else
    ((++count))
  fi
}

count=0
while [[ $count -lt 3 ]]; do
  for i in $ips
  do
    test $i
  done
done

# only reboot if all pings fail
reboot


Ugly I know, I'd be happy to hear about more elegant solutions.  For now I paired this with an action:

Code Select Expand

# cat /usr/local/opnsense/service/conf/actions.d/actions_testnet.conf
[check]
command:/root/testnet.sh
parameters:
type:script
message:testing network
description:network test


And have that running as a cronjob every 5 minutes.

I see these in my audit log:
   /firewall_rules.php: Web GUI authentication error for 'user12' from 10.0.0.7

I see sshlockout for that address in my firewall live view, the user can no longer load the UI at all.

How do I remove the ban hammer from that address?

Let me add the metallb configs:

Code Select Expand

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: mainips
      protocol: layer2
      addresses:
      - 10.20.16.86-10.20.16.89
      auto-assign: false
    - name: cheaplyips
      protocol: layer2
      addresses:
      - 10.20.80.91-10.20.80.249
      auto-assign: true
    - name: prodweb
      protocol: layer2
      addresses:
      - 10.20.80.3-10.20.80.85
      auto-assign: false


or for BGP

Code Select Expand

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 10.20.16.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: mainips
      protocol: bgp
      addresses:
      - 10.20.16.86-10.20.16.89
      auto-assign: false
    - name: cheaplyips
      protocol: bgp
      addresses:
      - 10.20.80.91-10.20.80.249
      auto-assign: true
    - name: prodweb
      protocol: bgp
      addresses:
      - 10.20.80.80-10.20.80.85
      auto-assign: false

I can add that there is still a dhcp server serving 10.27.16.0/24 on the 111-240 block on that network (opnsense is the dhcp server).  So I tried moving the loadbalancer to 10.20.80.x, but those addresses have yet to respond at all.  The only address that works every now and then is the 88 address.

moving discussion here:
https://forum.opnsense.org/index.php?topic=15756.0

I would certainly like to document what is going wrong with metallb and opnsense.  Have you solved your problem?  Even if you had to use something other than opnsense, let me know here:
https://forum.opnsense.org/index.php?topic=15756.0

I am having issues again with metallb and opnsense.  Everything seems to be working on the kubernetes side,

Code Select Expand

k get svc -n ingress-nginx
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.110.93.202   10.20.16.88   80:30420/TCP,443:30472/TCP   12m


However, from any machine on the same 10.20.16.0/24 network I cannot reach that address:

Code Select Expand

sudo arping -I br0 10.20.16.88
ARPING 10.20.16.88
Timeout
Timeout
Timeout
Timeout
^C
--- 10.20.16.88 statistics ---
5 packets transmitted, 0 packets received, 100% unanswered (0 extra)

curl 10.20.16.88/echo -v -H Host:echo1.yourdomain.com
*   Trying 10.20.16.88...
* TCP_NODELAY set
* connect to 10.20.16.88 port 80 failed: No route to host
* Failed to connect to 10.20.16.88 port 80: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.20.16.88 port 80: No route to host



Of note, I had this exact same issue last year:
https://forum.opnsense.org/index.php?topic=12878.msg59918#msg59918

I got around this ar one point by switching back to layer2 and not using BGP.  However, even then only one address worked, not all addresses (otherwise I would've posted back success in that issue).

I'm willing to try both methods again, or anything else that might work.

Also, another similar issue I'd like to point out:
https://forum.opnsense.org/index.php?topic=14654.msg67006#msg67006

Any suggestion on what I might do to solve this issue?

I've also enabled BGPv4 and tried both enabling and disabling OSPF, but I still do not seem to be able to get static hosts to have the same sort off peerage as the DHCP hosts in my network

I need DHCP on this network, however, the range that DHCP uses does not conflict with the static IP I am using.

As an alternative I have added static mapping for many of the hosts which does seem to work. 

However, hosts that are set to static IP addresses within the OS are not routable.

Furthermore, when attempting the layer 2 metallb tutorial:

https://metallb.universe.tf/tutorial/layer2/

I can see in `k get svc` I do get a static IP address from the service:

```
k get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx   LoadBalancer   10.106.58.106   10.2.0.86   80:30025/TCP   94m
```

But, again, anything with a static IP address not set by DHCP cannot be routed to.

In my NAT port forwarding rules I have:

A host that gets its IP address from DHCP:

Code Select Expand


WAN TCP * * WAN address 2222 10.2.0.19 22 (SSH)


And another with a static IP address:

Code Select Expand


WAN TCP * * WAN address 2322 10.2.0.220 22 (SSH)


I can ssh through the WAN to the first host by port 2222, but not the second host by 2322

Curiously from the first host I can ssh to the second host (behind the firewall) by port 22, so I am fairly certain that the second host has correctly set it's static IP.  The only difference that I can discern is that the first host is DHCP and the second is static.