Messages

Alright thank you guys  :)

And yes it's a static setup for additional ipv6 prefixes.

I'm using FRR to announce BGP to my upstream provider and I'm not sure whether I'm missing something or it is the way it is.

The thing is FRR is blocking any announcement defined at "Routing -> BGP -> General -> Network" if there is no route for it on the firewall, but there are only routes for smaller networks e.g. /56-64 on it so If a /40-48 is defined it simply is not announced.

Since upstream providers only accept anything >=/48 this breaks everything.

I can bypass it by setting a /48 null route via "System -> Routes" on OPNsense which makes FRR announce the /48 but it feels like a hack.

Little update on this, OPNsense is actually stateless in this regard (state is tracked via floating interface)

Issue was something else with my upstream provider...

I have to setup bgp peering in a colo which gives us 2 upstream connections e.g. ISP1 & ISP2

So I setup OPNsense with having WAN1 and WAN2 announcing the bgp routes via frr plugin.

If I announce only to one ISP e.g. via WAN1 everything works as expected.

As soon as I announce to both ISP I get asymmetric routing and therefore TCP connections fail.

Is it possible to fix this in OPNsense without putting a dedicated router in front of it ?

Basically I just want OPNsense to be stateless for WAN1 & WAN2.

What are my options ?

I'm currently in the progress of installing OPNsense under kubernetes using kubevirt (kvm)

I can boot the VM just fine and connect to it via serial.

The boot menu responds to input, the startup scripts e.g. configure interfaces all respond to input and works just fine.

The input works just until :

Code Select Expand

Welcome!  OPNsense is running in live mode from install media.  Please
login as 'root' to continue in live mode, or as 'installer' to start the
installation.  Use the default or previously-imported root password for
both accounts.  Remote login via SSH is also enabled.


Right after it's printed input stops working, I can not login as root or installer nothing happens if i type.

However output still works as is, so the serial console is just fine, doing a aacpi shutdown prints

Code Select Expand

>>> Invoking stop script 'beep'
..................
Syncing disks, vnodes remaining... 0 0 0 0 done
All buffers synced.
Uptime: 6m30s


I'm now going to install via VNC but I really want this VM to run headless if possible.

Any solution to this bug ?

edit: In the end I went with VNC

So I just followed https://docs.opnsense.org/manual/how-tos/wireguard-client-mullvad.html which broke my internet.

Wireguard will install a 0.0.0.0/1 route following the documentation which will override the default route.

It is important that under VPN -> Wireguard -> Local -> Configuration:  "Disable Routes" is checked.

The step2 in documentation is how to setup dynamic routing, so the disable routes is clearly missing in it, otherwise makes no sense.

Firewall -> Rules -> Floating: pass port 53 tcp/udp source any dest 192.168.1.1

Services -> Unbound -> General: Network Interfaces: only select lan (192.168.1.1)

Services -> Unbound -> Access Lists: Allow 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8

Services -> DHCPv4 -> Set each interface dns to 192.168.1.1

Repeat for ipv6


This way only 192.168.1.1 is returned for a dns request to the firewall.

I noticed unbound leaks all subnets configured in opnsense.

And this is a problem, because ...?

Never said it's a problem, I just dislike it. There is simply no reason to leak all networks.

I setup a floating rule, changed unbound to only listen on a single interface and changed the dns via dhcp.

I noticed unbound leaks all subnets configured in opnsense.

Just query the firewall host, can be easily found out with a ptr lookup.

Is there some way to prevent unbound from returning all the addresses ?

Code Select Expand


# check dns server
user@docker1:~# nslookup docker1
Server:         192.168.1.1 <- used dns server
Address:        192.168.1.1#53

Name:   docker1.example.com
Address: 192.168.1.11

# ptr on dns server
user@docker1:~# nslookup 192.168.1.1
1.1.168.192.in-addr.arpa        name = firewall1.example.com.

# get all subnets
user@docker1:~# nslookup firewall1.example.com
Server:         192.168.1.1
Address:        192.168.1.1#53

Name:   firewall1.example.com
Address: 192.168.1.1
Name:   firewall1.example.com
Address: 192.168.2.1
Name:   firewall1.example.com
Address: 192.168.3.1
... (removed entries)
Name:   firewall1.example.com
Address: 10.10.1.0
Name:   firewall1.example.com
Address: 10.20.2.0
Name:   firewall1.example.com
Address: 10.20.0.2
... (removed entries)
Name:   firewall1.example.com
Address: 2a02:****
Name:   firewall1.example.com
Address: 2a02:****
Name:   firewall1.example.com
Address: 2a02:****
... (removed entries)


Suricata works fine for me on 21.1.5 with proxmox and virtio.

I'm not using jumbo frames however.

@SFC opnsense is the upstream gateway for the cluster and ha requires load balancing.

@mimugmail traefik plugin sounds nice, I will set it up for ingress cli is more then fine, no eta yet though.

Has anyone a kubernetes setup with opnsense ?

There is barely content about it so I did some research and testing myself.


Opnsense does not come with load balancing by default but offers 3 plugins: os-relayd, os-nginx and os-haproxy

os-relayd was deprecated in the past, solely for load balancing.

os-nginx and os-haproxy are mainly for http reverse proxying, os-haproxy has some more load balancing options but lacks udp load balancing if required.

os-nginx lacks load balancing algorithms, round robin is the only one but that's a limitation of the opnsense gui.

Best option for future support seems to be os-haproxy.

Wow netgate wireguard implementation reads great.

There were random sleeps added to "fix" race conditions, validation
functions that just returned true, catastrophic cryptographic
vulnerabilities, whole parts of the protocol unimplemented, kernel
panics, security bypasses, overflows, random printf statements deep in
crypto code, the most spectacular buffer overflows, and the whole litany
of awful things that go wrong when people aren't careful when they write
C. Or, more simply, it seems typical of what happens when code ships
that wasn't meant to. It was essentially an incomplete half-baked
implementation – nothing close to something anybody would want on a
production machine.

64 prefix means you are limited to 1 subnet = wan, so you can not setup ipv6 for your lan/dmz.

"Interfaces -> Overview -> WAN -> IPv6 delegated prefix"

If you have a prefix <=63 you have to setup router advertisement for SLAAC.

Linux uses the kernel implementation, opnsense the go usermode implementation.

There is a huge performance impact. But kernel implementation for bsd is on it's way.

Edit: nevermind didn't read, but virtio support on bsd is lacking, I think that's the issue.