Topics

Hi,

for a number of reasons that are not important here, I want to run some VMs in Bhyve on my OPNsense box (homelab).

Since setting this up needs to happen outside OPNsense default features, building the libraries from the ports tree etc, my question here becomes: what happens when an OPNsense update is installed? Is there a chance that the Bhyve setup will break somehow, and if so, how likely is that to happen and are there ways to minimize those chances or perhaps avoid that scenario entirely (other than just not doing this kind of thing on an OPNsense box obviously)?

I suppose this question could be generalized to any scenario where one needs/wants to run additional packages that need to be manually built and installed from the ports tree.

PS: I'm aware of the controversy around adding a virtualization layer to a firewall OS, adding tons of libraries, code, and as a consequence potential attack vectors etc
To anyone feeling the need to raise these concerns, I say: duly noted, I appreciate it and I do understand, but this is not the point of this topic  :)

[EDIT:]

EDIT: updated topic title as this turns out to have nothing to do with Wireguard even.

For context, this is kind of a follow-up for this topic.

TLDR: Wireguard just does not run well enough yet on OPNSense.
Between the issue I've described in the topic linked above and the kernel panics it seems to introduce as described here, I've decided to look for another solution:
I still want to use Wireguard, but I don't want it to mess with OPNSense functionality and break all my networking whenever it decides to act up.
-> abstract Wireguard stuff from OPNSense.

I've set up a VM running Alpine Linux to serve as a Wireguard gateway.
Nothing fancy, just

• a plain Alpine Linux install,
• eth0 is connected to the LAN network,
• using wg-quick up wg0 to bring up the interface,
• using these rules to enable forwarding (I haven't bothered with killswitches yet)

Code Select Expand

sysctl -w net.ipv4.ip_forward=1
sysctl -p
iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o wg0 -j ACCEPT
iptables -A FORWARD -i wg0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT


In OPNSense I then add a gateway on the LAN interface for the IP address that was assigned to the VM, et voilà.

This seemed to work fine, however when testing throughput I've come across a new issue.
Download speed is perfect: I'm getting close to full line-speed (300Mbps).
Upload speed is almost non-existent: I'm getting 0.1Mbps (line-speed is 30Mbps).
I'm just using a Windows 10 machine to run speedtest.net to do this.

So here's the interesting part: if I set the default gateway of that machine to the IP of the VM, as opposed to OPNSense's IP (i.e.: traffic is going straight to/from the VM as opposed to being routed to/from it by OPNSense);
I DO get the near full line-speed for uploads as well.
I also get near full line-speed for uploads when:

• I just go out WAN;
• I use the OpenVPN client running on OPNSense.

So it's not like OPNSense can't deliver.

To summarize:

• Upload is crap when: w10 machine -> OPNSense -> VPN gateway -> internet (via OPNSense)
• Upload is all good when: w10 machine -> VPN gateway -> internet (via OPNSense)

So I'm posting this here seeing as the only variable that seems to make the difference is OPNSense' presence in that part of the path.
Any ideas on how OPNSense seems to be slicing my upload performance here?

So I just discovered this gem.
A little context first:

• I have Wireguard/Mullvad set up with a gateway, as described here
• I have an alias defined for all hosts that must only access the internet through a VPN connection
• I have a VPN gateway group set up for load balancing/failover
• The firewall is configured to send all VPN destined traffic through the VPN gateway group
• The firewall is also configured to reject all VPN destined traffic that somehow makes it past that routing rule
• This screenshot shows the firewall config, these are the last 3 rules in the config for the LAN interface
  

This works perfectly when PIA (OpenVPN) is used in the VPN gateway group.
As you'd expect, when the client is disabled, the last but one rule is triggered and prevents VPN destined traffic from going out the WAN gateway group.
OpenVPN gateway is down:

Traffic is blocked:


Here comes the kicker though: replace the PIA gateway in the gateway group with the Mullvad gateway, and traffic merrily flows out from the WAN interface when the Wireguard gateway is down/disabled.
Even better: it's actually the VPN pass rule that still directs traffic to the gateway group, even though its members are all down!
Wireguard gateway is down:

Traffic is passed to the "VPN gateway"?!

To be crystal clear: when I access wtfismyip.com and similar in that situation, it shows my ISP IP, NOT any VPN's.

WHAT?!
What am I missing here, if anything  :-\ ?

PS:
The rule that routes traffic to the VPN gateway group also sets a NO_WAN_EGRESS tag.
I have a floating rule defined on my WAN interface to reject all traffic with that tag.
This used to work, but doesn't seem to anymore?
This is why I resorted to the explicit reject rule in the first place.

PPS:
I also came across this gem in trying to fix this issue. That could most definitely use some clarifying as well...
I fiddled around with that setting, but it didn't make any difference in my case, and still wouldn't explain the difference in behavior between an OpenVPN and Wireguard gateway.