Messages

Hmm, interesting!

I was unaware of the package locking functionality!
I'm not at all well versed in things BSD btw, I'll just mention that too. I'm just tinkering here  :P

So to be clear, in case of Bhyve for example, I would build it from the ports tree and then I should be able to find it in the package list in the OPNsense GUI and lock it there?

I assume this does the same as using pkg-lock, as described in the manual?

Just making sure I understand correctly :)

Anyone?

Hi,

for a number of reasons that are not important here, I want to run some VMs in Bhyve on my OPNsense box (homelab).

Since setting this up needs to happen outside OPNsense default features, building the libraries from the ports tree etc, my question here becomes: what happens when an OPNsense update is installed? Is there a chance that the Bhyve setup will break somehow, and if so, how likely is that to happen and are there ways to minimize those chances or perhaps avoid that scenario entirely (other than just not doing this kind of thing on an OPNsense box obviously)?

I suppose this question could be generalized to any scenario where one needs/wants to run additional packages that need to be manually built and installed from the ports tree.

PS: I'm aware of the controversy around adding a virtualization layer to a firewall OS, adding tons of libraries, code, and as a consequence potential attack vectors etc
To anyone feeling the need to raise these concerns, I say: duly noted, I appreciate it and I do understand, but this is not the point of this topic  :)

Seeing as this is quickly disappearing into the background again I've also posted this question on reddit.
So for reference: https://www.reddit.com/r/OPNsenseFirewall/comments/g3lscc/upload_speed_inexistent_using_opnsense_to_direct/

[any]

So tonight I've taken another shot at this.

New information: as expected it doesn't even have anything to do with Wireguard.
If I just set up the Alpine VM with 2 NICs, one (eth0/LAN) being NATted to the other (eth1/WAN) and set it (eth0) as the default gateway of a test machine, the exact same symptoms as described in the OP occur.

• OPNSense doing the gateway routing -> crap upload
• Alpine VM set as gateway directly on test machine NIC -> all good

This topic has now been read 260+ times, does truly no one have any idea on where to even start looking here :-\ ?!

no one  :-\ ?

Some additional information:

I just tested this setup with a virtualized instance of OPNSense.
The problem persisted there as well, so two determinations:

• The OPNSense VM was on the same ESXi host as the Alpine Linux one -> the problem is not physical wiring or any other external device.
• The OPNSense VM was a factory install: all I did was do a fresh install, assign WAN and LAN interfaces and then add the gateway and a firewall rule to route a client device (also a VM on the same host) to the gateway -> it's not caused by any fancy configurations on my main instance.

Does anyone have any insights?

[EDIT:]

EDIT: updated topic title as this turns out to have nothing to do with Wireguard even.

For context, this is kind of a follow-up for this topic.

TLDR: Wireguard just does not run well enough yet on OPNSense.
Between the issue I've described in the topic linked above and the kernel panics it seems to introduce as described here, I've decided to look for another solution:
I still want to use Wireguard, but I don't want it to mess with OPNSense functionality and break all my networking whenever it decides to act up.
-> abstract Wireguard stuff from OPNSense.

I've set up a VM running Alpine Linux to serve as a Wireguard gateway.
Nothing fancy, just

• a plain Alpine Linux install,
• eth0 is connected to the LAN network,
• using wg-quick up wg0 to bring up the interface,
• using these rules to enable forwarding (I haven't bothered with killswitches yet)

Code Select Expand

sysctl -w net.ipv4.ip_forward=1
sysctl -p
iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o wg0 -j ACCEPT
iptables -A FORWARD -i wg0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT


In OPNSense I then add a gateway on the LAN interface for the IP address that was assigned to the VM, et voilà.

This seemed to work fine, however when testing throughput I've come across a new issue.
Download speed is perfect: I'm getting close to full line-speed (300Mbps).
Upload speed is almost non-existent: I'm getting 0.1Mbps (line-speed is 30Mbps).
I'm just using a Windows 10 machine to run speedtest.net to do this.

So here's the interesting part: if I set the default gateway of that machine to the IP of the VM, as opposed to OPNSense's IP (i.e.: traffic is going straight to/from the VM as opposed to being routed to/from it by OPNSense);
I DO get the near full line-speed for uploads as well.
I also get near full line-speed for uploads when:

• I just go out WAN;
• I use the OpenVPN client running on OPNSense.

So it's not like OPNSense can't deliver.

To summarize:

• Upload is crap when: w10 machine -> OPNSense -> VPN gateway -> internet (via OPNSense)
• Upload is all good when: w10 machine -> VPN gateway -> internet (via OPNSense)

So I'm posting this here seeing as the only variable that seems to make the difference is OPNSense' presence in that part of the path.
Any ideas on how OPNSense seems to be slicing my upload performance here?

[EDIT:]

EDIT:

I've created a new topic to follow up on this.

TLDR: Wireguard just does not run well enough yet on OPNSense.
Between the issue I've described here and the kernel panics it seems to introduce as described here, I've decided to look for another solution:
I still want to use Wireguard, but I don't want it to mess with OPNSense functionality and break all my networking whenever it decides to act up.
-> abstract Wireguard stuff from OPNSense.

[very]

If the gateways behave that differently, then that should at least be made very clear to the end-user.

Again, to summarize: the bug here is that, even though the gateway is marked down, used in a group or solo, the firewall still matches the routing rule for it, pushing traffic to it (so subsequent killswitch rules don't even have a chance), on top of which traffic somehow goes out over WAN instead of the Wireguard interface, unbeknownst to the user.

I get the implementation details might be different, but functionally it should exhibit the same behaviour. So classifying this as a "limitation" is being very mild. This is a bug/design flaw.
I mean let's be real, why do people use VPNs like this? To route traffic through they'd rather not have their ISP see. If you can't even trust the firewall to kill traffic when the gateway goes down, in any way, then what's the point?

Long story short: I'd argue Wireguard is currently unsuited to replace OpenVPN in this setup.
If you think I'm wrong in drawing that conclusion, then please tell me why, because I want to be (wrong, that is)!

Damn, came here from my own thread of Wireguard policy-based routing problems.
And here I thought I was having hardware issues!
"Good" to see I'm not alone getting these kernel panics.
It seems Wireguard and/or its implementation in OPNSense isn't quite stable enough yet after all :(

First of all: I have since updated to 20.1, it hasn't made any difference.

Can you verify if the traffic really leaves WAN? The log with accepted packets just indicates it tries to push these packets via the correct rule, no matter if gw group is down or not.

I don't see how there is any other way than for this to be the case?
Like I said: if I visit wtfismyip.com and similar sites in that state, they all show my ISP data.

Second test would be to add a rules to Wireguard gateway explicit, not the whole group, and try to reproduce.

Unfortunately this makes no difference, the exact same behaviour remains.

What is different about a Wireguard gateway from an OpenVPN gateway?
I mean, shouldn't the whole gateway logic be protocol agnostic?
If a gateway monitor says the gateway is down, then that's all that should matter -> don't pass traffic, evaluate next rule, done.
Yet there is a clear difference between how the 2 protocols' gateways behave.

Assuming I'm not missing anything, I would argue this is a bug and/or design flaw.

I have a new issue with Wireguard/Mullvad policy-based routing.
I created a separate topic here (Policy-based Wireguard(/Mullvad): firewall rules ignored when gateway is down) as to not hijack this one.
But I thought I'd mention it at least.

So I just discovered this gem.
A little context first:

• I have Wireguard/Mullvad set up with a gateway, as described here
• I have an alias defined for all hosts that must only access the internet through a VPN connection
• I have a VPN gateway group set up for load balancing/failover
• The firewall is configured to send all VPN destined traffic through the VPN gateway group
• The firewall is also configured to reject all VPN destined traffic that somehow makes it past that routing rule
• This screenshot shows the firewall config, these are the last 3 rules in the config for the LAN interface
  

This works perfectly when PIA (OpenVPN) is used in the VPN gateway group.
As you'd expect, when the client is disabled, the last but one rule is triggered and prevents VPN destined traffic from going out the WAN gateway group.
OpenVPN gateway is down:

Traffic is blocked:


Here comes the kicker though: replace the PIA gateway in the gateway group with the Mullvad gateway, and traffic merrily flows out from the WAN interface when the Wireguard gateway is down/disabled.
Even better: it's actually the VPN pass rule that still directs traffic to the gateway group, even though its members are all down!
Wireguard gateway is down:

Traffic is passed to the "VPN gateway"?!

To be crystal clear: when I access wtfismyip.com and similar in that situation, it shows my ISP IP, NOT any VPN's.

WHAT?!
What am I missing here, if anything  :-\ ?

PS:
The rule that routes traffic to the VPN gateway group also sets a NO_WAN_EGRESS tag.
I have a floating rule defined on my WAN interface to reject all traffic with that tag.
This used to work, but doesn't seem to anymore?
This is why I resorted to the explicit reject rule in the first place.

PPS:
I also came across this gem in trying to fix this issue. That could most definitely use some clarifying as well...
I fiddled around with that setting, but it didn't make any difference in my case, and still wouldn't explain the difference in behavior between an OpenVPN and Wireguard gateway.

I got here looking for the same thing!
Did you ever get any kind of reply to this question?