Upload speed inexistent using OPNSense to direct traffic to a Linux gateway VM

[ownerer]

EDIT: updated topic title as this turns out to have nothing to do with Wireguard even.

For context, this is kind of a follow-up for this topic.

TLDR: Wireguard just does not run well enough yet on OPNSense.
Between the issue I've described in the topic linked above and the kernel panics it seems to introduce as described here, I've decided to look for another solution:
I still want to use Wireguard, but I don't want it to mess with OPNSense functionality and break all my networking whenever it decides to act up.
-> abstract Wireguard stuff from OPNSense.

I've set up a VM running Alpine Linux to serve as a Wireguard gateway.
Nothing fancy, just

• a plain Alpine Linux install,
• eth0 is connected to the LAN network,
• using wg-quick up wg0 to bring up the interface,
• using these rules to enable forwarding (I haven't bothered with killswitches yet)

Code Select Expand

sysctl -w net.ipv4.ip_forward=1
sysctl -p
iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o wg0 -j ACCEPT
iptables -A FORWARD -i wg0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT


In OPNSense I then add a gateway on the LAN interface for the IP address that was assigned to the VM, et voilà.

This seemed to work fine, however when testing throughput I've come across a new issue.
Download speed is perfect: I'm getting close to full line-speed (300Mbps).
Upload speed is almost non-existent: I'm getting 0.1Mbps (line-speed is 30Mbps).
I'm just using a Windows 10 machine to run speedtest.net to do this.

So here's the interesting part: if I set the default gateway of that machine to the IP of the VM, as opposed to OPNSense's IP (i.e.: traffic is going straight to/from the VM as opposed to being routed to/from it by OPNSense);
I DO get the near full line-speed for uploads as well.
I also get near full line-speed for uploads when:

• I just go out WAN;
• I use the OpenVPN client running on OPNSense.

So it's not like OPNSense can't deliver.

To summarize:

• Upload is crap when: w10 machine -> OPNSense -> VPN gateway -> internet (via OPNSense)
• Upload is all good when: w10 machine -> VPN gateway -> internet (via OPNSense)

So I'm posting this here seeing as the only variable that seems to make the difference is OPNSense' presence in that part of the path.
Any ideas on how OPNSense seems to be slicing my upload performance here?

[ownerer]

Some additional information:

I just tested this setup with a virtualized instance of OPNSense.
The problem persisted there as well, so two determinations:

• The OPNSense VM was on the same ESXi host as the Alpine Linux one -> the problem is not physical wiring or any other external device.
• The OPNSense VM was a factory install: all I did was do a fresh install, assign WAN and LAN interfaces and then add the gateway and a firewall rule to route a client device (also a VM on the same host) to the gateway -> it's not caused by any fancy configurations on my main instance.

Does anyone have any insights?

[ownerer]

no one  :-\ ?

[ownerer]

So tonight I've taken another shot at this.

New information: as expected it doesn't even have anything to do with Wireguard.
If I just set up the Alpine VM with 2 NICs, one (eth0/LAN) being NATted to the other (eth1/WAN) and set it (eth0) as the default gateway of a test machine, the exact same symptoms as described in the OP occur.

• OPNSense doing the gateway routing -> crap upload
• Alpine VM set as gateway directly on test machine NIC -> all good

This topic has now been read 260+ times, does truly no one have any idea on where to even start looking here :-\ ?!

[ownerer]

Seeing as this is quickly disappearing into the background again I've also posted this question on reddit.
So for reference: https://www.reddit.com/r/OPNsenseFirewall/comments/g3lscc/upload_speed_inexistent_using_opnsense_to_direct/