[CLOSED] OPNsense very low performance with/out Wireguard (with vlan, bridges)

[Martinf]

Hello

after using a Linksys wrt1900acs V2 with openwrt for many years, it is time up upgrade. Based on reading some reviews I decided to try OPNsense on my new HW: Intel N100, 6x 2,5 Gbit eth, 16 GB DDR5 and more than enough SSD disk space.

Installation
------------
The basic installation and configuration was no problem with the tutorials. For the configuration of the bridges and Wireguard I was closely following these three:
- https://docs.opnsense.org/manual/how-tos/lan_bridge.html
- https://docs.opnsense.org/manual/how-tos/wireguard-client-proton.html
- https://gist.github.com/morningreis/eeda36e8bb07dcb750d77e9a744776e8

Requirements
---------------
The attached picture show my simple setup. I need only three vlans with bridges. Two of them will connect over Wireguard with the Internet and the thired Vlan will connect directly with the Internet. No wifi, no vpn-policiy-routing nor more Wireguard clients are required on the router.

Status and Issue
------------------
Based on the tutorials I was able to setup everything. Yes it is working but with one issue: PERFORMANCE!
OPNsense shows an orange latency warning for Wireguard on the dashbord. This might explain the low performance on Wireguard but also the direct Internet connection is not acceptable.
To ensure that I do not have an issue with the HW or Proxmox, I installed openwrt on it. This showed that the issue is most likely my OPNsense configuration. Here I need support of you please.

Q1 - How can I get rid of the organge latency warning for Wireguard in the OPNsense dashboard?
============================================================

Q2 - How can I improve my OPNsense configuration to get a better performance for the direct wan access?
==================================================================

Solutions and workarounds which did not work
----------------------------------------------------
- Of course I read some recommendations to avoid using bridges and vlans together due to the performance impact. Vlan should be better installed with one port for each vlan - with this you hand over the vlan handling to the managed switch. This is no option for me because my old router can do it and has a better performance...
- I also placed the endpoint IP in the monitoring IP field. Nothing changed.


Looking forward to read your experience!


PS if your are interested in two numbers -

1 The performance difference between my old installation and this new one is approx 40-50% worse for Wireguard connections on this new system (of course there is the latency warning!).

2 The performance difference from this installation compared with a quick installation of openwrt also on the new hardware, shows what is possible for the wan connection. Assuming that the wan connection of OPNsense and openwrt will be similar with a good configuration and tuning of OPNsense - the wan throughput of my current OPNsense installation can double!!!

[Martinf]

Update

I have updated my topic. Situation unchanged.

Maybe you can write me about your performance
with a similar configuration - wireguard, vlan and bridges?

I your performance was like mine - what have you done to improve it?

[s4n]

Hi,

sorry to say but for me your setup makes no sense.
If you are using VLANs the only use-case for a bridge is for WIFI (bridge LAN <-> WIFI). (and bridging physical ports to VLANs on Firewall)

Microsegmentation with VLANs is absolutley fine but why you are using so many bridges?
What do you want achieve with your setup? especially with "bridge lan", "bridge IoT" and "bridge guest"?

What type of Wifi AP do you have? What type of switch do you have?


You already have VLANs which act as "Local LAN" - enable the VLAN interfaces and give them IP addresses. Enable DHCP for those interfaces, set the tagging on the switch correctly.

Tag the ports correctly on the switch (it will be only layer 2 - no ip addresses are involved) so your switch should that handle easily.
The VLAN routing it self is done on the OPNsense.

Make the firewall rules as described here:
https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html
for your two VLANs which should use Wireguard/Proton VPN to access the internet.

Another policy for your "normal internet" access clients. thats it.

[Martinf]

Hello and thank you for your feedback,

in the recent months it was really handy to have one nic per vlan on the router. maybe not that important in the test environment (picture above) but in production it was.

Hardware
- 'dumb' APs, TP-Link Archer C6 EU V2.0 with openwrt
- managed switches, TP-Link TL-SG108E
I am satisfied with them so far. Stable and fast enough for the purpose.

If I got it right, your solution is to use the Vlan interfaces with IP and DHCP instead of the bridges.
Consequently, the bridges will get deleted or deactivated. My tagged bridge port with all three vlans is not required anymore, I need a different solution for my virtual nic (for the local VMs) and in future I need three lan cables from the router to the switch instead of one.

Will this really double the overall OPNsense throughput?
Did you test/ measure this scenario - what was the outcome?

Why I am asking - this would indicate that the standard bridge functionality here should be used very, very carefully because of an possible heavy impact on the throughput. And somehow this does not sound practical in a way...

[s4n]

If I got it right, your solution is to use the Vlan interfaces with IP and DHCP instead of the bridges.

that is correct.
add the VLANs to your switch,
vlan2 (LAN), vlan3(IoT), vlan4(Guest), vlan5, vlan666 (internet)

add the vlans to the nic in PROXMOX also. keep in mind to you will have to configure a TAGGED port for SWITCH <-> PROXMOX with all needed VLANs afterwards
check out this video.
https://www.youtube.com/watch?v=stQzK0p59Fc

add the necessary nics to your OPNsense installation, add VLANs, add IP addresses to VLANs, add DHCP, add firewalls rules, add NAT (if you don't use the auto feature)

connect isp router to your switch, give this port its own VLAN (666) UNTAGGED port.

the ports connecting wired devices are UNTAGGED.
I assume that OPENwrt can handle also VLANs - same game: add the vlans you need (Guest, IoT, LAN), make wireless interfaces/bridges to the VLANs.

You will have a clean setup with OPNsense responsible for all traffic also inter VLAN traffic, DHCP, DNS, Wireguard, etc.
This should do the trick.

will this really double the overall OPNsense throughput?
Did you test/ measure this scenario - what was the outcome?

Why I am asking - this would indicate that the standard bridge functionality here should be used very, very carefully because of an possible heavy impact on the throughput. And somehow this does not sound practical in a way...

(20+ years experience)
keep your setup clean keep it simple. Only you are able to do it you don't need to :P
if you really have performance issues afterwards it will be easier to identify the issuer.

the performance itself depends on hardware and configuration. a good start is to use iperf to measure the internal throughput. for external you can use external test sites.

just to be clear after reading your posts several times: one physical link can carry multiple VLANs. if you want to use only one physical NIC you can do it.

[Martinf]

Hello again,

thank you for your input.

So far I have done some more tests and based on the results I can exclude HW issues. Consequently it is either my OPNsense configurartion - I assume that this is it and that is my level of experience with OPNsense - or the implementation of the required functionality.

I will continue with some more testing, using your feedback which might show that by not using the bridge functionality the wan throughput will double to rise on the same level as with other sw solutions.

Concerning the Wireguard warning, I do not have a clue yet.

In the meantime I am back on openwrt.