Xen WAN Performance Poor

[nlaird80]

Here is one puzzling me...
Network is simple. vLAN 1 is my WAN, vLAN 2 is my LAN.
OPNsense is on a VM in my Xen farm and connected to both vlans via 10GbE.

If I put another VM on the LAN vLAN and iperf3 to/from the client and the OPNsense LAN IP, it hits 9.35Gbps. Great!
If I iperf3 to the WAN interface on OPNsense... same.. fast!
If I iperf3 to a peer VM (just cloned my client and started iperf3 as a server) I get 9+ Gbps... great!
If I iperf3 from the same VM on the LAN to a system out on the WAN I get 600Kbps and it sawtooths 600Kbps / 0 / 0 /600 / 0 / 0 and averages 200Kbps overall.

So I ran iperf3 from the OPNsense VM directly to the same target and get 2.45Gbps and tons of retries. Thousands of retries per second... only on the WAN. LAN looks clean for retries (low numbers).

So I am thinking it is something with the NAT between the LAN/WAN.

I'm testing now removing NAT and just letting OPNsense route that way. Thankfully I don't NEED NAT on my network, but for my ultimate purpose I need NAT functional.

Will post results of getting NAT out of the loop.

Any ideas would be appreciated. I would blame Xen, but the LAN to LAN tests are super fast even from OPNsense to a client VM.

Let me know if anything needs clarified. I was debating drawing it up as a diagram if needed.

EDIT: Turning off packet filtering totally and disabling NAT rule generation changed OPNsense to external WAN host iperf results from ~2.4Gbps to 3.0Gbps. Retries also went from around 5K down to 300 total.

[franco]

Hi there,

Xen can be challenging for OPNsense/FreeBSD for several reasons.

There are docs here that describe to install the plugin, make sure to disable hardware offloading features, use the em(4) driver, etc.:

https://docs.opnsense.org/manual/virtuals.html

It looks like hardware offloading is causing your particular issue as NAT rewrites packets?


Cheers,
Franco

[nlaird80]

I've unfortunately tried it with offloading both enabled and disabled. It had no impact. The biggest change so far was removing NAT from the mix. It dramatically lowered the number of packet re-transmissions.

My understanding of Xen's virtual switch is that between VMs on the same LAN and the traffic doesn't have to actually touch an adapter. However I have five hosts in the pool and they are all running on different physical hosts. So I know my traffic is crossing my physical 10GbE switch. The adapters are capable of hitting almost 10GbE in the VMs. It's once OPNsense's WAN interface gets involved it drops. LAN side is great.

I'm still working on getting my routes added so I can fully test without NAT. I'll have an update later today.

Sidebar: I have the Xen plugin installed in OPNsense. My understanding of that plugin though is it's the standard xenstore and xe-guest-tools. Those really aren't drivers or changes to a system for performance. They are essentially the connector to Dom0's xenstore databse and scripts to collect and push performance metrics there. No networking changes, no video drivers, etc. Not really on par with Virtualbox or VMWare guest tools/drivers which install new virtual hardware drivers into the OS.

Inspecting what is installed, you basically are installing a scraper for ifconfig and sysctl output. Check /usr/local/sbin/xe-update-guest-attrs to see what I am rambling about.

Updates on my NAT-less test coming soon. Still puzzled why WAN gets ~3Gb when LAN can get 9+. Both are talking to a peer VM on the same switch. Both are initiated from OPN to the other VM who is listening for connections.

[nlaird80]

Update #1: With NAT off, testing again from OPN > WAN Server I get about 2.3Gbps. Testing from OPN > LAN Server is about the same. So NAT seems to be drastically impacting some of the WAN traffic. I'm now working to get WAN Server to talk to LAN Server through OPN without NAT.

edit: Also the firewall is set to disabled. I'm going to try with/without it too.

[nlaird80]

Here is the current state of things. All tests are conducted with iperf3.

OPN_WAN > WAN_Server = ~9.3Gbps (retrans around 150 packets)
OPN_LAN > LAN_Server = ~9.3Gbps (retrans around 60 packets)
LAN_Server <> WAN_Server = ~280Kbps (sawtoothed between 600 and 0Kbps, retrans around 300 packets)
This is with NAT Outbound disabled. Xen package installed and functional. All hardware offloading disabled.
Firewall rules for NAT Port forwarding are "No redirect allow * to *"
Rules for LAN and WAN are allow * to * as well. (Basically getting this thing to work purely as a router and not block anything.)

It seems that when OPN needs to process a packet by moving it from LAN to WAN it simply fails to do that quickly.

If I pfctl -d to totally shut down the packet filter speeds don't change. Same average speed. I'm at a loss here.

[franco]

pfctl -d is pretty harsh and would suggest this is a FreeBSD networking issue or indeed in conjunction with Xen. Someone once suggested these tweaks in the host as well...

nettool -K <vif name> tx off
nettool -K <xen bridge> tx off

[nlaird80]

I agree on the Xen > FreeBSD issue. Here is my latest test result:

I took the VM with OPNsense in it and without changing the VM itself, booted it from an ISO and installed Ubuntu 14.04. Didn't put any Xen tools in it, just a straight from the CD install. Setup iptables masquerading/NAT and gave it the same IPs as when it was running OPNsense. Repeating the tests I get:

Ubuntu WAN > WAN Server ~9.38Gbps (4 retrans)
Ubuntu LAN > LAN Server ~9.39Gbps (11 retrans)
LAN Server <> WAN Server ~8.96Gbps (573 retrans)

So all I can conclude at the moment is OPNsense, since it's on FreeBSD has an issue with the networking stack and Xen.

I remember a long time ago using a bunch of Dom0 tweaks for pfSense. I'll go try them. I think it was this:
xe vif-param-set uuid=$PIFUUID other-config:ethtool-gso=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-ufo=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tso=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-sg=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tx=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-rx=“off”

I'll try changing them now...

[nlaird80]

xe vif-param-set uuid=$PIFUUID other-config:ethtool-gso=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-ufo=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tso=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-sg=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tx=“off”
xe vif-param-set uuid=$PIFUUID other-config:ethtool-rx=“off”

Made a huge difference applying those settings to WAN and LAN.
The sawtoothing stopped. Packet retrans dropped to under 200.
LAN Server <> WAN Server is now at ~1.9Gbps. It's not as fast as Ubuntu but it's tolerable now.

It must be the networking drivers in FreeBSD and how they interact with the virtual adapters.

With pfctl -d I can get ~2.5Gbps. So the packet filter is also impacting the speeds.

Not a complete solution, not the best performing option. But much more functional than before.

[franco]

Thanks for checking back. Hopefully 17.1 with FreeBSD 11 can help this performance a bit, although that's still January 2017, not counting any test builds we may release.


Cheers,
Franco

[cdburgess75]

Running 16.7 with major traffic (NAT and not:)  Cannot reproduce given his environment.. I am doing IDS/IPS, NAT, With serious traffic. Please let this post know what we are missing, nlird80 so we can possibly find the cause with you.  YeeHaw!

[nlaird80]

How is your network setup within XenServer? Are you using a bond to a physical adapter or is it internal networking?