Messages

I cannot seem to determine if DHCP and Unbound can be setup to behave like I would prefer.

I have the OPN system on the domain arl.lab.xyz with hostname opnsense.
I have the LAN DHCP server setup to provide the domain lan.arl.lab.xyz
I have another interface called DEV setup to provide the domain dev.arl.lab.xyz
Both are different interfaces to different networks with different subnets naturally.

When I query DNS I can ask for opnsense.arl.lab.xyz no problem.
I have a client (client1) on the lan interface, so it's DHCP lease says it's client1.lan.arl.lab.xyz HOWEVER when I query DNS, this is invalid. It appears to be setting them as client1.arl.lab.xyz and ignoring the subdomain used in the DHCP server.

Is this a bug in the DHCP server setting values in unbound? All leases on all interfaces just get assigned to the domain of the system itself.

How is your network setup within XenServer? Are you using a bond to a physical adapter or is it internal networking?

xe vif-param-set uuid=$PIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-rx="off"

Made a huge difference applying those settings to WAN and LAN.
The sawtoothing stopped. Packet retrans dropped to under 200.
LAN Server <> WAN Server is now at ~1.9Gbps. It's not as fast as Ubuntu but it's tolerable now.

It must be the networking drivers in FreeBSD and how they interact with the virtual adapters.

With pfctl -d I can get ~2.5Gbps. So the packet filter is also impacting the speeds.

Not a complete solution, not the best performing option. But much more functional than before.

I agree on the Xen > FreeBSD issue. Here is my latest test result:

I took the VM with OPNsense in it and without changing the VM itself, booted it from an ISO and installed Ubuntu 14.04. Didn't put any Xen tools in it, just a straight from the CD install. Setup iptables masquerading/NAT and gave it the same IPs as when it was running OPNsense. Repeating the tests I get:

Ubuntu WAN > WAN Server ~9.38Gbps (4 retrans)
Ubuntu LAN > LAN Server ~9.39Gbps (11 retrans)
LAN Server <> WAN Server ~8.96Gbps (573 retrans)

So all I can conclude at the moment is OPNsense, since it's on FreeBSD has an issue with the networking stack and Xen.

I remember a long time ago using a bunch of Dom0 tweaks for pfSense. I'll go try them. I think it was this:
xe vif-param-set uuid=$PIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=$PIFUUID other-config:ethtool-rx="off"

I'll try changing them now...

Here is the current state of things. All tests are conducted with iperf3.

OPN_WAN > WAN_Server = ~9.3Gbps (retrans around 150 packets)
OPN_LAN > LAN_Server = ~9.3Gbps (retrans around 60 packets)
LAN_Server <> WAN_Server = ~280Kbps (sawtoothed between 600 and 0Kbps, retrans around 300 packets)
This is with NAT Outbound disabled. Xen package installed and functional. All hardware offloading disabled.
Firewall rules for NAT Port forwarding are "No redirect allow * to *"
Rules for LAN and WAN are allow * to * as well. (Basically getting this thing to work purely as a router and not block anything.)

It seems that when OPN needs to process a packet by moving it from LAN to WAN it simply fails to do that quickly.

If I pfctl -d to totally shut down the packet filter speeds don't change. Same average speed. I'm at a loss here.

Update #1: With NAT off, testing again from OPN > WAN Server I get about 2.3Gbps. Testing from OPN > LAN Server is about the same. So NAT seems to be drastically impacting some of the WAN traffic. I'm now working to get WAN Server to talk to LAN Server through OPN without NAT.

edit: Also the firewall is set to disabled. I'm going to try with/without it too.

I've unfortunately tried it with offloading both enabled and disabled. It had no impact. The biggest change so far was removing NAT from the mix. It dramatically lowered the number of packet re-transmissions.

My understanding of Xen's virtual switch is that between VMs on the same LAN and the traffic doesn't have to actually touch an adapter. However I have five hosts in the pool and they are all running on different physical hosts. So I know my traffic is crossing my physical 10GbE switch. The adapters are capable of hitting almost 10GbE in the VMs. It's once OPNsense's WAN interface gets involved it drops. LAN side is great.

I'm still working on getting my routes added so I can fully test without NAT. I'll have an update later today.

Sidebar: I have the Xen plugin installed in OPNsense. My understanding of that plugin though is it's the standard xenstore and xe-guest-tools. Those really aren't drivers or changes to a system for performance. They are essentially the connector to Dom0's xenstore databse and scripts to collect and push performance metrics there. No networking changes, no video drivers, etc. Not really on par with Virtualbox or VMWare guest tools/drivers which install new virtual hardware drivers into the OS.

Inspecting what is installed, you basically are installing a scraper for ifconfig and sysctl output. Check /usr/local/sbin/xe-update-guest-attrs to see what I am rambling about. :)

Updates on my NAT-less test coming soon. Still puzzled why WAN gets ~3Gb when LAN can get 9+. Both are talking to a peer VM on the same switch. Both are initiated from OPN to the other VM who is listening for connections.

Here is one puzzling me...
Network is simple. vLAN 1 is my WAN, vLAN 2 is my LAN.
OPNsense is on a VM in my Xen farm and connected to both vlans via 10GbE.

If I put another VM on the LAN vLAN and iperf3 to/from the client and the OPNsense LAN IP, it hits 9.35Gbps. Great!
If I iperf3 to the WAN interface on OPNsense... same.. fast!
If I iperf3 to a peer VM (just cloned my client and started iperf3 as a server) I get 9+ Gbps... great!
If I iperf3 from the same VM on the LAN to a system out on the WAN I get 600Kbps and it sawtooths 600Kbps / 0 / 0 /600 / 0 / 0 and averages 200Kbps overall.

So I ran iperf3 from the OPNsense VM directly to the same target and get 2.45Gbps and tons of retries. Thousands of retries per second... only on the WAN. LAN looks clean for retries (low numbers).

So I am thinking it is something with the NAT between the LAN/WAN.

I'm testing now removing NAT and just letting OPNsense route that way. Thankfully I don't NEED NAT on my network, but for my ultimate purpose I need NAT functional.

Will post results of getting NAT out of the loop.

Any ideas would be appreciated. I would blame Xen, but the LAN to LAN tests are super fast even from OPNsense to a client VM.

Let me know if anything needs clarified. I was debating drawing it up as a diagram if needed.

EDIT: Turning off packet filtering totally and disabling NAT rule generation changed OPNsense to external WAN host iperf results from ~2.4Gbps to 3.0Gbps. Retries also went from around 5K down to 300 total.

That would most likely be enough. We just want users to be able to change their own passwords and/or get their QR code without the admin from manually distributing them all (50+).

We allow our users of VPN to sign into the management GUI and only access the password management page. This lets them self-service a password change. What we would also like to allow is self-service for TOTP seeds (at a minimum the ability to get their QR). I cannot determine if there is already a permission in the access-control.

Any way to allow self-service for this? I would even be willing to accept self service to their own account management page (but not other users')

Thanks!

Last comment tonight...
I think I was right about the Nighthawk not wanting to NAT anything that wasn't on it's LAN subnet. I added a NAT rule on OPNSense to take "Server" traffic and rewrite it and now I can ping 8.8.8.8

Interestingly enough nothing else broke. I'm using Manual NAT now. I'm still going to disable it for my final setup as I don't need it, but I'm finding that my problems continue to be a rule somewhere else on the network in most cases.

OK I got it working. Here is what it took.
I setup a new test network like this:

My cable modem's router (a Nighthawk 7000) is 10.1.10.1/24
My Laptop is 10.1.10.50/24 it's only gateway is the Nighthawk.
My OPNSense was on 10.1.10.74/24 WAN and 10.0.1.1/24 LAN. It's static IPs and a fixed default gateway entry.
A "server" was on the OPNSense LAN at 10.0.1.10/24.

I disabled NAT and set rules on both WAN and LAN to allow everything. It still didn't let me hit the webgui on OPNSense on the WAN. It seemed to correct the problem getting to the WAN IP. I couldn't pass traffic between networks though until I added a route on the Nighthawk pointing 10.0.1.0/24 to 10.1.10.74 (makes sense since I don't have another GW on the laptop) and now everything seems happy.

I'm going to say it was the NAT. However when I re-enabled Auto Outbound NAT it kept working... so I'm at a loss. But it's working as expected after all the changes. I'm sure I missed something and will build this one more time.

It appears I was fighting a few things. 1) A missing route so my traffic was only working one way. 2) NAT was getting in the way.

The only thing not working at the moment is if I ping 8.8.8.8 from the LAN "server" it doesn't work. I think my Nighthawk is getting the traffic (it should the routes are all there) but since the packet's source IP is not in the 10.1.10.0/24 network I think it's refusing to NAT it outbound. I don't have granular control over it's rules unfortunately. This however is fine as my home network isn't where I am ultimately going to deploy this.

If I have any news to share about what else I learn I'll revisit this and continue to tack on a comment or two.

Thanks for a few things to check. I've togged enough settings here and there I don't remember if I had NAT off at the same time as the floating rule or not. I'll try a few more configurations. I have block bogon and RFC1918 unchecked. The goal is to use it pretty much as a router and NAT shouldn't be required. I wanted to be able to route but apply firewall rules if/when needed to close out things. (Route this but not that)

My understanding is NAT changes the packet before the input filter kicks in. So I can see how that could be an issue. Since my problem occurs on multiple hypervisors (and even in pfSense when I tried falling back to it once) I can see how my config is most likely to blame.

I'll report back in maybe an hour after some more tests. Thanks again.

Here is what I added to get traffic moving across the WAN interface. I'm stumped why a rule here worked but on the WAN it didn't. I ran pfctl -sr and get a huge pile of rules to sort through. I'm parsing them now.

Yes, I did read that. I didn't switch Xen to Intel because I didn't want to impact the entire farm. However my virtual box and vmware fusion both use Intel and the issue persisted.

I have all offloading disabled in both the guest and at the Dom0 level. Performance is good. I can move data across the interfaces at good rates.

I have more information I'm going to edit into the main question it gets more curious. Give me about 90 seconds and I'll put and <ADDED> block on the bottom of my post.