[SOLVED] Odd PacketFilter behavior in VM Guest

[nlaird80]

I am having many issues with OPNSense in a virtual environment. What it boils down to is when the packet filter is enabled and has any rules to allow traffic inbound they do not work. If I drop the filter (pfctl -d) then the traffic flows. This happens in three different hypervisors with both 16.1, 15.7, and the current Alpha.

A more specific example is if I open 443 to the WAN IP for management. Traffic is blocked. I can see the block on the firewall log even though there is a rule allowing the traffic. So I click the green arrow to create a new rule. The rule appears, and the traffic is still blocked. Once again if I disable the pf totally it works. It's baffling that I cannot get a virtualized copy working with the pf also fully functional.

Thoughts where to look next? I created an OPT1 interface and put a allow all to everywhere rule and it seems to be working. My LAN segment is working with NAT for VM clients. I just can't open the WAN for anything. I'll gather and post any data that might be helpful if asked.

<EDIT>
Adding a floating rule seems to have allowed traffic through. My wan rule (attached) and my layout (attached) should allow the client to talk to the server. The server can ping/ssh to the client no problem.
As for the float rule. I added another allow everything to everything and it started working.

[franco]

Did you see this? https://docs.opnsense.org/manual/virtuals.html

All hypervisors require a bit of tinkering with the setup, mostly disabling hardware acceleration features.

E.g. one user reported that switching Realtek emulated drivers to Intel in Proxmox removes all observed hiccups. XEN requires fiddling, Hyper-V requires patches that just today landed in FreeBSD 10.2 and thus weren't in the original 16.1. Focusing the effort on one type will probably yield better results than testing different ones.

So, which one shall we debug?

[nlaird80]

Yes, I did read that. I didn't switch Xen to Intel because I didn't want to impact the entire farm. However my virtual box and vmware fusion both use Intel and the issue persisted.

I have all offloading disabled in both the guest and at the Dom0 level. Performance is good. I can move data across the interfaces at good rates.

I have more information I'm going to edit into the main question it gets more curious. Give me about 90 seconds and I'll put and <ADDED> block on the bottom of my post.

[nlaird80]

Here is what I added to get traffic moving across the WAN interface. I'm stumped why a rule here worked but on the WAN it didn't. I ran pfctl -sr and get a huge pile of rules to sort through. I'm parsing them now.

[franco]

It looks like you're trying to access a server within the WAN, NAT can play tricks on you here, because WAN is not supposed to pass traffic to LAN by default. That's what port forwarding or 1:1 are used for traditionally, especially when the private networks are not supposed to leak outwards or public IP access is blocked.

You could also revert to a routing platform, although that will remove the firewall functionality.

You could also disable outbound NAT, it looks like your setup doesn't need it. In that case you can still use the firewall, but you'll need the rule in order to let traffic flow from WAN to internal networks still.

[franco]

Oh, the WAN rules automatically block RFC 1918 networks so this might interfere too. Your client IP falls into those ranges, thats most likely why you have to add the floating rule.

[nlaird80]

Thanks for a few things to check. I've togged enough settings here and there I don't remember if I had NAT off at the same time as the floating rule or not. I'll try a few more configurations. I have block bogon and RFC1918 unchecked. The goal is to use it pretty much as a router and NAT shouldn't be required. I wanted to be able to route but apply firewall rules if/when needed to close out things. (Route this but not that)

My understanding is NAT changes the packet before the input filter kicks in. So I can see how that could be an issue. Since my problem occurs on multiple hypervisors (and even in pfSense when I tried falling back to it once) I can see how my config is most likely to blame.

I'll report back in maybe an hour after some more tests. Thanks again.

[nlaird80]

OK I got it working. Here is what it took.
I setup a new test network like this:

My cable modem's router (a Nighthawk 7000) is 10.1.10.1/24
My Laptop is 10.1.10.50/24 it's only gateway is the Nighthawk.
My OPNSense was on 10.1.10.74/24 WAN and 10.0.1.1/24 LAN. It's static IPs and a fixed default gateway entry.
A "server" was on the OPNSense LAN at 10.0.1.10/24.

I disabled NAT and set rules on both WAN and LAN to allow everything. It still didn't let me hit the webgui on OPNSense on the WAN. It seemed to correct the problem getting to the WAN IP. I couldn't pass traffic between networks though until I added a route on the Nighthawk pointing 10.0.1.0/24 to 10.1.10.74 (makes sense since I don't have another GW on the laptop) and now everything seems happy.

I'm going to say it was the NAT. However when I re-enabled Auto Outbound NAT it kept working... so I'm at a loss. But it's working as expected after all the changes. I'm sure I missed something and will build this one more time.

It appears I was fighting a few things. 1) A missing route so my traffic was only working one way. 2) NAT was getting in the way.

The only thing not working at the moment is if I ping 8.8.8.8 from the LAN "server" it doesn't work. I think my Nighthawk is getting the traffic (it should the routes are all there) but since the packet's source IP is not in the 10.1.10.0/24 network I think it's refusing to NAT it outbound. I don't have granular control over it's rules unfortunately. This however is fine as my home network isn't where I am ultimately going to deploy this.

If I have any news to share about what else I learn I'll revisit this and continue to tack on a comment or two.

[nlaird80]

Last comment tonight...
I think I was right about the Nighthawk not wanting to NAT anything that wasn't on it's LAN subnet. I added a NAT rule on OPNSense to take "Server" traffic and rewrite it and now I can ping 8.8.8.8

Interestingly enough nothing else broke. I'm using Manual NAT now. I'm still going to disable it for my final setup as I don't need it, but I'm finding that my problems continue to be a rule somewhere else on the network in most cases.

[franco]

Nice work pushing through this and the explanations along the way. Made it way easier to troubleshoot / help from the forum side of things.