SSH NAT not working

[FLguy]

Hey all,
I can't believe I can't solve this one but failing to get either 1:1 or Port Forward NAT to work.  The goal is to create a "DMZ host, " just a 1:1 NAT with filters for what ports you want to be exposed.  6+ hours later, and I'm still in this rabbit hole.  I first tried using my main FW, then a test VM FW, and 3rd brand new hardware firewall.  No matter what I do.  I can't get SSH NAT'ed over OPNsense right now.

I have tried using Port Forward, & 1:1 NATs, setting rules, and even finding the configuration for proxy arps.  In my heart, 1:1 NATs should be doing proxy arp by default.  In some configuration cases, I can see the traffic passing in Live View and capturing packets in tcpdump on the ssh server side.  But SSH will not connect.

Let's use the test lab to talk about the problem.  An ESXi host running two VMs + ssh client on Windows:

Windows host (putty): 192.168.169.10

OPNsense 23.1 FW vm:
WAN (DHCP):192.168.169.44 [esxi Port Group: VM Network]
LAN: 192.168.1.1 [esxi Port Group: PG Test]

Fresh ubuntu VM (sshd):
LAN: 192.168.1.101 [esxi Port Group: PG Test]

But let's go really basic, just a basic Port Forward.  I have a "management" host at .100.  .100 and .101 can SSH into each other just fine.  If I put either of these VMs into the VM Network port group (bypass the firewall), the Windows host can ssh.  Inside VMs can reach the internet.

OPNsense was installed fresh and ran the setup wizard.  The only non-default change was unchecking Block Private networks on WAN interface.   

Code: [Select]

Firewall:NAT:Port Forward
Interface:WAN
Destination: WAN address
Dest Port: 22 ssh
Redirect target IP: 192.168.1.101
Redirect target port: 22 ssh
With everything else set to defaults:

Code: [Select]

NAT reflection: system default = disabled
Filter rule association: Add associated filter rule
Save and Apply.  I see both the NAT rule and WAN rule get created. 

In the Live View, traffic is allowed:


Attached is the pcap from .101.  3way doesn't complete, .10 doesn't Ack.  :|  What is going on?  Getting a 1to1 NAT is less successful than this (ARP is an issue).  But I want to know what I'm going wrong?  I see no errors on the firewall, but SSH will not connect. 

Thanks,
Nick

[FLguy]

Is this a bug for NAT for 23.1?  Can anyone tell me if this works for you in >23.1?  A simple diagram, nothing fancy:

Very simple Firewall:NAT:Port Forward:





After this Saving and Applying this config, I get these two new Rules created:




Should this work?  Live View shows one single green pass.  Attaching two captures, one from the ssh client and the other from the ssh server.  Don't want to open captures, no worries. Here is a summary:

• openssl ssh client - always sends the same 5 syn packets for dport:22 (with OpenSSL SSH client will ARP after 2nd/3rd non-reply)
• sshserver - will get all 5 packets from the client and replies to all five syn x 2.  Always get about 14 to 15 packets in the capture.

I have tried this both in the physical hardware firewall and VM lab.  Using the online write-up, and limited videos on OPNsense Port Forward, this should work?  Does it for you?  After 18+ hours on this, I want to ensure I'm not going crazy?

[Patrick M. Hausen]

In the firewall rule the destination must be WAN_Address.

[FLguy]

In the firewall rule the destination must be WAN_Address.

@pmhausen Thank you for the suggestion.  I believe this could be where the problem lies.  Filtering vs NATing.  I believe the NAT is working (the SSH server is getting the packets).  OPNsense is dropping packets going back to ssh client.  I don't see this anywhere; just my assumption.

So we are clear, the firewall rule above is automatically created when I create the NAT rule and can not be edited.  This happens due to this setting in Port Forward:

• None - Does not care about a WAN rule
• Add associated (default)- Create a linked WAN rule that can not be edited
• Add unassociated - Creates a WAN rule unlinked to the NAT.  Rule can not be edited
• Pass - Does not create a WAN rule.

So I followed your suggestion.  This makes me feel like there is a bug, maybe not with NAT but with this Associated filter setting itself!


It doesn't work.  Live view shows a pass to the internal IP.  I have tried setting the WAN rule to WAN address with an unassociated filter and None.  Neither works; with the None setting, I ssh server doesn't see the traffic, and I get a block in Live View.  With the WAN rule to the internal address, the ssh server does see the traffic.  This has to be a bug.  I tried this on 21.1.

I would like to know if anyone can create a Port forward using the default associated filter and if it works?  That is, this software setting has a significant flaw in it.  I figured out how to get 1:1 nat working with floating rules.  But that is another can of warms! 

[Patrick M. Hausen]

I always use "Pass" for inbound port forwarding. Filtering for particular source addresses can be done in the NAT rule. With "Pass" no additional firewall rule is necessary.

[franco]

Would you mind enabling the log option in the port forward? It would help with debugging.


Cheers,
Franco

[FLguy]

@pmhausen  VERY much appreciate your replies.  Yes, you are correct.  Pass does work.  In fact, it's the only way I can get Port Forward NAT's to work if I use Pass.  My concerns are with the other 3 options.  They simply don't work.  For example, you should be able to configure a NAT rule and a filter rule independently of each other.  It's how pf works. 

I'm starting to see why.  The SSH server -> SSH client Return traffic is not hitting the correct rule, I believe.



The return traffic is hitting a rule I don't know of. 



I don't know where this rule is set.

[FLguy]

Would you mind enabling the log option in the port forward? It would help with debugging.


Cheers,
Franco

Did I get the right log?  Thank you! 

Just to confirm, Live View and Plain view are the same data, correct?

[FLguy]

Oh my FACE...  The return traffic is hitting an automatic floating rule.   

It's the ip4+6 out any:any rule, which seems horrible.  Why does that rule exist? For reference, it is Rule 64 from my last screenshots.  How is NAT working for anyone at this point?   

"let out anything from firewall host itself."
If I put an "any direction" floating rule to 192.168.169.101 port 22. that works.  This can not be acceptable.  Aren't Floating rules evaluated before interface rules?  I never knew there were so many auto-floating rules.  Never thought to look in there.   

[franco]

Testing from the directly attached WAN can be tricky. Go to Firewall: Settings: Advanced and check "Disable reply-to on WAN rules". It should work without an additional rule then?


Cheers,
Franco

[FLguy]

Testing from the directly attached WAN can be tricky. Go to Firewall: Settings: Advanced and check "Disable reply-to on WAN rules". It should work without an additional rule then?

Yes "Disabling reply-to on WAN rules" worked, THANK YOU.  To be honest, I had looked at this option many times, but never set it.  Because the help reads, "With Mulit-WAN ... ensure traffic leaves the same interface it arrives on  ..." In my test case, that is happening, traffic will leave the same interface that it came on?  Also, I'm not using Mulit-WAN. 

In your post, you say, "testing from the directly attached WAN is tricky".  So I'm taking this as my SSH client being on the same subnet as the WAN IP.  So I created another OPNsense FW in front just to route, no nat'ing on this firewall.  So my ssh client is not directly attached anymore:



So now I ssh from 169.10 to 10.13.37.79 (WAN IP of the inside NAT FW).  That does not work.  Until I disable the "reply-to on WAN" on the new outside fw. (192.168.169.41 interface).  No natting, just routing, Traffic will exit the same interface it came in on.  So I have to ask, what is this rule or setting doing? 

When setting disable reply-to, I was expecting to see that auto floating rule disappear that didn't happen.  Thank you,
Nick