I'm doing something wrong, can't figure out what

[silent_mastodon]

So, I have a 16.1.x install, running in a VM (xenserver). It is connected to a RFC1918 private network (192.168.5.0/24) which itself is controlled by a baremetal Opnsense install.

The VM router's private subnet is set to 192.168.15.0/24, with a LAN address of 192.168.15.1. The WAN address is a static IP 192.168.5.136.

Hosts behind the VM, on the .15.0/24 network can contact the internet just fine out of the default gateway 192.168.5.1. DNS works fine.

What I can't get working is NAT port forwards into the .15.0/24 network.

I have the RFC1918 block *disabled* on the WAN interface, so that isn't the issue.
My NAT port-forward is a WebGUI redirect in the form

WAN rule WAN address:45876 -> 192.168.15.1:443 (LAN address)

The rule doesn't work. I've examined this rule in comparison to the exact same rule I have set on my baremetal router, and they are identical. My baremetal rule works perfectly, but the VM rule does not.

I'm at a loss, I've toggled all sort of the things, and ultimately reloaded the install fresh and have applied just this rule, and no go.

The firewall appears to be working (I tried adding a ICMP pass rule so that ping against the WAN address would work), but the NAT itself is ignoring/dropping my packets.

If I use nmap to try and probe 192.168.5.136:45876 I just get back "filtered". Testing my baremetal WAN address shows the same port as open like expected, so something in the NAT/firewall is getting in the way.

I'm stumped.

edit: My NAT port forward has the TCP protocol set, but I've tried TCP/UDP as well, which didn't make a difference.

[franco]

Hi there,

Can you find the rule in /tmp/rules.debug and paste it here?  Just for double-checking as a starting point.


Cheers,
Franco

[silent_mastodon]

Sure, and thanks for the reply!

I can't seem to make the insert code function work, so I apologize for the lack of formatting.

set optimization normal
set timeout { adaptive.start 0, adaptive.end 0 }
set limit states 96000
set limit src-nodes 96000

#System aliases
 
loopback = "{ lo0 }"
WAN = "{ xn0 }"
LAN = "{ xn1 }"

# SSH Lockout Table
table <sshlockout> persist
table <webConfiguratorlockout> persist
# Other tables
table <virusprot>
table <bogons> persist file "/usr/local/etc/bogons"
table <bogonsv6> persist file "/usr/local/etc/bogonsv6"
table <negate_networks>

# User Aliases
 
# Gateways
GWopnsense_main = " route-to ( xn0 192.168.5.1 ) "

 
set loginterface xn1

set skip on pfsync0

scrub on $WAN all    fragment reassemble
scrub on $LAN all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"


# Outbound NAT rules (automatic)

# Subnets to NAT
tonatsubnets  = "{ 127.0.0.0/8 192.168.15.0/24 }"
nat on $WAN  from $tonatsubnets to any port 500 -> 192.168.5.136/32  static-port
nat on $WAN  from $tonatsubnets to any -> 192.168.5.136/32 port 1024:65535 

# Load balancing
rdr-anchor "relayd/*"
# Anti lockout, prevent redirects for protected ports to this interface ip
no rdr on xn1 proto tcp from any to ( xn1 ) port { 443 80 22 } # NAT Inbound Redirects
rdr on xn0 proto { tcp udp } from any to 192.168.5.136 port 45876 -> 192.168.15.1 port 443

# UPnP
rdr-anchor "miniupnpd"

anchor "relayd/*"
anchor "openvpn/*"
anchor "ipsec/*"

#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log inet all label "Default deny rule IPv4"
block out log inet all label "Default deny rule IPv4"
block in log inet6 all  label "Default deny rule IPv6"
block out log inet6 all  label "Default deny rule IPv6"

# IPv6 ICMP is not auxilary, it is required for operation
# See man icmp6(4)
# 1    unreach         Destination unreachable
# 2    toobig          Packet too big
# 128  echoreq         Echo service request
# 129  echorep         Echo service reply
# 133  routersol       Router solicitation
# 134  routeradv       Router advertisement
# 135  neighbrsol      Neighbor solicitation
# 136  neighbradv      Neighbor advertisement
pass log quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136}  keep state

# Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
pass out log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136}  keep state
pass out log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136}  keep state
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136}  keep state
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136}  keep state
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136}  keep state

# We use the mighty pf, we cannot be fooled.
block log quick inet proto { tcp, udp } from any port = 0 to any
block log quick inet proto { tcp, udp } from any to any port = 0
block log quick inet6 proto { tcp, udp } from any port = 0 to any
block log quick inet6 proto { tcp, udp } from any to any port = 0

# SSH lockout
block in log quick proto tcp from <sshlockout> to (self) port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webConfiguratorlockout> to (self) port 443  label "webConfiguratorlockout"
block in log quick from <virusprot> to any label "virusprot overload table"
# block bogon networks (IPv4)
# http://www.cymru.com/Documents/bogon-bn-nonagg.txt
block in log quick on $WAN from <bogons> to any  label "block bogon IPv4 networks from WAN"
# block bogon networks (IPv6)
# http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
block in log quick on $WAN from <bogonsv6> to any label "block bogon IPv6 networks from WAN"
antispoof log for $WAN
antispoof log for $LAN
# allow access to DHCP server on LAN
pass in log quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67  label "allow access to DHCP server"
pass in log quick on $LAN proto udp from any port = 68 to 192.168.15.1 port = 67 label "allow access to DHCP server"
pass out log quick on $LAN proto udp from 192.168.15.1 port = 67 to any port = 68 label "allow access to DHCP server"
# allow access to DHCPv6 server on LAN
# We need inet6 icmp for stateless autoconfig and dhcpv6
pass log quick on $LAN inet6 proto udp from fe80::/10 to fe80::/10 port = 546 label "allow access to DHCPv6 server"
pass log quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 546 label "allow access to DHCPv6 server"
pass log quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 547 label "allow access to DHCPv6 server"
pass log quick on $LAN inet6 proto udp from ff02::/16 to fe80::/10 port = 547 label "allow access to DHCPv6 server"

# loopback
pass in log on $loopback inet all label "pass IPv4 loopback"
pass out log on $loopback inet all label "pass IPv4 loopback"
pass in log on $loopback inet6 all label "pass IPv6 loopback"
pass out log on $loopback inet6 all label "pass IPv6 loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out log inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out log inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out log route-to ( xn0 192.168.5.1 ) from 192.168.5.136 to !192.168.5.0/24 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in log quick on xn1 proto tcp from any to (xn1) port { 443 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  in  quick  on $WAN reply-to ( xn0 192.168.5.1 )  proto { tcp udp }  from any to 192.168.15.1 port 443 keep state  label "USER_RULE: NAT derp"
pass  in  quick  on $WAN reply-to ( xn0 192.168.5.1 ) inet proto icmp  from any to any keep state  label "USER_RULE: ping"
pass  in  quick  on $LAN inet from any to any keep state  label "USER_RULE: Default allow LAN to any rule"
# at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

# VPN Rules

Hope this helps!

[silent_mastodon]

Any ideas Franco? Or anyone else? Still totally stumped on this one.

[silent_mastodon]

I noted in the sticky that someone solved a problem with their VM-based opnsense router by turning off all the hardware assist in the configuration.

I figured I would give that a shot, since I didn't check for it explicitly before.

The docs are a little out of date (they say General->Settings-Networking but the location has moved to Interfaces->Settings now)
but I found the options. Two of them were already disabled, but one, Hardware Checksum was still enabled.  I disabled it.

Unfortunately, it didn't resolve my issue. I still can't reach my WebGUI through the WAN Nat port forward.

I also went ahead and disabled the VLAN tagging option, but that also made no difference.

[franco]

Does this only pertain to 443? The following looks suspicious as 192.168.15.1 is on xn1...

no rdr on xn1 proto tcp from any to ( xn1 ) port { 443 80 22 } # NAT Inbound Redirects
rdr on xn0 proto { tcp udp } from any to 192.168.5.136 port 45876 -> 192.168.15.1 port 443

Try moving the OPNsense HTTPS GUI port to a different one (e.g. 8443) just for testing purposes.

We added this redirection exception line to keep users from locking themselves out of the GUI when misconfiguring the transparent proxy NAT rule.

EDIT: The anti-lockout rules can both be switched off by flipping System: Settings: Administration: "Disable webConfigurator anti-lockout rule".

[silent_mastodon]

No luck I'm afraid.

I tried both moving the webgui port (to 8443) and adjusting the NAT port forward to match, and simply turning off the anti-lockout rule, and still don't have access.

[Julien]

Just to Clarify, what is between the Internet and the OPnsense VM Firewall ? ISP Router ?

[silent_mastodon]

No, another opnsense router, installed on baremetal (an old PC). It has the same rule I'm attempting to make work on this VM, and it worked without me having to twiddle with anything at all. That's why this is so confusing and irritating.

The only differences are the WAN interface on the VM being on an RFC1918 network, and that it is a VM.

There are no VLANs, nothing. It's just one router behind another router.

[Julien]

i beleive you are ona RJ45 internet connected to the OPNsense Firewall,
You have NAT the port 443 to the WAN IP of the VM OPNSENSE ?
Arent you using port 443 for the anti-lock ?

[franco]

I remembered these command required for Xen server now that you mention it ought to work...

nettool -K <vif name> tx off
nettool -K <xen bridge> tx off

Another useful tip was:

and the trick is name your vif in the xen config file so that it uses the same vif on reboot


Cheers,
Franco

[silent_mastodon]

Well, I tried disabling the various offload engines on the vif (the command is different in the xen console but the effect is the same) and it didn't do anything for my issue.

Unfortunately, at this point my project is so far behind that I'm just giving up and moving on with a different router software that I've used in this situation before. Thanks for the tips and such, wish it had worked.

[franco]

I completely understand. Thanks for trying!


Cheers,
Franco