Messages

In a new thread, @dseven pointed out that "Firewall > Settings > Advanced > Disable reply-to" is the way to get the replies directly to the originating host (versus the GW). Details here.
It's policy-based routing related as well: Policy based routing

This does seem to work as well

If I unplug my wire into the vlan and have multiple hops into the wan interface it works.

If I disable that reply-to and source my traffic from the wan subnet it also works.


I experienced all kinds of bugginess this morning with the hardware/software. I had left the lan cable that was running to the device unplugged ( ran off with my laptop). Upon plugging the wire back in the link would not come up, Accessing LAN or overview in the gui just resulted in forever spins. I tried to reboot and after 5 mins the pings to it never dropped off. Had to run in there and hard power it off. IDK what it was doing or attempting to do but something was clearly in a dysfunctional state. After flipping these 2 options a couple of times :'Force gateway on WAN' and 'Disable reply-to on WAN GW' the firewall seemed to behave in all kinds of random ways until I went in and reloaded the packet filter and  reset the state tables.

But this particular issue of 'why is nat from wan subnet broken unless pass', and the subissue of 'why isn't my routing behaving like a standard router should' seems solved

[but it shouldnt need to be forced either if the system routing table is sane.]

Yeah if you did it from another vlan it would have had to go through the gateway for inter-vlan routing to workso there's your extra hop. I was coming directly at it from an access port in the same vlan and ip space as the wan interface.

>On the other hand, I plain fail to get my simple 80 port forward test to work when initiated from 10.3.3.3 to 10.3.3.2 (PF to 192.168.1.100).

This is what i was seeing when not source from a network beyond the WAN subnet. I think the ACKS for the SYNs are ending up in the wrong place the gateway has no idea why it's recieving them.

>The backup OPN is all blue/green.

This was also what I was seeing in the firewall logs, which was even more confusing when I would turn the firewall completely off and it would start working

 
I know this setting is trying to be helpful and can make sense for most ISPS, but it shouldnt need to be forced either if the system routing table is sane.. It seems like a sledgehammer approach to a non-issue, unless im just ignorant of it's use case. Maybe it's this mechanism that toggles between wans for multiwan.  'automagical intransparent' features as he called them make it very difficult to try to sort things out when you're coming at the problem in a bottom up principles first fashion because things are interfering in ways you dont know. Maybe a disable unless multi-wan active would be appropriate here


The way I understand it is basically every egress on WAN is redirected at that forced gateway, which would normally just be your isp's gateway on your ip block. Which is fine, because thats the way things came in anyways, stateful firewalls will know how to deal with it.  If you treat it like any other interface and any other network, attempting to lab it in the most straightforward way possible before you put it on the real wan..... you'll run face first into what i ran into

I noticed something similar the other day and re-added the widget

I know these a tier-1 type answers but

I think you can re-add the widget by editing the dashbaord

The interfaces on the overview may need the table expanded as well or including downed interfaces

[Thank you for your help.]

What a journey. This seems to work as it should now, with the setting enabled after changing the way I was coming at it. 

I just pulled my cord into the 99.99.99.0 space and setup rules in our actual firewall to allow flow between my main workstation ip and vlan 999's network and with that extra layer the 99.99.99.99 gateway got it to me.

It ended up being an automagical redirecter setting in the firewall/advanced section. Making the scenario simpler when troubleshooting is what bit me.

Gateway was probably very confused why it was getting packets with responses to things it never asked.

Thank you for your help.

>In a lab with WAN connected via Ethernet and some default gateway, if your PC from which you try SSH is connected to the same network, you need to disable the gateway enforcement on WAN. OPNsense will send the reply packets to the GW instead of your PC otherwise.

If this mattered pings wouldnt work?

I just tried this and could ssh to it.

I don't understand what this is doing, is this dnat or d rdr?

Its all on 99.99.99.0/24 and the gateway is 99.99.99.99

This also explains why in my virtual lab it worked, the wan was dhcp'ing in KVM/QEMU's natted subnetwork. The networks were not flat between my host and the wan interface


>because I don't like "magic intransparent features"

This very thing may have just cost me about 20 hours of tinkering

I'm going to try to re-image this hardware from scratch. Something is very wrong with this in either the software or hardware.

I switched it to 24.7 community that I had gotten to work in my virtual lab last night and now the packet filter doesn't function properly with tcp traffic at all.

This reminds me why I had the 33400 natted to internal as a 'pass' nat rule that Eric was asking about. I had figured out I needed to do it that way for it to work in the beginning and kept doing it.  I bet its been busted tcp filtering for wan the whole time and the 'pass' mode circumvents the firewall.

I shut off the packet filter with console cable.
Connected on ssh
did an allow in on wan 22
did an allow in on wan 33400
did an allow icmp on wan
re-enabled the packet filter
The icmp works.
All the traffic on wan on those tcp ports doesn't, even after reboot.

When I take these actions on my virtual opnsense in kvm/qemu it works properly. On the real hardware it doesn't.

The hardware is a dec 2700 opnsense brand firewall.

If a reimage from scratch doesn't get it, i'll see about getting the business support people involved and possibly RMA'ing this hardware.

Here's a picture of my WAN rules for allow, and a picture of my ssh session being very not allowed when turning the packet filter back on

https://imgur.com/a/L082gow

I ran a slightly different test (minimal changes to use 80 on WAN):

• Software Version is Business 24.7.10_2, virtualized within my internal network
• Factory Reset
• Assigned WAN to vtnet0, Assigned LAN to vtnet1 at the console (missed the prompt during boot...)
• Connected to Web GUI, ran through the wizard accepting defaults, reconnected to default IP
• Installed nginx on Ubuntu desktop connected to LAN (same machine used to access OPN)
• Disabled http --> https redirect in OPN, freeing 80 (I didn't bother moving HTTPS to alternate port)
• NAT rule TCP in on WAN Address port 80, redirected to the ip of the nginx instance on 80. Used associated rule
• Accessed http://wanip from my internal network (after I adjusted FW rule on my main OPN to allow that leg
• Welcome to nginx! page displayed

After disabling the redirect, I checked that the port was no longer used using 'sockstat | grep :80'.
I also checked that the anti-lockout rules no longer included 80 (not that it matters much because it's on LAN).
After creating the PF rule, I also verified the presence of the FW rule.

For the sake of it, I deleted the PF rule, recreated one with 9999 on WAN, also with associated rule.
I obviously had to add a rule on my main OPN too (main PC to spare OPN WAN leg), and success as well.
I re-enabled web GUI redirect and accessed from my phone. Still success.
While I was at it, I edited the rule from 9999 back to 80, and it still works (although the internal server listens on 80, it is ignored since the redirect rule takes over). Note that the WAN rule stays unchanged throughout this section, because it features the redirect target port 80 which remained unchanged.

I tried this at home also virtualized on community 24.7.11_2 and it worked properly. I wonder if something is off in Business 24.10.1

• NAT rule WAN 33400 to lan 33400, use pass so it works and you can work from the wider network

Why would you do that instead of creating a firewall rule from:*/* to:WAN address/33400, allow?

• NAT rule TCP in on WAN Address port 9999, out on the ip of the physically connected nginx instance on 80. Use associated rule

I expect this to work with an associated rule and I will take the time to recreate that setup. Tomorrow, probably.

I got in that habit because i was originally using the webgui as the test for this and moved to an actual external device to rule out any sort of weirdness with local interfaces.

also I forgot to put in the directions, you need to turn off the blocks for private networks on wan

• Factory Reset
• Software Version is Business 24.10.1
  
• Assign WAN to 1 interface DHCP
• Assign Lan to 1 interface static
• Physically connect device with nginx listening port 80 to lan and verify it can ping the gateway
• Move gui to 33400, disable http --> https redirect
• NAT rule WAN 33400 to lan 33400, use pass so it works and you can work from the wider network
• NAT rule TCP in on WAN Address port 9999, out on the ip of the physically connected nginx instance on 80. Use associated rule
• test it by going to http://wanip:9999
• No work
• Delete rule
• Recreate rule using pass
• Works

I dug deeper into this yesterday on the firewall log

For associated rule I see
ingest on wan
Ingest translated on wan
outgest on lan

For pass I see
 ingest on wan
outgest on lan

For state tracking using pfctl -s state
I see SYN:ESTABLISHED on associtated
I see ESTABLISHED:ESTABLISHED on pass

On wireshark, I see no acks for my syns on the tcp handshake. So I think something is off with the reverse flow back out of translation


I've got to run, I can be more thorough with details later on

You tinkered until you got it working but we still don't really know what went wrong initially.

You probably didn't have to stop listening on WAN. Port forward will have taken precedence (I have ssh on all interfaces).
If you want to port forward 80 and 443 on WAN, it's indeed wise to move the web GUI off of these (At least for SSH, it is unnecessary).
I have no clue how anti-lockout (likely on LAN) could get in the way of WAN port forwards. (here again, I had this in place for my ssh tests).

That success also didnt carry over when I replicated those steps at work. I factory reset it and went through the fairly minimal set of steps involved in creating a port forward and it just behaves the same each time. Works with pass, doesn't without

I even had my boss who has been using opnsense since it forked from pfsense attempt it and he couldn't get it to work either. We did manage to load a  config from a different firewall and edit preexisting entries/linked rules and see it work but creating new ones doesn't, so we know the hardware is good. This appears to be some issue with the generation and linking of these things

I found this searching for similar experiences and found this in a bug report thread about opnsense not generating reflective snat/dnat rules properly, which mirrors my experience exactly.

I don't know what to say, I've spent about 10 hours banging my head against the wall on making a portforward that doesn't involve 'pass'. It's really not that complicated or difficult a task, and I feel this is a genuine bug.



https://imgur.com/a/6zghwJK

So you come calling a bug a misconfiguration based on not knowing how the product works; starting the thread without sings of courtesy (not even a Hello), not thanking anyone but finding a way to have a side dig at someone who took time out to try to help you (one of the most knowledgeable networking gurus out there).
Nice.

One must undo several pre-configurations to NAT http and https without using the firewall bypass mode "pass"

Your easy test is to access the web GUI from WAN by redirecting to an internal proxy that pointed back at the web GUI on LAN?
I'm not really sure what you were testing but that seems convoluted.

A simple FW rule was sufficient in my case, as expected.

Testing NAT by using NAT to access the gui on wan

Really not that complicated a concept

The proxy those other 4 ports go yo is an nginx proxy for internal lan services. It has nothing at all to do with the web gui aside from using 80 and 443, which seem to have been part of the automagical mess.

Like i said turning off the wan listener, the redirects, and antilockout rule has NAT behaving like it should be now

33400 wouldnt work with just a fw allow on wan

33400 wouldnt work with nat + add associated rule

33400 would work with only nat pass. The same was true of 80 and 443. No work with associated rule, work with pass. Thats when I made this thread and the other guy tried to teach me how to make some rules ...

I have 4 other ports I am NATing to a proxy 80 443 5334 and 5335

The web gui was just my canary in the coal mine, and easy test.

Im my home virtual instance I just shut off the listener for the webgui on wan, and the webgui redirect rules, and disabled the antilockout rules and the nat + create associated rule function seems to work now. I think it was just making a speghetti mess with all the ports I was trying to use because they are all assosciated with webgui redirect and lockout things.... just automagically populated stuff in the way looking like a bug

The anti-lockout setting creates a FW rule (allowing 22, 80, 443 on self) AND a NAT port forward rule preventing redirection of these ports.
Only on one interface though, LAN or opt1 (or WAN as fallback if no other interface exists).
The easiest way to find which interface is used is to look at FW > NAT > Port Forward.

The ports are adjusted when the web GUI is moved to an alternate port.

I've played with SSH port forward recently (backup install) and since I like to doublecheck with the logs, I enabled FW rule creation (not pass) and logging.
Everything worked as intended, even with the anti-lockout rules which didn't get in the way because they are on LAN.

You're doing HTTPS port forwarding on WAN?

I'll have to double check when I get back into the office tomorrow, it's got to be some kind of tangledness going on with these 22 80 and 443 ports

The way I am accessing the GUI from wan is basically forwarding 33400 wan to 33400 lan with nat set to pass

It's not on the real internet though, it's just in its own VLAN while I'm provisioning it

Move the UI from port 443 to something different and disable the HTTP --> HTTPS redirection for the UI. These are mandatory if you want any public services on HTTP/HTTPS. Also - not entirely sure but I have it that way on all of my systems - disable the global "anti-lockout" function.

The admin guu is living on 33400. But there are automagical redirects and such related to http and https that may be interfering

I will try shutting off the redirects and disable the antilockout rules tommorrow and see if that's whats creating the difference in functionality we're seeing