Hi all, after upgrading to version 18.1, none of my port forwarding based on port aliases works. looking at /temp/rules.debug I just found following
......
UnifiPorts = "{ 8080 8443 8880 8843 27117 3478 }"
table <UNIFISERVER> persist
UNIFISERVER = "<UNIFISERVER>"
......
which seems ok, but
......
rdr on igb0 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan10 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan3 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan1003 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan1007 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan1005 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on ovpnc2 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on openvpn inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan8 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb0_vlan101 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan10 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan3 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan1003 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan1007 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb1_vlan1005 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on ovpnc2 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on openvpn inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
rdr on igb2_vlan8 inet proto {tcp udp} from {any} to {(self)} port $UnifiPorts -> $UNIFISERVER port 8080
......
the rules for redirecting just resolve to the first port in the alias
What can I do ? in the 17 branch everything worked .... Is there a fix ? I have lots of those definitions in my rules. O do I have to revert to the 17 branch ? If, how is it done ?
This way I rendered mi production setup unusable ;-(
Thanks a lot
Stefan
I can confirm. same issue here.
seems that the first value of an alias (when more then one) is always fetch by the rule.
The NAT rule is not using all values or definitions.
Tested here as well, and I can recreate the issue you are referring to.
Now I say that, to say this: I wasn't using Port aliases before, and was planning on doing so, but will most likely wait at this point.
My individual NAT Port Forward rules are working fine IF I'm not doing port aliases with them.
Port Aliases are working in my system using NAT. Still running 18.1.rc2.
So I have Binat rules as I have multiple external IP's. Create that and then create a WAN firewall rule any to my internal mail server with the ports that I have open 443,465 and 993 using a ports alias list and it works fine. I have just deleted the rules, checked that the ports were not accessible and then re-created them again, working fine.
Well, I think that's the difference, this is on the production release.. so something changed between that RC and the production release it appears..
I can check that but not until after 17:30 GMT, I cannot take opnsense down at the moment, I will get an ear bashing! :)
Hahahaha.. I hear you :)
Have one alias for web-traffic (ports 80 and 443).
Using that alias to forward traffic to respected servers (Mail and Cloud)
NAT rule uses ALWAYS the first parameter (it was 80). Means also HTTPS traffic was forwarded as port 80 traffic.
As i changed the sequence of parameter from 80,443 to 443,80, the traffic is now forwarded to 443, but not port 80 traffic.
Until V17 both parameter was handeled by the rule like expected 80->80 and 443->443
OK, I have just done a test with my 'TEST' unit, virgin 18.1
By default this unit had just the rules it comes with out of the box.
I created a port forward from the WAN to the LAN, forwarding ports 80,443,465 and 993, directing them at my laptop.
Then on my main machine I ran a piece of software called Hercules, if you do not know it then find it, it's a very useful tool, whatever, Hercules allows me to test by trying to connect on whatever port I specify to a given address.
I then ran wireshark on the laptop to see what was coming through the firewall.
I have to say, all the port forwards worked. I don't know what's going on with others, but on my test unit - perfect.
This was create the port alias(es) then create the port forward, apply.
Note, before someone says, I had to turn of the block private networks as the WAN network was 192.168.1.0
and the LAN network was 192.168.3.0 - but it worked.
Probably it has to do with "translated NAT rules" from V17 -> V18 during update?
If yes, it can be the reason, that a fresh installation works and an updated installation not.
More than likely, I had issues when bouncing 17.7.11 to 18.1.rc1 & rc2.
I've drawn the conclusion that when doing a major upgrade for me it's best to do the configuration from scratch anyway. In the past, with pfS**** I've also had issues so now I just bite the bullet and get on with it.
I know it should not happen, but these things do, and I end up with less down time in the long run.
I got bitten by the ICMP - ICMP6 issue too, so there was a lot of swearing going on.
Well actually the upgrade did not work for me .... So I reinstalled and imported the config .... but sill the same with the aliases
Try doing it with a fresh install and MANUALLY create the config. I know it's a PITA but see if that works, obviously something is not right, but I suspect its in converting the config.
Quote from: frank_p on January 30, 2018, 05:08:05 PM
Probably it has to do with "translated NAT rules" from V17 -> V18 during update?
If yes, it can be the reason, that a fresh installation works and an updated installation not.
Then the question becomes this: Do we have to delete all NAT-related rules, even for Outbound bypass rules for say, a VPN, then re-enter from scratch?
Sadly there is only one way to prove it. My NAT and Aliases work on a fresh install.
I don't have time at this minute, but in a while I will compare my old 17.7.11 config to my 18.1.rc2 and see if there are any differences, I will also bounce my live firewall to 18.1 and see what happens... but as I said earlier, that needs to be after 17:30 GMT.
Well, thank you so far for the troubleshooting. I may pull one of my backup configs from before the upgrade and see what might be different as well.
So I've just bounced my 18.1.rc2 to 18.1_1 and all is working.
I just installed an older version and checked the ruleset for a similar situation, it looks like the old version dropped the target port when a destination port alias was provided.
I'm not 100% sure this is intended behaviour for pf, but let me prepare a fix which does the same.
Strange, because I tested this in a setup that did not use port aliases.
I added the port aliases, then redid the port forward rules, same thing happened, but I look forward to the patch to see if this fixes it.
Ok, here's the patch
https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c (https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c)
installable using:
opnsense-patch 57f51d2943
Which in my test setup seems to deliver the same rule output.
I've applied the patch.. .not sure what I need to look for.. because now I'm thinking it was working before, but meh.. I'm still testing. THank for the patch though!
Well it does look like that Port aliases are working now, at least on my side. Just tested by adding a new rule that included them all, then shut down each forwarded port in other rules one by one and tested. .. seems to have worked, but will wait for other confirmations before I move over to it as a permanent solution.
thanks ad!!
I also installed an older version, got my opnsense working again and waiting for more confirmation before I switch again. Thanks a lot!
Stefan
Looks like port-forwarding is working again :) on installation came from V17 and upgrade to V18
Thanks for the patch !!
Thank you for confirming. We will discuss releasing another hotfix for this and let you know soon. :)
Cheers,
Franco
Hello
I noticed GeoIP Alias isn't working after upgrading to 18.1_1 and tried applying the hotfix, sadly it didn't helped.
I then tried with source any, which seemed to help, but after some time I am unable to connect again (OpenVPN in this case).
Sadly I can't provide any logs at the moment, because I'm not at home and I don't have a working VPN ;).
Regards
Hi all,
patch was successful for us also but we had to "clone" the old rules and delete the original rules (after applying the patch and "reloading all services" on the console).
We had also a side effect of this bug: an old (useless) inbound NAT VOIP rule (using a port alias with SIP and some media ports) that prevented all OUTBOUND SIP connections (which was very surprising to us). After patching & cloning this side effect disappeared as well.
Robert wanted me to precise this in case this is useful to anybody.
Raynald
The patch did not work here.
Quote from: AdSchellevis on January 30, 2018, 08:51:24 PM
Ok, here's the patch
https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c (https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c)
installable using:
opnsense-patch 57f51d2943
Which in my test setup seems to deliver the same rule output.
Quote from: hirschferkel on January 31, 2018, 12:54:08 PM
The patch did not work here.
Quote from: AdSchellevis on January 30, 2018, 08:51:24 PM
Ok, here's the patch
https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c (https://github.com/opnsense/core/commit/57f51d2943d964032770574605397006616e935c)
installable using:
opnsense-patch 57f51d2943
Which in my test setup seems to deliver the same rule output.
Same here. I was having some oddities, not related to the patch or aliases, but outbound policy nat rules were messing up.
Just went through this morning and cleaned the entire firewall up and now it appears to be behaving... fingers crossed.
My forwarding rules are just > take all incoming connections on a range of ports to one destination and it's corresponding ports.
Host is defined as an Alias (but that's not the problem).
Port range is defined as another Alias.
But what I found is, that old imported rules can not be edited!
On the other hand I can edit a new rule, but this one will not be available with NAT port forwarding!
Something has gone quite wrong here...
At the moment it only works if I choose "pass" as an option, in a manual, single port forwarding. But I can't select new rules which are set to pass. I guess old rules loose their definition, as they can not be edited either. So in the end I cannot set a portrange to be passed... that's wired.
I believe this is a hold over from then first jump to the 17.x series.
There was a change in how the rule was defined and some other options that were added/removed from the tabs/pages.
I had a similar issue originally, but spoke with franco and he suggested I recreate the rules and get rid of old ones. Once I did that, things were back to normal.
Quote from: hirschferkel on January 31, 2018, 03:29:31 PM
My forwarding rules are just > take all incoming connections on a range of ports to one destination and it's corresponding ports.
Host is defined as an Alias (but that's not the problem).
Port range is defined as another Alias.
But what I found is, that old imported rules can not be edited!
On the other hand I can edit a new rule, but this one will not be available with NAT port forwarding!
Something has gone quite wrong here...
At the moment it only works if I choose "pass" as an option, in a manual, single port forwarding. But I can't select new rules which are set to pass. I guess old rules loose their definition, as they can not be edited either. So in the end I cannot set a portrange to be passed... that's wired.
Is it possible that you configuered the old rules in the NAT > Port Forward menu? They should be editable there, they are only visible in the rules if you choosed "create new rule"
AFAIK you can create rules only in the section Firewall > Rules > choose Interface > edit rules.
I created the old rules there, and the new ones.
The old ones stay not editable.
The new rules will not be available in Firewall > NAT > Port forward > edit forwarding rule > Filter rule association
> The old ones stay not editable.
>
> The new rules will not be available in Firewall > NAT > Port forward > edit forwarding rule > Filter rule association
I think that's how the association always worked, no? Non-editable if auto-created via association or manually selectable if not.
Cheers,
Franco
18.1_1 working well. Nice one guys.👍
Hi Franco,
you're right, I missed that.
So if I autocreate a port forwarding, it will not work!
If I setup a rule one manually, it will not be available for a new port forwarding. So it won't work, anyway at the moment?
Ah ok, that sounds like a viable theory. The problem with the auto-created association rules is that they are not real rules so their edit button was removed to prevent further breakage. Ideally, they shouldn't exist in a state that an user should feel the need to edit, but may therefore be in a twilight state that the new alias system cannot cope with yet. We'll take a closer look.
Thank you,
Franco
18.1 update also killed my NAT. Patch fixed it for me.
18.1.1 has been prepared and is ready for release tomorrow morning.
Cheers,
Franco
Hopefully 18.1.1 will fix the Alias problem.
18.1 breaks a lot of things for me; All Aliases not working, NAT-Patch not working, IPS Rule Updates not working...
Quote from: Phobus on February 01, 2018, 07:04:10 PM
Hopefully 18.1.1 will fix the Alias problem.
18.1 breaks a lot of things for me; All Aliases not working, NAT-Patch not working, IPS Rule Updates not working...
i have the same problems. on 17.7.12 everything was working fine.
Even a full reinstall didn´t help!
Quote from: Phobus on February 01, 2018, 07:04:10 PM
Hopefully 18.1.1 will fix the Alias problem.
18.1 breaks a lot of things for me; All Aliases not working, NAT-Patch not working, IPS Rule Updates not working...
Aliases are working for me, though i have a cron job to update them.
Btw, @Franco, using the 'Aliases Resolve Interval' from Firewall: Settings: Advanced is indeed broken.
IPS rules / updates have a patch which fixed the issue.
With the fix, port aliases are working, but GeoIP alias (still) isn't.
Quote from: Evil_Sense on February 01, 2018, 08:05:55 PM
With the fix, port aliases are working, but GeoIP alias isn't.
Gesendet von meinem ONEPLUS A5000 mit Tapatalk
It's all a bit strange. My geo aliases and all others are working fine... I must have done something wrong . ???
Maybe we should separate "not working" into two categories:
(a) Firewall: Diagnostics: pfTables -- alias empty
(b) generally not working in NAT or firewall rule
Then also check (b) under Firewall: Diagnostics: pfInfo (Rules) whether these non-working rules actually see traffic
Thanks,
Franco
Quote from: franco on February 01, 2018, 11:51:13 PM
Maybe we should separate "not working" into two categories:
(a) Firewall: Diagnostics: pfTables -- alias empty
(b) generally not working in NAT or firewall rule
Then also check (b) under Firewall: Diagnostics: pfInfo (Rules) whether these non-working rules actually see traffic
Thanks,
Franco
Got it, my GeoIP alias falls under (a), the pfTable is empty and therefore there's nothing to compare to, since I'm using it as source nothing passes :)
Okay, that's good and bad... Good in the sense it's not a fundamental firewall issue, bad because whatever prevents your system from fetching the aliases may prevent it from reaching out in the first place... Is that table populated when you run this from the console?
# configctl filter refresh_aliases
Quote from: franco on February 02, 2018, 09:13:53 AM
Okay, that's good and bad... Good in the sense it's not a fundamental firewall issue, bad because whatever prevents your system from fetching the aliases may prevent it from reaching out in the first place... Is that table populated when you run this from the console?
# configctl filter refresh_aliases
The command only returns 'OK'.
Sure, now check the table...
Quote from: franco on February 02, 2018, 09:34:34 AM
Sure, now check the table...
Seems it's still empty :(
What does this return then?
# ls -lah /var/db/aliastables/
Quote from: franco on February 02, 2018, 09:38:01 AM
What does this return then?
# ls -lah /var/db/aliastables/
CH is my GeoIP alias, and it's empty, NAS contains the address I configured.(https://uploads.tapatalk-cdn.com/20180202/c666730ddf36d35618f39169288c8698.jpg)
Are you using the CH alias in a floating rule?
Quote from: franco on February 02, 2018, 09:57:37 AM
Are you using the CH alias in a floating rule?
No, only in WAN rules, but currently it's removed from them because I tried to recreate the alias at the time displayed by the ls command.
So you can't fetch the GeoIP alias even though it's not used?
We can try to increase the pressure:
# rm /var/db/aliastables/CH*
# configctl filter refresh_aliases
Still empty?
Quote from: franco on February 01, 2018, 11:51:13 PM
Maybe we should separate "not working" into two categories:
(a) Firewall: Diagnostics: pfTables -- alias empty
(b) generally not working in NAT or firewall rule
Then also check (b) under Firewall: Diagnostics: pfInfo (Rules) whether these non-working rules actually see traffic
Thanks,
Franco
For me (Alias problem):
(a) Firewall: Diagnostics: pfTables -- alias empty
# configctl filter refresh_aliases
Still empty
# rm /var/db/aliastables/EBL*
# configctl filter refresh_aliases
Still empty
Output: Error (1)
Strange Output now files and Aliases are missing:
root@*****:~ # ls -lah /var/db/aliastables/
total 12
drwxr-x--- 2 root wheel 512B Feb 2 10:29 .
drwxr-xr-x 18 root wheel 1.0K Feb 2 08:36 ..
-rw-r----- 1 root wheel 257B Feb 2 10:29 EBL.self.txt
Quote from: franco on February 02, 2018, 10:03:08 AM
So you can't fetch the GeoIP alias even though it's not used?
We can try to increase the pressure:
# rm /var/db/aliastables/CH*
# configctl filter refresh_aliases
Still empty?
Sadly yes, the three files are created but the txt file is still empty.
Under Firewall: Settings: Advanced, is " Verify HTTPS certificates when downloading alias URLs" checked or unchecked? Are you using a proxy server in your network doing HTTPS MITM?
Cheers,
Franco
Quote from: franco on February 02, 2018, 10:37:29 AM
Under Firewall: Settings: Advanced, is " Verify HTTPS certificates when downloading alias URLs" checked or unchecked? Are you using a proxy server in your network doing HTTPS MITM?
Cheers,
Franco
Setting is unchecked and I'm not using a proxy server who intercepts https..
QuoteUnder Firewall: Settings: Advanced, is " Verify HTTPS certificates when downloading alias URLs" checked or unchecked? Are you using a proxy server in your network doing HTTPS MITM?
In my Situation also:
Setting is unchecked and I'm not using a proxy server who intercepts https..
After the update to 18.1.1 "IDS rule update problem" seems to be solved.
Unfortunately Alias problem still exist - aliases aren't working e.g. hosts :(
Same outputs as posted before...
NAT / Portforwarding is even in 18.1.1 not working correctly.
only if i disable port forwarding rule to proxy 127.0.0.1 (https) and disable blocking https rule, i get a 100% working connection!
On 17.7.12 these all works without any problems.
my no ssl bump list is the same as in version 17.7.12.
OK I've found the "bug" with aliases (hosts) not working.
In my case I've a alias list with hosts they are used from MS for data collection.
One of them can't be resolved anymore so this entry should be skipped (in my opinion), but in that case it ended up with an error -> table generation (all) will be aborted -> aliases in that case will not work.
One deceased entry in an alias list is enough to stop the whole table generation :o
This behavior should be changed to skip such entries.
After Updating to 18.1.1 it runs again. Obviously there were some more issues.
Tried reinstalling with 18.1 and updated to 18.1.2_2.
Still the same issue, geoip alias is empty..
Executed refresh_aliases and also deleted the tables and retried, still empty..
Same on test VM.
Another patch to try... https://github.com/opnsense/core/commit/b514992
# opnsense-patch b514992
Cheers,
Franco
Quote from: franco on February 13, 2018, 05:32:47 PM
Another patch to try... https://github.com/opnsense/core/commit/b514992
# opnsense-patch b514992
Cheers,
Franco
Applied patch, retried the previous steps, still empty :(
How are you populating the alias? Sorry if you mentioned it, but I don't want to dig through the entire thread.
Quote from: Dominian on February 13, 2018, 05:43:00 PM
How are you populating the alias? Sorry if you mentioned it, but I don't want to dig through the entire thread.
I open an alias and save it, apply the changes and check the pfTables result for the GeoIP alias.
Since it's still empty, I try these mentioned commands:
# rm /var/db/aliastables/CH*
# configctl filter refresh_aliases
And still empty.
What is your configuration on the Alias itself?
Can you post a screenshot of the config?
Quote from: Dominian on February 13, 2018, 05:53:58 PM
What is your configuration on the Alias itself?
Can you post a screenshot of the config?
Only one country checked, even tried another one, same result..
So, I just tested a brand new alias, using this: https://iplists.firehol.org/files/firehol_level1.netset
Set the alias to URL Table (IPs) set the expiration to 1 day 0 hours (So it will refresh daily) and submitted, pfTables immediately shows them.
I've attached what the alias config looks like. Can you screenshot YOUR alias similar to how I did mine so I can see what you're doing exactly.
@Evil_Sense it's not for populating aliases, it's for repairing alias usage in the outbound rules. run this before retesting:
# configctl filter reload
Cheers,
Franco
Quote from: Dominian on February 13, 2018, 06:03:19 PM
So, I just tested a brand new alias, using this: https://iplists.firehol.org/files/firehol_level1.netset
Set the alias to URL Table (IPs) set the expiration to 1 day 0 hours (So it will refresh daily) and submitted, pfTables immediately shows them.
I've attached what the alias config looks like. Can you screenshot YOUR alias similar to how I did mine so I can see what you're doing exactly.
That's how I configured the GeoIP alias..
Edit: IP and port aliases are working and also populated(https://uploads.tapatalk-cdn.com/20180213/930c0100bab5f692e5d06bb793fade77.jpg)
Configured mine the same way, submitted, pfTables shows it is populated.
Quote from: Dominian on February 13, 2018, 06:53:03 PM
Configured mine the same way, submitted, pfTables shows it is populated.
Hmm.. Seems it doesn't get populated if the name is the same as the value
Edit: Deleting aliases also doesn't remove the tables..
That KIND of makes sense. I set the name to GeoIP and just selected switzterland.. I didn't even think about the naming just as yours is in the interface, my bad.
Quote from: Dominian on February 13, 2018, 07:08:53 PM
That KIND of makes sense. I set the name to GeoIP and just selected switzterland.. I didn't even think about the naming just as yours is in the interface, my bad.
Thanks for testing and pointing me in the right direction :)
Did you get it working now?
Quote from: Dominian on February 13, 2018, 07:42:14 PM
Did you get it working now?
Yes, I changed the name and it's working perfectly fine :)
Also tested with other countries, it's true, if the name matches the value, nothing gets populated.
Now pfTables is showing some non existent tables, even after the refresh_aliases and reload command, but that doesn't bother me much ^^
Ok, mystery almost solved. Added a follow-up for this https://github.com/opnsense/core/issues/2199 to discuss what to clean up to prevent users running into it.
Thanks!
Franco