OPNsense Forum
Archive => 19.1 Legacy Series => Topic started by: gstuartj on April 12, 2019, 02:40:19 am
-
I'm having an odd and difficult to diagnose problem with both the latest production (OPNsense 19.1.6-amd64) and development branch releases.
- If I create a host alias the corresponding pf table does not populate with the addresses and rules do not work. This is worrying.
- Deleting alias does not delete corresponding pf tables. I have tried killing the tables manually as well as removing related /var/db/aliastables files. The pf tables continue to be recreated despite not existing as aliases or being referenced in any rules, even after reboot. The alias data is not found in config.xml, so I'm not sure where it persists.
- New aliases of any type do not seem to get populated. While my deleted geoip alias continues to be recreated, new geoip alias tables remain blank, breaking my rules.
To try to diagnose this, I have:
- Rebooted
- Tried killing tables, deleting aliases, recreating aliases, running update_tables.py, configctl filter refresh_aliases, etc.
- Reinstalled OPNSense from scratch, after which my existing config aliases work fine. New aliases continue not to work, newly deleted aliases continue to exist as pf tables.
- Checked logs for any useful info, of which there is none.
Nothing has seemed to make aliasing work correctly in the current versions. I cannot find other complaints about this problem. I have not had this issue until recently. What could be going on?
-
To create the pfTable entries you have to press the apply button.
(Firewall: Aliases -> Apply)
-
Yeah, I'm aware of the apply button. Does anyone have troubleshooting suggestions for this?
-
My hosts alias and geoip alias are working in 19.1.5_1.
Please check your "Firewall Maximum Table Entries" the default is 200000 which may be too small to hold all the alias IPs, especially if you use a lot of GeoIP Aliases or URL Table (IPs). I had the same issue and had to fix it by increasing the Firewall Maximum Table Entries value. I changed mine to 800000 but you will need to tweak it to what works for you.
Firewall -> Settings -> Advanced -> "Firewall Maximum Table Entries"
After increasing the Firewall Maximum table Entries, Update Bogons (Firewall -> Diagnostics -> pfTables -> "Update bogons" button) which should also update the geoips.
Then check your System Logs to see if the geoip and bogons were updated, or if you have a table-entries limit warnings. (System -> Log Files -> General). See attachment for example.
-
Update: Still having issues. (https://forum.opnsense.org/index.php?topic=12407.msg57065#msg57065)
Thank you, Steven, good suggestion. I've actually removed all of my geoIP and iplist aliases from my configuration and done another fresh install to remove persistent pfTables. So far new aliases are working fine, I'll see what happens when I recreate them. I upped my table entry limit on the new disk image from 400k to 800k.
I run OPNSense in a VM and still have my faulty disk image to investigate further. I did not see any log errors before about reaching my table entry limit, though admittedly I had several large tables.
I still find it odd that I could not kill the tables or aliasdb files corresponding to deleted aliases. They just kept coming back. Since I don't fully understand the backend mechanisms behind how those tables are built yet, I hadn't figured out how to get them to fully go away. This part seems like a possible bug, even if I was hitting table entry limits.
-
Update: More details. Does not seem to have to do with dynamic aliases. (https://forum.opnsense.org/index.php?topic=12407.msg57068#msg57068)
Still having issues. After removing all dynamic aliases, creating a fresh install, reloading my config, and creating a new static host alias successfully, I attempted to create a geoIP alias. The pf table for this alias stayed blank, new static host aliases no longer fill the corresponding pf table (blank), and pf tables are not killed upon deletion of the alias.
Something's very wrong. It's like creating any "dynamic" alias (geoip, url table, etc) poisons OPNsense's alias functionality for me.
Can someone try this on their system for me? Create a new geoip alias for a random region. See if the pf table is filled. Then try deleting it.
-
Interestingly, OPNSense also fails to download bogon info over IPv6. "Prefer IPv4 even when IPv6 is available" must be turned on.
-
Alright. Thanks to the magic of snapshots I can confirm that alias functionality works in 19.1.4 as expected, including alias deletion (and related pftable). However, immediately after upgrading to 19.1.6 all new alias functionality is broken. Creation of static host aliases fails to fill the pf table, deletion of aliases fails to remove the pf table (even after reboot), etc. No "dynamic" aliases have to exist for me to encounter this bug.
I am not familiar with the codebase. Can anyone think of any changes to aliasing or in the upgrade process in 19.1.6 that would cause this?
This is not a small problem so I'm rolling back to my 19.1.4 snapshot for now. I have the fresh 19.1.6 snapshot available to test, as well.
-
I think it's more than dynamic aliases - I have several port aliases setup that quit working also. I've been trying to figure out why, but keep getting distracted.
-
I think it's more than dynamic aliases - I have several port aliases setup that quit working also. I've been trying to figure out why, but keep getting distracted.
Thanks for confirming. Agreed, after retracing my steps and troubleshooting the issue (https://forum.opnsense.org/index.php?topic=12407.msg57068#msg57068), it does not seem to be related to dynamic aliases. In my case almost all alias functionality is broken by upgrading to the 9.1.6 release, except aliases that were created before the upgrade.
I'll create a github issue in a bit. Hopefully we can get some dev eyes on this, as it has potential to break firewall rules in production, if it's a bug and not an obscure issue with my configuration.
-
Something's very wrong. It's like creating any "dynamic" alias (geoip, url table, etc) poisons OPNsense's alias functionality for me.
Can someone try this on their system for me? Create a new geoip alias for a random region. See if the pf table is filled. Then try deleting it.
I am able to create geoip alias, see them in the pfTable, and then delete them without issue. The pfTable will still have them listed until I reboot.
I'm on OPNsense 19.5.1_1 though, I haven't upgraded to 19.1.6 yet.
-
Okay, I was looking before at the General logs when I (obviously) should have been looking at the Backend logs. In 9.1.6 when creating any new alias I get the following error:
Apr 12 22:03:40 configd.py: [d15a2999-00f5-4a54-9026-5dbec36e63b0] refresh url table aliases
Apr 12 22:03:39 configd.py: [bbf98ba9-1d60-4a76-ad76-59ecd919e3c4] Reloading filter
Apr 12 22:03:38 configd.py: [ece01db5-8606-4134-b5dd-8d01e6d6496f] Inline action failed with OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 509, in execute return ph_inline_actions.execute(self, inline_act_parameters) File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute filenames = tmpl.generate(parameters) File "/usr/local/opnsense/service/modules/template.py", line 332, in generate raise render_exception Exception: OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long
Apr 12 22:03:38 configd.py: generate template container OPNsense/Filter
Apr 12 22:03:38 configd.py: [ece01db5-8606-4134-b5dd-8d01e6d6496f] generate template OPNsense/Filter
I also found issue #3399 on GitHub (https://github.com/opnsense/core/issues/3399) which may describe the problem.
-
Oh no, I just upgraded to 19.1.6 and have the same problem! :(
I created a new alias and tried to use it in a new firewall rule, but the rule does not work.
when creating a rule, I get the following message in Backend log:
configd.py: [4b57c046-1239-4414-975f-c686fb8fd54f] Inline action failed with OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 509, in execute return ph_inline_actions.execute(self, inline_act_parameters) File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute filenames = tmpl.generate(parameters) File "/usr/local/opnsense/service/modules/template.py", line 332, in generate raise render_exception Exception: OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long
Update:
I was able to get aliases working again by removing all three "|encode_idna" in file /usr/local/opnsense/service/templates/OPNsense/Filter/filter_tables.conf for now.
-
Update:
I was able to get aliases working again by removing all three "|encode_idna" in file /usr/local/opnsense/service/templates/OPNsense/Filter/filter_tables.conf for now.
Good tip. You may also be able to use opnsense-patch to roll back to before the idna patch a couple months ago. I did not have any luck patching forward to the latest commit. I just rolled my snapshot back to 9.1.4 until I have more time to look at it or until the devs release an official hotfix/update.
-
Same issue. Old Alias are working, new ones are emtpy.
Error message
configd.py: [5c57e160-70ec-44c2-bee6-c84bd10f3acf] Inline action failed with OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 509, in execute return ph_inline_actions.execute(self, inline_act_parameters) File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute filenames = tmpl.generate(parameters) File "/usr/local/opnsense/service/modules/template.py", line 332, in generate raise render_exception Exception: OPNsense/Filter OPNsense/Filter/filter_tables.conf label empty or too long
br
-
Someone may want to open an issue on github and link this thread if they have not already.
-
I'm having similar issues with aliases with 19.1.6. I've added a CIDR to an existing alias that only has about 28 other CIDRs to in order to block continuous SPAM from the new CIDR I'm adding and it as if I didn't even add it. I keep getting SPAM from an address in the CIDR block that I added to the existing alias. CIDR's added to the alias prior to the 19.1.6 upgrade are blocked as expected.
-
Someone may want to open an issue on github and link this thread if they have not already.
https://github.com/opnsense/core/issues/3399
https://github.com/opnsense/core/commit/00b46e05752ed5e8e98b0256ce34070ea71dfb17
patch locally, using (on >= 19.1.4):
opnsense-patch ea2f217cf
-
A am running 19.1.6.
After the patch the symptoms still exist.
I also ran this patch,
opnsense-patch 50c25ea
from https://github.com/opnsense/core/issues/3214 (https://github.com/opnsense/core/issues/3214)
This fixed the issue.
-
Someone may want to open an issue on github and link this thread if they have not already.
https://github.com/opnsense/core/issues/3399
https://github.com/opnsense/core/commit/00b46e05752ed5e8e98b0256ce34070ea71dfb17
patch locally, using (on >= 19.1.4):
opnsense-patch ea2f217cf
If I apply patch, will I need to do anything special before next upgrade? e.g. remove patch before doing upgrade to 19.1.7
-
Hi,
Aliasing is totally broken here too on 19.1.6, even with both patches applied. Older aliases are still present in as pf tables while newer ones are blank. Am I the only one ?
-
Am I the only one ?
Definitely not, same issue here.
New Aliases, no matter if created manually or downloaded, are empty. For old ones, entries in pftables available.
Br
-
For what it is worth I had so many annoying issues (this was one) that I decided this morning to wipe my box out and start from scratch. Not only was the aliasing broken but the event reports had the wrong names - almost like the rules and names were out of sync. DNS also stopped working, neither DNSMasq nor Unbound would resolve anything.
Seems to be working OK now but it's taken all darn day to put everything back in (I didn't restore from a backup to be sure I didn't re-introduce the issues).
I think this is something to do with the upgrading process - which I have done since I can remember - this is only the second time I've done a clean install. I haven't the time spare to go figuring out what was broken but clearly things were, finding out what is made more difficult by the lack of a command prompt and file functions via the GUI.
-
Hi,
I've applied the patches in the following order and now it seems to work:
opnsense-patch 50c25ea
reboot
opnsense-patch ea2f217cf
reboot
br
-
thankyou, this worked (on 2 x boxes so far):
opnsense-patch 50c25ea
reboot
opnsense-patch ea2f217cf
reboot
-
finding out what is made more difficult by the lack of a command prompt and file functions via the GUI.
I understand where you're coming from, the 19.1.6 upgrade was pretty frustrating for me, too. That said, the command prompt is always available via SSH and the filesystem is exposed via SFTP, provided you turn on SSH administration. I don't think they should expose that via the web UI as it would involve significant development overhead with very little benefit over the existing native services.
For what it's worth, I also reinstalled from scratch at one point to wipe out some lingering bugs, but generally speaking the config file is abstracted away enough from the rest of the system that you can usually restore it without fear of bringing back the odd glitch. In this case, for example, restored aliases worked fine.
-
This is still goofed up for me. I'm not sure that I should have applied these patches to 19.1.8 but my aliasing wasn't updating in the same manner reported by others. After applying the patches it's still not updating, but also attempting to rectify the problem by using the "Quick add address" function via the pfTables /alias_util/ replaces 100% of the other addresses listed within the particular entry at /alias/. Described otherwise:
/alias/ entry for "poo" contains:
poo.google.com
poo.amazon.com
poo.rundmc.com
pfTables tool at /alias_util/ reflects IP addresses for the above.
Adding "100.100.100.100" using pfTables tool at /alias_util/ results in:
- "poo" entry at /alias/ containing only 100.100.100.100 (all others erased)
- pfTables tool at /alias_util/ reflects IP addresses for all previous entries + 100.100.100.100
-
19.1.8 should be good without any modifications, tried it on my end (again) with the following alias:
name: test
type: Host(s)
Content: 1.1.1.1 www.nu.nl www.google.com
Applied the changes, checked the content in pfTables (/ui/firewall/alias_util/) and added a host (192.168.1.1), pfTables looks good, alias content contains "1.1.1.1 www.nu.nl www.google.com 192.168.1.1".
If you have steps we can reproduce, I'll gladly try the same steps on my end.
Please make sure you run an unmodified version of OPNsense (without previous patches applied), applying already installed patches might revert functionality.
To reinstall the core package:
pkg install -f opnsense
-
Applied the changes, checked the content in pfTables (/ui/firewall/alias_util/) and added a host (192.168.1.1), pfTables looks good, alias content contains "1.1.1.1 www.nu.nl www.google.com 192.168.1.1".
If you have steps we can reproduce, I'll gladly try the same steps on my end.
I was doing exactly as you described to encounter the issue and it's still occurring. I also attempted to flush the table and re-applying, as well as adding a specific IP rather than host name for dns resolution...neither populated the pfTable. :(
Please make sure you run an unmodified version of OPNsense (without previous patches applied), applying already installed patches might revert functionality.
To reinstall the core package:
pkg install -f opnsense
This is something I've never tried. How confident can I be in retention of settings? I've spent hours tweaking this installation and I'd hate to lose (much less remember) the countless configuration variables.
-
The configuration doesn't change on package reinstall, but you can always create a backup of your config.xml first if you're in doubt.
If you can't reproduce my test (same items, same steps), it looks like an issue specific to your machine. Please make sure to check the contents of the test alias right after apply, to validate if your starting point is as expected.
-
I can reproduce successfully like Ad wrote, add new alias test, paste "1.1.1.1,www.nu.nl,www.google.com" and checked pfTables via Diagnostics:
1.1.1.1
172.217.16.132
2a00:1450:4001:815::2004
52.84.163.119
52.84.163.159
52.84.163.194
52.84.163.7
Latest 19.1.8 without any modifications
-
The configuration doesn't change on package reinstall, but you can always create a backup of your config.xml first if you're in doubt.
Seems it was definitely something wonky with the opnsense pkg. Aliases are back to working already without even rebooting after reinstallation. Thanks AdSchellevis!!