Messages

Thanks again, that fixed the problem here too.

Ben

Great! I will go and test that too after the weekend (when I'm physically at the firewall  :) ).

Ben

Perhaps it's better to create an issue about this on github?

These look related:
https://github.com/opnsense/core/issues/5477
https://github.com/opnsense/core/pull/5925

BTW, I tried to opnsense-revert to 22.7.2, but that got nowhere fast. UI would not start up I should reinstall this one and start using boot environments...

Ben

Upgraded to 27.2.4, problem persists.

Ben

I am having the same problem. Every host override (A and Alias records) I have in the unbound config now fails the reverse lookup. Same error messages: "...PTR record already exists for...".

I haven't added all these hosts/aliases multiple times, so why does it throw this error message?

I would appreciate some guidance on how to fix this.

Thanks,
Ben

I updated to 21.7.3 yesterday. Same here, suricata and unbound were killed with lots of out-of-swapspace messsages. I could manually restart them.

Will try the reboot now.

Ben

All solved now, I can put in a pull request on GitHub if you are interested in adding the ISP TransIP to the list of supported ISP's.

I'm also interested in TransIP ddns support. I did not see a pull request yet. Is your code available somewhere?

Thanks,
Ben

back to 16.7.10 it's the same (all categories active) plus a error in the log:

configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {b62421f1-b3be-4e2c-b502-366d1a140aa0}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall('%s\n' % result) File "/usr/local/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe

That's probably the same issue I have. The timeout for the GUI is set to 120 s, but the ACL script isn't finished by then.

Ben

Hi Ad,

I understand the problem. This also explains some odd behaviour I have seen in the past when clicking on 'Apply' and the list not sticking correctly. Perhaps the same problem as pr3p described in his latest message? For now the workaround could be to Download the list, and watch the cpu meter until all is quiet. And then hit Apply.

Increasing the timeout would help in the short run, but it is also highly dependant on the hardware of course. And what if some list suddenly takes even longer? Making it async would be best, but is probably a lot more work?

While we're on this subject: I also noticed that OPNsense currently blacklists every address in that list. But some categories are explicitly whitelisted, see for example liste_blanche. I currently have to delete that index in the GUI.

Kind regards,
Ben

Hi Ad,

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

Code Select Expand

/usr/local/opnsense/scripts/proxy/fetchACLs.py


Indeed, it looks like processing the list is taking an awful long time:

Code Select Expand

# time curl -C - -O 'ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8373k  100 8373k    0     0  1284k      0  0:00:06  0:00:06 --:--:-- 1619k
0.107u 0.071s 0:06.57 2.5% 148+120k 0+65io 0pf+0w

# time /usr/local/opnsense/scripts/proxy/fetchACLs.py
168.865u 12.138s 3:09.56 95.4% 7+167k 0+748io 12pf+0w

This is on an OPNsense A10 Quad Core SSD rack system.

Kind regards,
Ben

Hi Ad,

Yes, that line is in there. I have restarted squid and am now unable to reproduce the bungled message. The timeout is still there tough. Download ACLs gives this:

Code Select Expand

Nov 23 13:27:15 OPNsense configd.py: [8ed9d971-89dc-4d69-bb59-c99578afaccb] request proxy status
Nov 23 13:27:31 OPNsense configd.py: [776d1217-5e8f-4f66-8e3d-8aca0b8c8744] generate template OPNsense/Proxy
Nov 23 13:27:32 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:27:34 OPNsense configd.py: [c7a06f6b-5253-4251-af1a-6740ef916ed5] download proxy ACLs from remote locations
Nov 23 13:29:36 OPNsense configd[18360]: Timeout (120) executing : proxy downloadacls

The Web UI has at that point returned to normal (no spinning indicator). At that point a Python process is still chewing up 100% cpu (fetchACLs I think). When that's done I get:

Code Select Expand

Nov 23 13:30:39 OPNsense configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {c7a06f6b-5253-4251-af1a-6740ef916ed5}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall('%s\n' % result)   File "/usr/local/lib/python2.7/socket.py", line 228, in meth     return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe

And then I hit Apply:

Code Select Expand

Nov 23 13:33:07 OPNsense configd.py: [61d38b33-64d8-410d-86a4-dd8f13397041] request proxy status
Nov 23 13:33:07 OPNsense configd.py: [18e041fb-8f75-41b1-af92-f5b6f8c2563c] generate template OPNsense/Proxy
Nov 23 13:33:08 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:33:10 OPNsense configd.py: [63646e00-3382-4624-89c9-dfcc8f63fbd6] reconfigure proxy

Perhaps the bungled message was because I hit apply before the Python process actually ended.

Kind regards,
Ben

Oops, it seems there is another problem after applying the download:

Code Select Expand

Nov 22 18:48:52 OPNsense configd.py: [10b51670-e81f-426e-8a60-ebd7eaa3192a] request proxy status
Nov 22 18:48:52 OPNsense configd.py: [9f07c783-7099-4f49-87c1-b7fc14f9a298] generate template OPNsense/Proxy
Nov 22 18:48:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:48:55 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] reconfigure proxy
Nov 22 18:48:59 OPNsense squid: Bungled /usr/local/etc/squid/squid.conf line 38: acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
Nov 22 18:48:59 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] returned exit status 1

Line 38 looks like this:

Code Select Expand

acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"

And that file actually exists:

Code Select Expand

root@OPNsense:/usr/local/etc/squid # ll acl/
total 27942
-rw-r-----  1 root  squid  28580995 Nov 22 18:43 UT1
-rw-r-----  1 root  squid      1444 Nov 22 18:42 UT1.index
-rw-r-----  1 root  squid       991 Nov 22 18:43 yoyoads
-rw-r-----  1 root  squid         2 Nov 22 18:43 yoyoads.index

Kind regards,
Ben

Hi Ad,

Thanks for the patch. Fetching the ftp data works again, but in system.log I now see a timeout:

Code Select Expand

Nov 22 18:40:52 OPNsense configd.py: [b2cf595d-8d13-43a5-869e-b33dddac1949] generate template OPNsense/Proxy
Nov 22 18:40:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:40:55 OPNsense configd.py: [73e319a1-7595-4240-be5d-c671820f6ab3] download and reload proxy ACLs from remote locations
Nov 22 18:42:57 OPNsense configd[6698]: Timeout (120) executing : proxy fetchacls

But the data is updated in /usr/local/etc/squid/acl, and I can see/choose the categories again in the web UI. So perhaps this is only a cosmetic problem.

Kind regards,
Ben

Ok, good to know I'm not the only one!

Ben

No one? Any tips on helping me debug this?

Thanks,
Ben