Messages

Hi all

upgraded to 21.7.6 and all problems around buggy config of the transparant proxy seem to have disappeared. It works smoothly and with no issues. Also error pages in case of blocking by ACL in https work OK now.

I get a SSL_ERROR_RX_RECORD_TOO_LONG on https://quad9.net but this is the only site currently resulting in this error. I do not mind. And certainly is not related to the original problem.

problem & questions closed

Update:
opnsense 21.7.6.

I have exact the same issue, with the difference that DNSBL never downloaded/updated. Unbound does not come up after download try, resulting in no dns services next morning and rush to get it started.

I noticed this:

Date
   
Process
   
Line
   
2021-11-29T03:03:16   configd.py[36655]   unable to sendback response [Error (1) ] for [unbound][dnsbl][None] {6a572782-7c47-44b0-b4db-6d296f156192}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-29T01:15:39   configd.py[36655]   unable to sendback response [OK ] for [proxy][fetchacls][None] {95d1a118-8a58-454d-a810-da18c6f4c508}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:54:06   configd.py[36655]   unable to sendback response [OK ] for [interface][newip][['igb1']] {29900ec9-440f-4158-8b0a-b9eeaed5a15c}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:54:02   configd.py[36655]   unable to sendback response [OK ] for [interface][newipv6][['igb1']] {e200eb2f-6678-4370-ad75-f365515cd77f}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:54:00   configd.py[36655]   unable to sendback response [OK ] for [unbound][cache][['dump']] {3f68dc82-ad25-48cb-b475-7c0da132c62b}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:49:02   configd.py[36655]   unable to sendback response [OK ] for [unbound][cache][['dump']] {b0a9c3d1-e0da-4049-b978-185af844ef3f}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:30:54   configd.py[3794]   unable to sendback response [OK ] for [interface][newip][['igb0']] {4b7a2cbf-f722-4ff4-97e1-2a7ddddd4e7e}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:30:52   configd.py[3794]   unable to sendback response [OK ] for [unbound][cache][['dump']] {f5aa77d1-07a3-4a39-bb4a-cf00710fc614}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:30:10   configd.py[3794]   unable to sendback response [OK ] for [interface][linkup][['start', 'igb0']] {5c67c3e6-3f7d-46ec-828a-a9da5a23096a}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T16:28:11   configd.py[3794]   unable to sendback response [{"status":"ok","size":1000000,"used":128134,"details":{"Netflix":{"count":12,"updated":"2021-11-27T16:26:02.433660"},"TheKids":{"count":6,"updated":"2021-11-27T16:26:01.505264"},"bogons":{"count":1368,"updated":null},"bogonsv6":{"count":126748,"updated":null},"sshlockout":{"count":0,"updated":null},"virusprot":{"count":0,"updated":null}}} ] for [filter][diag][['table_size']] {52b0b9e1-5978-4d2b-bc1c-e837956bbfdb}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-27T15:31:59   configd.py[3794]   unable to sendback response [OK ] for [unbound][cache][['dump']] {33f991b3-811d-4039-b773-1907d0ca85a4}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-23T22:05:21   configd.py[72527]   unable to sendback response [OK ] for [unbound][start][None] {11560e0b-b844-4350-9227-841e5e47f446}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe   
2021-11-23T11:49:22   configd.py[72527]   unable to sendback response [OK ] for [unbound][cache][['dump']] {de8d4245-af33-4df2-a323-5c4fc302e2d6}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe

So
1) BrokenPipeError is consistent there and also applicable in other downloads/actions
2) BrokenPipeError seems to be the reason of DNS not coming back up. Funny thing is the BrokenPipeError at 2021-11-29T03:03:16 fails exactly 3min 16 sec after the start of the configured cron job. Might be related ?
For now I disabled the download of the DNSBL, but of course I like to enable it again.

Also interested to know what is the relationship with this report : https://forum.opnsense.org/index.php?topic=19432.0. By the way Intrusion Detection is NOT enabled on my system, so this might not be related at all...

Happy to provide more information if needed

Thanks for the feedback.

I understand Socks is not an option, worth of digging into, but not in the current situation

I added some updates in the original question being:
I added the result of the access.log
I added the configuration of the bump settings of the squid.conf
I updated the label of the "SNI"

tx
dirkp

Dear

I installed OpnSense 19.7.3 (I noticed today there is a 19.7.4, I will upgrade very soon). I created a transparent proxy both for HTTP and HTTPS. I strictly followed the manual and as fas as HTTP goes everything seems to be working fine.

As for the HTTPS I noticed some issues, not sure if these are the result my "misconfiguration" or expected behaviour or simply bugs.

There are a few conditions that are paramount for me in order to be able to maintain access throughout the networdk :
1) No certificates to be installed on the client machines. This is absolute horror and unmaintainable. That means that step 8 in https://docs.opnsense.org/manual/how-tos/proxytransparent.html is not executed.
[sitenote : can't I buy a certificate (not sure which kind) somewhere to avoid the creation of an own Certificate Authority (CA setting), that would be brilliant, I simply think this is a bit of a mess]
2) No maintenance in the " SSL no bump sites", again, this is not feasable from a maintenance point of view. That means that step 6 in https://docs.opnsense.org/manual/how-tos/proxytransparent.html is not an option, and hence not executed.
[sitenote, see further belwo] simply not executing step 6 results in no filtering, and I do want the filtering in SSL]
3) I want to apply (a rather stringent) Remote Access Control Lists to be applicable both in  HTTP and HTTPS (I already installed the yoyo (as suggested by the manual at https://docs.opnsense.org/manual/how-tos/cachingproxy.html#remote-black-list-ad-blocking), but I also want to use the UT1 as suggested by the manual here : https://docs.opnsense.org/manual/how-tos/proxywebfilter.html#step-2-configure-blacklist

Now these are the issues/questions I have
1) documentation : would it be possible in https://docs.opnsense.org/manual/how-tos/proxytransparent.html to indicate for step 6 how to avoid the bumping (cf my condition 2) & do enable the web filtering. I understand that you need to select the "SNI" option, but this is nowhere mentioned in the manuals. On the web some people refer to a field called "SSL/IP only" but this field no longer seem to exist.
Can someone confirm that this is the correct setting ? I might have missed something, but for my testing revealed to do transparent SSL filtering I needed to enable the "Enable SSL mode " option AND the "Log SNI information only" option right beneath it ([UPDATE: I added the correct label of the field and removed comment:ENDUPDATE], an option which is not mentioned in the manual), otherwise I get SSL errors all around the place (related to not trusted certificate), and the filtering seems simply not to take place.
2) As explained in 1 - I enabled "Enable SSL mode " option AND the "SNI". But this results in some weird behaviour :
- HTTP is keeps on working fine
- HTTPS : if I have a site in the enabled remote black list indicated by IP address, I receive an "void certifcate" & certifcation warning instead of squid redirecting me to the "not allowed page". If I access the same IP address with HTTP I get the "not allowed page"
- HTTPS : if I go to a site in the enabled remote black list indicated by a domain (style : .example.com) : I get PR_CONNECT_RESET ERROR, instead being nicely forwarded to the "not allowed page". At least the access is blocked, but I think it is not a nice handling, as the user is faced with somehting he does not understand instead of the "not allowed page" which he can interpret (and contact me about)
3) with big lists, I have the impression that not all "blocked" sites are taken into account. For instance, using the complete UT1 list as suggested by the manuel, some sites are blocked (with the PR_... error) and some are still accessible, which suggests that the list seems not be used "completely".
[UPDATE: logs from access.log with .mail.google.com in my Remote access Control List
1570783752.597      9 192.168.x.120 NONE/200 0 CONNECT 172.217.17.37:443 - HIER_NONE/- -
1570783752.599      1 192.168.x.120 TCP_DENIED/403 3721 CONNECT mail.google.com:443 - HIER_NONE/- text/html
1570783752.599      0 192.168.x.120 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -

-> I think the problem is related to the last line

:ENDUPDATE]

4) finally, when controlling the squid.conf the ssl_bump setting seems to be weird (or not consistent) if I compare this with suggestions on the squid-cache forum. But that is already another journey. I will create a separate question for this once things seem more clear.

[UPDATE: squid.conf
# setup ssl bump acl's
acl bump_step1 at_step SslBump1
acl bump_step2 at_step SslBump2
acl bump_step3 at_step SslBump3
acl bump_nobumpsites ssl::server_name "/usr/local/etc/squid/nobumpsites.acl"

# configure bump
ssl_bump peek bump_step1 all
ssl_bump splice all
ssl_bump peek bump_step2 all
ssl_bump splice bump_step3 all
ssl_bump bump

the configure of the bumps do not seem correct especially peek bump_step2 etc has no meaning

:ENDUPDATE]


Oh one more thing : is a socks proxy not an option ? And is it available by default or in the plugins ?

Happy to hear your input
rgds
dirkp