1
21.7 Legacy Series / Re: Odd TLS handshake issue with a few upstream servers
« on: November 18, 2021, 05:14:36 pm »
Thanks, will do.
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Pages: [1] 2
2
21.7 Legacy Series / Re: Odd TLS handshake issue with a few upstream servers
« on: November 18, 2021, 02:42:12 pm »
Bump
Anybody? I'm getting fed up with complaints about waiting minutes for reddit to load.
What more info would help?
Anybody? I'm getting fed up with complaints about waiting minutes for reddit to load.
What more info would help?
3
21.7 Legacy Series / Odd TLS handshake issue with a few upstream servers
« on: November 16, 2021, 04:45:53 pm »
Hello All,
We've discovered a strange issue related to connecting to one or two (only) upstream web servers. We're running OPNsense 21.7 (details attached) and it's working very well aside from this one thing that's come to light. We have a forward proxy configured to inspect full HTTP plus SNI only for HTTPS, which seems to be working fine, including banking sites etc as far as I've heard.
But a few users reported very slow connections (timeout or 30 seconds+) to one or two specific web sites, and I've been investigating, and I've managed to reproduce what I think is the problem for one particular instance from either my local bash (linux) or from the shell on the OPNsense firewall. Here's a openssl dialog from my local console:
'Connected' is returned fast and then after about 30 secs the error is reported.
Similarly, from the OPNsense shell I get:
This is only perhaps 30% of the requests, and this is the one and only upstream server I have managed to duplicate it for. The rest of the time the openssl handshake returns normally and fast, and lists the certificate, same as it does for all other servers I've tried.
I cannot find anything getting blocked at the firewall that might relate to this.
I've also tried from our infrastructure outside this firewall (AWS) and haven't been able to reproduce the problem.
For now the user workaround is to hit reload a few times, and eventually a request will go through normally.
I can't find much by googling the specific error (and only one other reference in the OPNsense forums), and most of the search hits seem to relate to upstream server misconfiguration, which seems unlikely in this case.
This is not a 'panic stations' issue, the workaround... works, but it does seem very odd, and I'm interested to try and solve it.
Any ideas welcome, thanks!
cwynd
We've discovered a strange issue related to connecting to one or two (only) upstream web servers. We're running OPNsense 21.7 (details attached) and it's working very well aside from this one thing that's come to light. We have a forward proxy configured to inspect full HTTP plus SNI only for HTTPS, which seems to be working fine, including banking sites etc as far as I've heard.
But a few users reported very slow connections (timeout or 30 seconds+) to one or two specific web sites, and I've been investigating, and I've managed to reproduce what I think is the problem for one particular instance from either my local bash (linux) or from the shell on the OPNsense firewall. Here's a openssl dialog from my local console:
Code: [Select]
$ openssl s_client -connect www.reddit.com:443 -prexit
CONNECTED(00000005)
140277959643584:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:
'Connected' is returned fast and then after about 30 secs the error is reported.
Similarly, from the OPNsense shell I get:
Code: [Select]
# openssl s_client -connect www.reddit.com:443 -prexit
1648418660352:error:0200203C:system library:connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:110:
1648418660352:error:2008A067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:111:
1648418660352:error:0200203C:system library:connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:110:
1648418660352:error:2008A067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:111:
connect:errno=60
The error is again reported after a long delay.This is only perhaps 30% of the requests, and this is the one and only upstream server I have managed to duplicate it for. The rest of the time the openssl handshake returns normally and fast, and lists the certificate, same as it does for all other servers I've tried.
I cannot find anything getting blocked at the firewall that might relate to this.
I've also tried from our infrastructure outside this firewall (AWS) and haven't been able to reproduce the problem.
For now the user workaround is to hit reload a few times, and eventually a request will go through normally.
I can't find much by googling the specific error (and only one other reference in the OPNsense forums), and most of the search hits seem to relate to upstream server misconfiguration, which seems unlikely in this case.
This is not a 'panic stations' issue, the workaround... works, but it does seem very odd, and I'm interested to try and solve it.
Any ideas welcome, thanks!
cwynd
4
21.7 Legacy Series / Re: 'Default Deny' blocking all OPT1 traffic
« on: October 17, 2021, 02:52:58 pm »
@franco @pmhausen Thanks so much for responding.
To be clear, I am definitely no firewall expert, just I seem to know slightly more than anyone else here, so... though having said that we do aim to be diligent and secure.
Your replies helped me clear up two things I missed in the heat of an attempted deploy yesterday:
* The default deny rule is a last resort rule, not a first match / quick rule. I missed that, although on reflection it should be obvious.
* My hurried attempt at a default pass any-to-any rule on the OPT1 interface (by cloning the LAN rule) failed to update the interface net it matched, so obviously it did nothing.
With the latter corrected everything appears to be working. Totally my carelessness.
Thanks again!
To be clear, I am definitely no firewall expert, just I seem to know slightly more than anyone else here, so... though having said that we do aim to be diligent and secure.
Your replies helped me clear up two things I missed in the heat of an attempted deploy yesterday:
* The default deny rule is a last resort rule, not a first match / quick rule. I missed that, although on reflection it should be obvious.
* My hurried attempt at a default pass any-to-any rule on the OPT1 interface (by cloning the LAN rule) failed to update the interface net it matched, so obviously it did nothing.
With the latter corrected everything appears to be working. Totally my carelessness.
Thanks again!
5
21.7 Legacy Series / Re: 'Default Deny' blocking all OPT1 traffic
« on: October 17, 2021, 02:55:00 am »
Anyone??...
6
21.7 Legacy Series / 'Default Deny' blocking all OPT1 traffic
« on: October 16, 2021, 08:07:10 pm »
Hello all,
I'm trying to get a relatively simple OPNsense 21.7.3_3-amd64 set up to the point where I can deploy it.
Brief summary:
* new bare metal hardware, 1 WAN 1 LAN and 1 OPT, one spare RJ45 port.
* One (1) device connected to each of LAN and OPT1 ports by a single piece of wire (no routers and before you ask several different cables tried).
The new setup is intended to clone & upgrade an existing production setup which is OPNsense 18 for a long time, and is aging, and the hardware is slow - hence the attempt to upgrade.
The new LAN is intended for secure business traffic only, and the OPT1 is for more open "retail" type traffic. There are no VLANs involved.
LAN <~> WAN was extensively tested before the attempted deploy and is working fine. However when I tried to deploy during off hours today and replace production OPNsense there is no traffic at all getting passed by OPT1, and I eventually had to roll back to the old system to avoid an extended outage.
On digging subsequently in my test set up I can ping and see packet logs back & forth just fine for LAN, but for OPT1 I can ping out from OPT1 to my test device and get replies, but pinging from the single device to OPT1 never gets a reply, and the firewall logs show
Everything's been restarted several times.
I've tried putting a OPT1 'allow any from any' rule at the top of the ruleset, but it appears to still get stomped by the Floating 'default deny' and has no effect. Other than that the rules for LAN and OPT1 are substantially identical (the plan had been to spend the day adding all the detailed rules once the basics were working - until we hit this roadblock).
Sorry for long intro, but finally I have three questions:
1) Why is OPT1 behaving differently than LAN interface, with the same rules, i.e. why are the floating rules being applied differently? I'm missing where the asymmetry is coming from.
2) What is the correct way to open anything in the 'Default Deny' rule??
3) Related to #2, I've seen some talk while googling of "weirdness" related to the Floating Default Deny in recent OPNsense releases - should I be thinking about downgrading?
Thanks for any and all advice!
I'm trying to get a relatively simple OPNsense 21.7.3_3-amd64 set up to the point where I can deploy it.
Brief summary:
* new bare metal hardware, 1 WAN 1 LAN and 1 OPT, one spare RJ45 port.
* One (1) device connected to each of LAN and OPT1 ports by a single piece of wire (no routers and before you ask several different cables tried).
The new setup is intended to clone & upgrade an existing production setup which is OPNsense 18 for a long time, and is aging, and the hardware is slow - hence the attempt to upgrade.
The new LAN is intended for secure business traffic only, and the OPT1 is for more open "retail" type traffic. There are no VLANs involved.
LAN <~> WAN was extensively tested before the attempted deploy and is working fine. However when I tried to deploy during off hours today and replace production OPNsense there is no traffic at all getting passed by OPT1, and I eventually had to roll back to the old system to avoid an extended outage.
On digging subsequently in my test set up I can ping and see packet logs back & forth just fine for LAN, but for OPT1 I can ping out from OPT1 to my test device and get replies, but pinging from the single device to OPT1 never gets a reply, and the firewall logs show
Code: [Select]
action block dir [in] src 192.168.129.185 dst 192.168.129.1 <this is OPT1> - details in the attached screen shot (OPT1 is called DDWRT there for historical reasons).Everything's been restarted several times.
I've tried putting a OPT1 'allow any from any' rule at the top of the ruleset, but it appears to still get stomped by the Floating 'default deny' and has no effect. Other than that the rules for LAN and OPT1 are substantially identical (the plan had been to spend the day adding all the detailed rules once the basics were working - until we hit this roadblock).
Sorry for long intro, but finally I have three questions:
1) Why is OPT1 behaving differently than LAN interface, with the same rules, i.e. why are the floating rules being applied differently? I'm missing where the asymmetry is coming from.
2) What is the correct way to open anything in the 'Default Deny' rule??
3) Related to #2, I've seen some talk while googling of "weirdness" related to the Floating Default Deny in recent OPNsense releases - should I be thinking about downgrading?
Thanks for any and all advice!
7
17.1 Legacy Series / Re: Correct swap configuration?
« on: June 27, 2017, 05:29:54 pm »
Hello All, this continues to be an issue. I gathered some details from the latest incident, reported here:
Log Messages:
OPNsense is running as a xen domU, and is allocated 3.8GB of memory.
I got to this about 20 minutes after squid was killed by oom, and the attached snippet shows details on the memory allocation at the time.
Can anyone offer any hints about swap configuration please?? If I am asking in the wrong place, or if I missed a write-up on, apologies, please point me to same.
Thanks!
Log Messages:
Code: [Select]
Jun 27 10:46:04 squid[32290]: Squid Parent: (squid-1) process 32803 exited due to signal 9 with status 0
Jun 27 10:46:01 kernel: pid 32803 (squid), uid 100, was killed: out of swap space
Jun 26 18:58:26 kernel: 906.076340 [ 792] generic_netmap_dtor Restored native NA 0
Jun 26 18:58:26 kernel: pid 37449 (suricata), uid 0, was killed: out of swap spaceOPNsense is running as a xen domU, and is allocated 3.8GB of memory.
I got to this about 20 minutes after squid was killed by oom, and the attached snippet shows details on the memory allocation at the time.
Can anyone offer any hints about swap configuration please?? If I am asking in the wrong place, or if I missed a write-up on, apologies, please point me to same.
Thanks!
8
17.1 Legacy Series / Correct swap configuration?
« on: May 16, 2017, 03:58:02 pm »
Hello All, we have a 17.1 production OPNsense + ICAP server, all running smoothly - until yesterday.
Yesterday I added the emerging threats rules to suricata (which was already enabled with a limited ruleset), and after a few hours, the system started killing processes, and reporting "out of swap space".
The system is running as a Xen VM with 3G of memory and 3 NICs, and was set up accepting default install options a month or two ago. When I check from the shell
Would appreciate any advice on what is the right way to configure OS swap.
Thanks!
Yesterday I added the emerging threats rules to suricata (which was already enabled with a limited ruleset), and after a few hours, the system started killing processes, and reporting "out of swap space".
The system is running as a Xen VM with 3G of memory and 3 NICs, and was set up accepting default install options a month or two ago. When I check from the shell
Code: [Select]
swapinfo -h no swap is present. After searching the forums, I am not clear if I should be configuring swap or not.Would appreciate any advice on what is the right way to configure OS swap.
Thanks!
9
17.1 Legacy Series / Re: Difficulty getting VoIP - asterisk IAX2 working (17.1 production)
« on: April 24, 2017, 11:25:02 pm »
Thanks for responding Bart. I'm afraid I ran out of time for working on this (after a couple more hours of trying numerous NAT settings including no nat and static port). As well this firewall is now running in production (maybe prematurely on my part!) so I cannot kill all the states, and try different things too much.
So I have dropped in a point to point tunnel (vtun) which solved my immediate problem right away. I was unable to get a NAT-based solution to work. Honestly I was surprised the pfSense config worked when I looked at it, but it did, so I did not explore why obviously. I seem to remember seeing "somewhere on the internet" that pfSense had included some code tweaks to make VoIP work...
I know this is not an ideal solution, and when I have time I will try to investigate some more, but for now I've "fixed the problem" (to quote Office Space). Thanks again for your help.
As well, if there are any OPNsense users out there with VoIP running (either SIP or preferably IAX2) I would love to learn about working configs.
cw
So I have dropped in a point to point tunnel (vtun) which solved my immediate problem right away. I was unable to get a NAT-based solution to work. Honestly I was surprised the pfSense config worked when I looked at it, but it did, so I did not explore why obviously. I seem to remember seeing "somewhere on the internet" that pfSense had included some code tweaks to make VoIP work...
I know this is not an ideal solution, and when I have time I will try to investigate some more, but for now I've "fixed the problem" (to quote Office Space). Thanks again for your help.
As well, if there are any OPNsense users out there with VoIP running (either SIP or preferably IAX2) I would love to learn about working configs.
cw
10
17.1 Legacy Series / Re: Difficulty getting VoIP - asterisk IAX2 working (17.1 production)
« on: April 24, 2017, 06:28:16 pm »
Thanks for quick reply. I just tried that, and restarted asterisk at both ends. I am still seeing inside registered to outside on a high port # (which seems odd to me). So unfortunately it did not work. Both ends are static IPs btw.
cw
cw
11
17.1 Legacy Series / Difficulty getting VoIP - asterisk IAX2 working (17.1 production)
« on: April 24, 2017, 06:13:19 pm »
Hi All, we cut over to OPNsense a few days ago, and everything is fine now, with one exception - asterisk IAX2.
Brief background: we have two asterisk servers one inside OPNsense (LAN) and one outside, and we have full control of both ends. There is (should be) an IAX trunk between them, which was working fine for several years with old pfSense box.
For starters I copied over the pfSense rules that were working, which were only:
NAT Port Forward
Interface: WAN
Proto: TCP/UDP
Source: outside-asterisk:4569
Dest: WAN-address:4569
NAT IP: inside-asterisk:4569
-with an autogenerated firewall inbound pass rule to match.
I have tried tweaking / adding to this rule many different ways with no progress. Here's the current 'state of play':
* Outbound calls work fine (inside ~> outside)
* Inbound calls ring until timeout on outside-asterisk with no evidence of any hit on inside-asterisk
* Inside-asterisk shows as registered with outside asterisk but with a high port #, not 4569
* Outside-asterisk shows inside-asterisk registered on same high port#
I am thinking that what is happening is that when a call comes in, outside-asterisk tries to contact inside on that same high port #, and is getting blocked at the firewall, since there is no NAT state (UDP) for it to key into. I dont have the old firewall still in the loop, but if I remember correctly the IAX2 traffic was not getting NATed on the way out, so it was all 4569 end-to-end.
Would really appreciate any hints or advice - or even ideas on what to try next. Thanks!!
Brief background: we have two asterisk servers one inside OPNsense (LAN) and one outside, and we have full control of both ends. There is (should be) an IAX trunk between them, which was working fine for several years with old pfSense box.
For starters I copied over the pfSense rules that were working, which were only:
NAT Port Forward
Interface: WAN
Proto: TCP/UDP
Source: outside-asterisk:4569
Dest: WAN-address:4569
NAT IP: inside-asterisk:4569
-with an autogenerated firewall inbound pass rule to match.
I have tried tweaking / adding to this rule many different ways with no progress. Here's the current 'state of play':
* Outbound calls work fine (inside ~> outside)
* Inbound calls ring until timeout on outside-asterisk with no evidence of any hit on inside-asterisk
* Inside-asterisk shows as registered with outside asterisk but with a high port #, not 4569
* Outside-asterisk shows inside-asterisk registered on same high port#
I am thinking that what is happening is that when a call comes in, outside-asterisk tries to contact inside on that same high port #, and is getting blocked at the firewall, since there is no NAT state (UDP) for it to key into. I dont have the old firewall still in the loop, but if I remember correctly the IAX2 traffic was not getting NATed on the way out, so it was all 4569 end-to-end.
Would really appreciate any hints or advice - or even ideas on what to try next. Thanks!!
12
17.1 Legacy Series / [SOLVED] Re: Looking for help re squidclamav redirect problem
« on: April 11, 2017, 08:46:57 pm »
ok figured it out. The problem was that suricata was blocking the eicar response before it even got to the proxy+squidclamav. Obvious with hindsight, but not at all in the heat of the moment. Once I unblocked that, everything works as intended. At least I understand ICAP protocol a little better from reading the RFP 
13
17.1 Legacy Series / [SOLVED] Looking for help re squidclamav redirect problem
« on: April 11, 2017, 04:28:10 pm »
Hi All, I am trying to get squidclamav working with OPNsense 17.1. My problem is I cannot get the redirect to work for eicar test virus files. I have everything working fine for clean urls, and with squidclamav debug enabled I see it receiving and processing the request::
But for an eicar test virus url, it seems like the request get's released by squidclamav (if I am reading the below correctly), but a response is never received, and never forwarded to clamd for scanning, and no response or redirect ever goes back to the client browser, which eventually reports a '504 Gateway Error'. This is the complete squidclamav debug log:
I have duplicated this problem on two completely separate c-icap/squiidclamav/clamd setups now, most recently with latest c-icap (0.5.2) and squidclamav (6.16) compiled from sources, so I'm sure it's something I am doing wrong. But after 3 days of poring over logs I cannot see it.
Any help or hints greatly appreciated!!
PS: clwarn.cgi is in the right place, and is visible to the client browser if I load it directly) According to the apache logs it never gets hit unless I load it explicitly.
PPS: This is all HTTP (no HTTPS... yet)
Edit: Marked solved.
Code: [Select]
DEBUG initializing request data handler.
DEBUG processing preview header.
DEBUG X-Client-IP: 192.168.1.150
DEBUG method GET
DEBUG url http://no.viruses.here
DEBUG URL requested: http://no.viruses.here
DEBUG Content-Length: -1
DEBUG No body data, allow 204
DEBUG Releasing request data.
DEBUG initializing request data handler.
DEBUG processing preview header.
DEBUG preview data size is 1024
DEBUG X-Client-IP: 192.168.1.150
DEBUG method GET
DEBUG url http://no.viruses.here
DEBUG URL requested: http://no.viruses.here
DEBUG Content-Length: 3699
DEBUG Content-Type: text/html
DEBUG End of method squidclamav_check_preview_handler
DEBUG ending request data handler.
Sending zINSTREAM command to clamd.
DEBUG Ok connected to clamd.
DEBUG: Scanning data now
DEBUG Write 3703 bytes on 3699 to socket
DEBUG received from Clamd: stream: OK
DEBUG Closing Clamd connection.
DEBUG Responding with allow 204
But for an eicar test virus url, it seems like the request get's released by squidclamav (if I am reading the below correctly), but a response is never received, and never forwarded to clamd for scanning, and no response or redirect ever goes back to the client browser, which eventually reports a '504 Gateway Error'. This is the complete squidclamav debug log:
Code: [Select]
DEBUG initializing request data handler.
DEBUG processing preview header.
DEBUG X-Client-IP: 192.168.1.150
DEBUG method GET
DEBUG url http://www.eicar.org/download/eicar.com
DEBUG URL requested: http://www.eicar.org/download/eicar.com
DEBUG Content-Length: -1
DEBUG No body data, allow 204
DEBUG Releasing request data.
I have duplicated this problem on two completely separate c-icap/squiidclamav/clamd setups now, most recently with latest c-icap (0.5.2) and squidclamav (6.16) compiled from sources, so I'm sure it's something I am doing wrong. But after 3 days of poring over logs I cannot see it.
Any help or hints greatly appreciated!!
PS: clwarn.cgi is in the right place, and is visible to the client browser if I load it directly) According to the apache logs it never gets hit unless I load it explicitly.
PPS: This is all HTTP (no HTTPS... yet)
Edit: Marked solved.
14
17.1 Legacy Series / Re: [SOLVED] ClamAV scanning for Proxy - Having trouble getting working
« on: April 10, 2017, 10:01:02 pm »
@kyferez thanks for a really good and detailed howto!
Just one suggestion: in your step 3.18:
Suggest to change 'Destination Address: ANY' to Destination Address: !This Firewall for this and the corresponding port 443 rule
I just locked myself out of the webgui when I was fiddling with icap server memory and couldn't get the c-icap service to start. The above change ensures that OPNsense does not try forward http(s) requests pointed exactly at OPNsense itself to c-icap. I had to get c-icap running again and responding before I could get into the OPNsense webgui.
Just one suggestion: in your step 3.18:
Quote
Once the proxy is working, if you want to block anyone not using the proxy, then add a new firewall rule below the one you created earlier. This rule should be Deny traffic, Source: Interface net, Destination Address: ANY, Dst Port: 80. ...
Suggest to change 'Destination Address: ANY' to Destination Address: !This Firewall for this and the corresponding port 443 rule
I just locked myself out of the webgui when I was fiddling with icap server memory and couldn't get the c-icap service to start. The above change ensures that OPNsense does not try forward http(s) requests pointed exactly at OPNsense itself to c-icap. I had to get c-icap running again and responding before I could get into the OPNsense webgui.
15
17.1 Legacy Series / Re: Web Proxy Not responding Transparent
« on: March 23, 2017, 04:44:02 pm »Got it working on 16.7. Going to test on 17.1 shortly. [UPDATE: It works on 17.1.2]
Complete guide to Proxy with AV Scanning: http://www.tcptechs.com/opnsense-transparent-caching-filtering-proxy-with-virus-scanning/
-snip-
Thank you very much! ^ guide got me out of hours of fiddling unsuccessfully with the default guide ruleset
Pages: [1] 2