Hi all,
I'm running into a frustrating issue where certain websites, primarily smaller sites hosted in China (e.g., www.dessmonitor.com IP: 8.210.123.202), are inaccessible when routing through my OPNsense firewall, while major international sites work fine.
Symptoms:
DNS Resolution Works: nslookup from a LAN client correctly resolves the domain names to their public IP addresses via OPNsense Unbound.
Standard Ping Works: ping 8.210.123.202 from the LAN client receives replies successfully.
Browser Fails: Accessing the site via a browser (Chrome, Edge) results in a timeout error (e.g., ERR_CONNECTION_TIMED_OUT).
curl Confirms Timeout: Using curl -v https://www.dessmonitor.com from the client successfully resolves the DNS name but fails during the TCP connection attempt:
* Trying 8.210.123.202:443...
* connect to 8.210.123.202 port 443 from 0.0.0.0 port [port] failed: Timed out
* Failed to connect to www.dessmonitor.com port 443 after [time] ms: Couldn't connect to server
curl: (28) Failed to connect to www.dessmonitor.com port 443 after [time] ms: Couldn't connect to server
Use code with caution.
My Setup:
OPNsense Version: Current 25.1.5_5, previous attempts 24.7.1 - 12
Basic Setup: WAN (DHCP/Static, ISP Gateway 10.1.1.1, 2 x NAT), LAN (10.0.0.0/16).
WireGuard is configured for selective routing (using alias) but the client experiencing this issue is intended to use the standard WAN gateway. WireGuard hosts can access the site.
Bypassing OPNsense and connecting directly to WAN router loads https://www.dessmonitor.com.
Unbound DNS is used for LAN clients. DNSBL is enabled but doesn't seem related as nslookup works and was failing even before enabling Unbound.
Client OS: Windows 10/11
Troubleshooting Steps Taken:
DNS: Confirmed resolution works via nslookup and curl. Disabled Chrome's Secure DNS.
Basic Connectivity: Standard ping to the destination IP works fine. Access to major international sites (Google, Cloudflare, etc.) works fine.
Firewall Rules: Reviewed Firewall -> Rules -> LAN. Temporarily disabled all rules except the default "allow LAN to any" rule. The issue persisted. No specific rules appear to be blocking this destination IP/port. Checked Live Logs while attempting connection - no blocks shown for the destination IP.
Outbound NAT: Using Hybrid mode with a specific rule for the WireGuard alias. Tested switching to Automatic Outbound NAT - issue persisted.
Gateway Status: WAN gateway is online and functioning for other traffic.
MTU/MSS:
Performed ping <dest_ip> -f -l <size> tests from the client.
Determined Path MTU is 1488 (largest successful payload -l 1460).
Enabled MSS clamping under Firewall -> Settings -> Advanced -> Enable Maximum MSS and set the value explicitly to 1448 (1488 - 40).
Reset firewall states (Firewall -> Diagnostics -> States -> Reset).
Result: This did NOT resolve the timeout issue.
Normalization: Reviewed Firewall -> Settings -> Normalization. Default scrub is enabled. A specific max-mss rule exists only for the WG1 (WireGuard) interface, not WAN. Tested disabling normalization globally - issue persisted.
IDS/IPS: Intrusion Detection (Suricata) is currently disabled.
States: Reset firewall states multiple times after configuration changes.
Request for Help:
Given that DNS resolution works, standard ping works, and MSS clamping based on tested Path MTU didn't resolve the TCP connection timeout specifically for these sites, what else should I investigate within OPNsense?
Are there specific logs (beyond Unbound and basic filtered live view) I should enable or examine?
Could this be related to state handling, specific NAT behavior, or upstream routing issues that OPNsense might interact with differently for these paths?
How can I effectively trace or debug the failing TCP connection as it passes through OPNsense? For example, what are the recommended steps/filters for using Packet Capture (Interfaces -> Diagnostics -> Packet Capture) on the LAN and WAN interfaces simultaneously to see where the SYN packet goes and if a SYN-ACK returns?
Any pointers on what specific settings or diagnostics to try next would be greatly appreciated.
Remember this ONLY affects some smaller Chinese sites not just https://www.dessmonitor.com, and the issue presents itself on new install with very little configured. Usually the Chinese site won't load like it's down, but then I connect either through browser proxy or upstream of opnsense box and it loads fine.
Thanks!
Quote from: raybies on April 23, 2025, 07:55:58 AMBypassing OPNsense and connecting directly to WAN router loads https://www.dessmonitor.com.
Run a Wireshark while accessing the site via the WAN router then another trace through OPNsense. See what is different.
Is Zenarmor or Crowdsec enabled? MTU might still be a problem: I can confirm that the real maximum achievable MTU to www.dessmonitor.com is indeed 1500, so if you see 1488, it is on your side. To fix that, see this (https://forum.opnsense.org/index.php?topic=45658.0).
Thanks for the responses.
Zenarmor or Crowdsec NOT enabled.
Capture Analysis - Tracking the Connection:
Client Sends SYN:
capture_client_23-4_2.csv Packet 126 (Time 8.449): 10.0.1.15 -> 8.210.123.202 [SYN] (Src Port 58821)
capture_client_23-4_2.csv Packet 128 (Time 8.584): 10.0.1.15 -> 8.210.123.202 [SYN] (Src Port 58822)
(Client capture shows many subsequent SYN retransmissions)
OPNsense Forwards SYN (and performs NAT):
capture_opnsense_23-4_2.csv Packet 11 (Time 1.239): 10.1.1.100 (OPNsense WAN IP) -> 8.210.123.202 [SYN] (Src Port 33933 - Note the NATed source port)
(This confirms OPNsense received the client's SYN on LAN and sent it out the WAN)
Server Responds with SYN-ACK:
capture_opnsense_23-4_2.csv Packet 12 (Time 1.450): 8.210.123.202 -> 10.1.1.100 (OPNsense WAN IP) [SYN, ACK] (Dest Port 33933)
(This is crucial! The server did respond, and OPNsense received the SYN-ACK on its WAN interface.)
OPNsense Completes Handshake (WAN side):
capture_opnsense_23-4_2.csv Packet 13 (Time 1.451): 10.1.1.100 -> 8.210.123.202 [ACK]
(OPNsense sends the final ACK for the handshake out the WAN)
The Failure Point - SYN-ACK Not Reaching Client:
Compare Packet 12 from the OPNsense capture (SYN-ACK arriving at OPNsense WAN) with the entire client capture.
There is NO corresponding SYN-ACK packet arriving at the client 10.0.1.15 from 8.210.123.202 in capture_client_23-4_2.csv.
I have dozens of times:
Reset States.
Reboot OPNsense.
Reviewed Firewall Logs (Filtered): All green.
I don't get it (could be my problem).
I ran a quick capture for some random web site, downloaded it, opened the caps in wireshark.
Both look identical apart from the NAT rewrites.
Everything OPN does is a reflection of what the client does.
In particular, I don't expect OPN to ACK anything on its own.
The ACK from OPN is the NAT equivalent of the same packet from the client, because the client got the [SYN, ACK].
This said, the time offsets don't necessarily align across both captures so it could be confusing.
The whole issue is confounding; hence why I am here. Why does it affect only some Chinese sites? Why did it work for a very short period just after reinstalling v25.1—before accessing the UI and after setting up the interfaces and IP ranges.
I've reinstalled OPNsense several times and even changed the entire machine, last resort, thinking it could be a HW issue.
Also note this doesn't just affect my testing Windows client, it affects every client eg: Ubuntu servers, Proxmox, etc.. the only commonality is IP range 10.0.0.0/16
Can you share the network captures from OPN for the case that fails?
OPN's WAN IP seems to be a private network IP so I don't expect the capture to contain anything private/sensitive.
Can you put your WAN router in modem mode? NAT strictly speaking breaks HTTP and double NAT doubly so.
OPNSense [WAN] capture: https://drive.google.com/file/d/13e1IGOWSGeyIsPOLwRwv8LTwHQQl6RVT/view?usp=sharing
Not sure what modem mode means, it's an Asus Merlin fw wifi router, it performs PPPoE with my ISP and does the first NAT.
I know 2xNAT is bad, but it's way less bad than having the Mrs go nuts on me because something weird has happened on the opnsense side, so she connects directly to Asus Merlin fw wifi, bypassing "my LAN". It also allows me to quickly troubleshoot an issue's location, among other benefits. I'm debating IPv6.
It's a little weird.
The capture refers to a web.dess... that resolves to the IP in the OP (apparently a Hong-Kong version of the site).
For me, www.dess... refers to the americas version of the site.
Interestingly, curl https://web.dess... (same IP as OP) returns something that looks like the nginx default page.
curl https://www.dess... returns a more complete page (and actually leaves the connection intact)
There's nothing obviously wrong in the capture.
The returned content is smaller than for me (675 versus 1137), but the request could be different, then the connection is torn down.
Then new connections are established. The sizes aren't similar so likely not retries. Chasing links in a browser (not curl)?
I obviously can't peek in the encrypted content...
That's my interpretation... I could be wrong.
I think they're just using Host Headers/Virtual Hosts to load different branded sites on the same server/IP.
So how do I resolve the OPNsense issue? It only fails when accessing through my OPNsense.
DNS resolving to different IPs based on source location is not unusual.
I was at least mentioning that the OP was about www.dess while the capture indicated web.dess
FWIW, the capture was not filtered to the target IP and only contained the WAN side...
I have nowhere near the amount of expertise as the other 2 that replied.
But I didn't see anything indicating a problem given the info available.
If you're going to redo captures (with the LAN side), please filter (so I don't have to) and perform a single curl operation (so there's less noise).
You don't have a proxy in the middle, right?
No proxy.
I ran a capture directly on a test machine for a single invocation of 'curl -v https://web.dessmonitor.com'
I used web (not www) to get the same IP as you did.
Screenshot 2025-04-25 MyCap.png
Again, I get the default nginx page back...
Here's your capture filtered down to that IP (first connection only, since I suspect your capture was done in a browser):
Screenshot 2025-04-25 231529.png
Same number of exchanges, which leads me to believe some content made it back in frame 29.
As a router, even with NAT, I believe OPN is entirely acting on behalf of the client (on the LAN side).
It's pretty clear there's some back and forth going on, including key exchange (which should result in some display with -v).
Then a GET with the URL returning a small payload (smaller than the default nginx page, an error maybe, but not a failure to connect).
Then the connection is torn down.
At this point, running a Wireshark capture on the Windows machine seems necessary (filter and export specified packets if you want to share).
Share the complete output of the curl -v command used while you're at it.
Here are client side captures.
1. Win PC on [LAN] access www.dessmonitor.com via [WAN]
C:\Users\Admin>curl -v https://www.dessmonitor.com
* Host www.dessmonitor.com:443 was resolved.
* IPv6: (none)
* IPv4: 8.210.123.202
* Trying 8.210.123.202:443...
* connect to 8.210.123.202 port 443 from 0.0.0.0 port 50099 failed: Timed out
* Failed to connect to www.dessmonitor.com port 443 after 21535 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to www.dessmonitor.com port 443 after 21535 ms: Couldn't connect to server
(https://i.ibb.co/Nn34xbM9/Capture-OPNsense-WAN.jpg)
2. Win PC on [LAN] access www.dessmonitor.com via [WG1] this is the WireGuard interface, on the same OPNsense machine.
C:\Users\Admin>curl -v https://www.dessmonitor.com
* Host www.dessmonitor.com:443 was resolved.
* IPv6: (none)
* IPv4: 8.210.123.202
* Trying 8.210.123.202:443...
* Connected to www.dessmonitor.com (8.210.123.202) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.x
> GET / HTTP/1.1
> Host: www.dessmonitor.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
< HTTP/1.1 200 OK
< Server: nginx/1.9.6
< Date: Sun, 27 Apr 2025 03:13:18 GMT
< Content-Type: text/html
< Content-Length: 13434
< Last-Modified: Wed, 26 Mar 2025 11:16:06 GMT
< Connection: keep-alive
< ETag: "67e3e1f6-347a"
< Expires: Sun, 27 Apr 2025 03:13:17 GMT
< Cache-Control: no-cache
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET,POST,OPTIONS
< Accept-Ranges: bytes
(https://i.ibb.co/BH2W85VG/Capture-OPNsense-WG1.jpg)
During the 2nd part, the machine (same IP as in part 1) was added to the alias containing hosts that are directed to your external WGD VPN provider?
Apart from validating that this machine can access that site, I'm not sure what I can do with that.
It would have been more straightforward to connect that machine to your edge router (bypassing OPN).
The capture from part 1 indicates a complete connectivity failure. That connection attempt is either outright blocked, or maybe it gets routed into a void.
I can't reconcile this with the previous capture (reply #8) on WAN.
The WAN side indicated several roundtrips to the target website (3-way handshake, client then server hello, key exchange and a HTTPS roundtrip, connection close).
I don't really care to see the DNS lookups. We know the IP by now.
In an attempt to connect the dots, for the part 1 use case (not going thru the VPN provider), please provide a WAN and LAN capture filtered to the target IP for that curl command.
Can you pls give me a full list of tests you would like me to perform, how you would like them filtered and the expected format?
One packet capture on OPN, with WAN and LAN selected, filtered by target 'host address' IP.
While that's going on, a single curl invocation. Then download the capture (it should contain 2 cap files and a json).
It's similar to what you did in reply #8, apart from the fact that we'd see both interfaces for the exact same exchange, not just WAN.
The filtering is for your privacy and reduces the size of the file. We don't need to see other traffic.
Again, the capture is #8 looked normal to me, but it makes no sense when compared with the Wireshark cap on the host.
Having the LAN side too, for a single exchange, would eliminate discrepancies due to looking at discrete exchanges that may have happened in different conditions.
Sorry for the late reply—life got in the way.
I have some more info and the requested capture.
I have an integration on Home Assistant that connects to dessmonitor, previously it worked, then it stopped working, in this capture it's working. The inability to get HA to connect to dessmonitor is what pushed me to reinstall OPNsense on a new machine. I disabled the integration, but the clients still can't connect to dessmonitor.
In the attached OPNsense capture, you can see the HA server 10.0.5.100, which is running as a VM on a Proxmox host, and the Windows client (10.0.1.15) from which I ran...
C:\Users\Admin>curl -v https://www.dessmonitor.com
* Host www.dessmonitor.com:443 was resolved.
* IPv6: (none)
* IPv4: 8.210.123.202
* Trying 8.210.123.202:443...
* connect to 8.210.123.202 port 443 from 0.0.0.0 port 61689 failed: Timed out
* Failed to connect to www.dessmonitor.com port 443 after 21611 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to www.dessmonitor.com port 443 after 21611 ms: Couldn't connect to server
There are no aliases, rules or anything in opnsense to distinguish the 2 IPs.
Non captured after disabling dessmonitor integration.
The proxmox server running HA:
root@proxmox:~# curl -v https://www.dessmonitor.com
* Trying 8.210.123.202:443...
* connect to 8.210.123.202 port 443 failed: Connection timed out
* Failed to connect to www.dessmonitor.com port 443 after 136454 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to www.dessmonitor.com port 443 after 136454 ms: Couldn't connect to server
An ubuntu 24.4 server:
root@a51m:~# curl -v https://www.dessmonitor.com
* Host www.dessmonitor.com:443 was resolved.
* IPv6: (none)
* IPv4: 8.210.123.202
* Trying 8.210.123.202:443...
* connect to 8.210.123.202 port 443 from 10.0.1.51 port 56914 failed: Connection timed out
* Failed to connect to www.dessmonitor.com port 443 after 133075 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to www.dessmonitor.com port 443 after 133075 ms: Couldn't connect to server
I'm going to paraphrase to make sure I get it.
Something in HA is making background requests to dessmonitor (about 4 in the 30 seconds span of the capture).
They succeeded here but they don't always.
FWIW, the requests appear to be targeted at web.dessmonitor.com (not www) although both FQDNs resolve to the same IP in your case.
While this is ongoing, you invoked curl on the www address (twice apparently, 4secs apart) from another machine on the same LAN interface.
Both failed miserably.
Question 1: what's the interface IP configuration looking like?
Question 2: what do you know about this HA "integration" with dessmonitor?
I ask because I observe 2 small differences between the SYN packets TCP options:
* Window Scale is 7 coming from HA, 8 coming from Windows
* The HA frame features a Timestamps option and is thus 8 bytes larger
This is way too low level for me to tell if it is significant.
Maybe I'll do some research at some point (but definitely not tonight).
Another thought, given usage pattern here, is the server limiting connections per IP.
That would be from the server's perspective, so even stopping the client may not be sufficient if the connections are not closed properly (even though they should eventually timeout).
And you're behind another router too so even that router could do something unbecoming with too many connections with same source (OPN) and same destination (dess).
Quote from: raybies on April 23, 2025, 07:55:58 AMI'm running into a frustrating issue where certain websites, primarily smaller sites hosted in China (e.g., www.dessmonitor.com IP: 8.210.123.202), are inaccessible when routing through my OPNsense firewall, while major international sites work fine.
Are they accessible if you use something different for a router? For example, pfSense (not that it would be different) or any Linux router or even a Windows connection-sharing box?
Are they accessible from the router itself if you log into SSH console and curl/wget/lynx from there?
I considered the endpoint throttling or even blocking IPs via some DDoS algo, but I've changed ISP (completely different IP range), and I've never had an issue with anything upstream of opnsense.
I've disabled all the security features of the primary router with no change.
I don't think I've changed anything on either WAN or LAN since the last reinstall.
Here's LAN.
(https://i.ibb.co/8nznXvWX/LAN.jpg)
Integration is OS so I know *everything*. Code: https://github.com/Antoxa1081/home-assistant-dess-monitor/blob/main/custom_components/dess_monitor/api/__init__.py
From opnsense shell, yes it accessible... also note that it has been accessible briefly on LAN via WAN opnsense I *think* it started failing after importing DHCP address but not sure, but that's the first thing I do or I can't access any of my devices. It still works fine LAN via WireGuard. I haven't tried pfSense recently, but I wouldn't go back.
Also note that previously for ~ 1 year probably all on 24.x to 24.7 lots of Chinese websites (like machinery products), and only Chinese websites were exhibiting this behavior. I didn't investigate too deeply and assumed it was just the Unbound BL, but then I disabled the block lists and they still weren't accessible so the resolution climbed my to do list.
root@OPNsense:~ # curl -v https://www.dessmonitor.com
* Host www.dessmonitor.com:443 was resolved.
* IPv6: (none)
* IPv4: 8.210.123.202
* Trying 8.210.123.202:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / prime256v1 / rsaEncryption
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=*.dessmonitor.com
* start date: Jun 2 00:00:00 2024 GMT
* expire date: Jul 3 23:59:59 2025 GMT
* subjectAltName: host "www.dessmonitor.com" matched cert's "*.dessmonitor.com"
* issuer: C=US; O=DigiCert, Inc.; CN=RapidSSL Global TLS RSA4096 SHA256 2022 CA1
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha1WithRSAEncryption
* Connected to www.dessmonitor.com (8.210.123.202) port 443
* using HTTP/1.x
> GET / HTTP/1.1
> Host: www.dessmonitor.com
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.9.6
Nothing to add on the LAN side. Looks good.
I took a quick look at the TCP options that differ.
They revolve around some high-performance extensions.
Likely harmless, unless the edge router chokes on one of them (the wikipedia article about TCP window scaling mentions this possible side-effect):
https://en.wikipedia.org/wiki/TCP_window_scale_option (https://en.wikipedia.org/wiki/TCP_window_scale_option)
OTOH, you might have indicated that curl from W11 connected directly to the edge router works (too lazy to re-read).
In any case, I don't see how OPN is at fault here.
The LAN-WAN captures indicate a faithful reproduction of the traffic coming from the client.
Clearly, in some cases, the SYN frame is unanswered. In both cases (source HA or W11), the WAN frame is sent to an ASUS device (edge router I guess).
What happens from there on is unknown.
The frame could be dropped, not retransmitted from edge router WAN, lost on its way to the server, dropped by the server, ...
You might be able to follow that traffic a little further in your infrastructure if your gear has that capability.
That's the most likely source of issues. It's fully up to date, right?
When using Wireguard (to 3rd party server), all the traffic is encapsulated from OPN to the VPN server, so your edge router is not seeing the connection...
It's forwarded in UDP packets all the way to the VPN server and the connection is established from there.
I took a look at that code. They are not shy with the number of roundtrips.
I don't know how many devices you have but I now have an explanation for the repeated calls (from HA).
They weren't from chasing links in a web page, but merely a slew of async requests (get device list, then 4 requests per device).
It makes it unlikely there's throttling on the server IMO.
Still have no idea how to resolve this apart from bypassing opnsense [LAN] > [WAN].
The only thing left for you to investigate is what happens to that SYN frame (from W11) when it reaches the edge router.
I don't know if your edge router (ASUS?) allows packet capture.
If it can't maybe you have a switch that supports port mirroring (to mirror router WAN to a PC with WireShark).
Or maybe you have a spare machine on which you can run OPN as a filtering bridge (only long enough to capture).
If you don't see the frame out on edge_router WAN, you know where the problem is...
Otherwise, you would look for a reply SYN_ACK. If you don't get it, the problem is between your edge router and the web server.
In any case, the top-level page appears to be the default nginx page.
The integration code shows a proprietary authentication (some application-level with hardcoded string, some user-level based on creds your problem supplied to HA).
Is there more to this web site/service?
What are you actually trying to do?