New site PPPoE PMTU woes

Started by ToasterPC, January 17, 2026, 02:36:42 AM

Previous topic - Next topic
Quote from: meyergru on Today at 09:06:16 AMNot quite. Obviously, you can currently use an MTU of 1492 bytes only according to your tests. That I read from your previous posts and it hold true unless you succeed in applying the method explained here to enlarge that WAN MTU to 1500 bytes. In order to do that on a Proxmox VM, the whole chain ISP -> physical NIC -> Proxmox bridge -> OS NIC -> OS VLAN -> OS PPPoE WAN Interface must be configured right and capable to support 1500 bytes MTU on the WAN interface.

Without that, at least on the WAN side, you obviously need a 1492 bytes MTU, probably because of PPPoE involved in your WAN setup.

From there, you have two options:

1. Use a LAN MTU of 1500 bytes and employ MSS clamping (Firewall: Settings: Normalization) to adapt the mismatch of WAN vs. LAN MTU.
2. (Better) Use a LAN MTU of 1492 bytes, too.

Okay, so considering that I'm indeed not able to go past an MTU of 1492 on the WAN, I should set every underlying bridge and interface in Proxmox to the default value of 1500, and then within OPNsense set both the LAN and WAN values to 1492, such as that there's no need for normalization to be involved.

Assuming that's the case, the TLS handshake in cURL still fails for downstream devices (even for the Debian CLI of the Proxmox host itself:
curl -vv "https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/unifi-os-server.sh"
08:40:10.365721 [0-0] * Host raw.githubusercontent.com:443 was resolved.
08:40:10.365823 [0-0] * IPv6: 2606:50c0:8002::154, 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8001::154
08:40:10.365842 [0-0] * IPv4: 185.199.109.133, 185.199.111.133, 185.199.110.133, 185.199.108.133
08:40:10.365868 [0-0] * [HTTPS-CONNECT] adding wanted h2
08:40:10.365884 [0-0] * [HTTPS-CONNECT] added
08:40:10.365902 [0-0] * [HTTPS-CONNECT] connect, init
08:40:10.365940 [0-0] *   Trying [2606:50c0:8002::154]:443...
08:40:10.366006 [0-0] * Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
08:40:10.366026 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366042 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366056 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366082 [0-0] *   Trying [2606:50c0:8003::154]:443...
08:40:10.366109 [0-0] * Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
08:40:10.366134 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366156 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366179 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366203 [0-0] *   Trying [2606:50c0:8000::154]:443...
08:40:10.366229 [0-0] * Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
08:40:10.366251 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366274 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366294 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366320 [0-0] *   Trying [2606:50c0:8001::154]:443...
08:40:10.366345 [0-0] * Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
08:40:10.366372 [0-0] *   Trying 185.199.109.133:443...
08:40:10.366412 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366429 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366451 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.367839 [0-0] * ALPN: curl offers h2,http/1.1
08:40:10.368293 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
08:40:10.373316 [0-0] *  CAfile: /etc/ssl/certs/ca-certificates.crt
08:40:10.373335 [0-0] *  CApath: /etc/ssl/certs
08:40:10.373401 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.373418 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.373439 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.426522 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.426532 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.426543 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.566743 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.566760 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.566778 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:11.567835 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:11.567871 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:11.567886 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:12.569072 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:12.569102 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:12.569111 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:13.570290 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:13.570320 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:13.570330 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:14.571508 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:14.571540 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:14.571549 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks

Attempting to do the same TLS handshake under these conditions does succeed for the OPNsense VM:
sudo curl -vv "https://kindleforpc.s3.us-east-1.amazonaws.com/70980/KindleForPC-installer-2.8.70980.exe"
08:41:04.788684 [0-0] * Host kindleforpc.s3.us-east-1.amazonaws.com:443 was resolved.
08:41:04.788742 [0-0] * IPv6: (none)
08:41:04.788748 [0-0] * IPv4: 16.15.188.88, 16.15.183.33, 52.216.114.22, 52.216.58.250, 16.15.203.218, 16.15.192.221, 3.5.24.218, 16.15.218.61
08:41:04.788755 [0-0] * [HTTPS-CONNECT] adding wanted h2
08:41:04.788762 [0-0] * [HTTPS-CONNECT] added
08:41:04.788770 [0-0] * [HTTPS-CONNECT] connect, init
08:41:04.788782 [0-0] *   Trying 16.15.188.88:443...
08:41:04.788818 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.788824 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.788832 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.851700 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.851716 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.851726 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.873268 [0-0] * ALPN: curl offers h2,http/1.1
08:41:04.873357 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
08:41:04.873384 [0-0] * SSL Trust Anchors:
08:41:04.873392 [0-0] *   CApath: /etc/ssl/certs
08:41:04.873400 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.873407 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.873415 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.957856 [0-0] * TLSv1.3 (IN), TLS handshake, Server hello (2):
08:41:04.957985 [0-0] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
08:41:04.957997 [0-0] * TLSv1.3 (IN), TLS handshake, Certificate (11):
08:41:04.958649 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.958659 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.958667 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.959173 [0-0] * TLSv1.3 (IN), TLS handshake, CERT verify (15):
08:41:04.959236 [0-0] * TLSv1.3 (IN), TLS handshake, Finished (20):
08:41:04.959258 [0-0] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
08:41:04.959274 [0-0] * TLSv1.3 (OUT), TLS handshake, Finished (20):
08:41:04.959305 [0-0] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
08:41:04.959315 [0-0] * ALPN: server accepted http/1.1
08:41:04.959323 [0-0] * Server certificate:
08:41:04.959334 [0-0] *   subject: CN=s3.amazonaws.com
08:41:04.959343 [0-0] *   start date: Jul 20 00:00:00 2025 GMT
08:41:04.959350 [0-0] *   expire date: Jun 25 23:59:59 2026 GMT
08:41:04.959359 [0-0] *   issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
08:41:04.959372 [0-0] *   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959382 [0-0] *   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959389 [0-0] *   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959402 [0-0] *   subjectAltName: "kindleforpc.s3.us-east-1.amazonaws.com" matches cert's "*.s3.us-east-1.amazonaws.com"
08:41:04.959413 [0-0] * SSL certificate verified via OpenSSL.
08:41:04.959421 [0-0] * [HTTPS-CONNECT] connect+handshake h2: 170ms, 1st data: 169ms
08:41:04.959428 [0-0] * [SETUP] query ALPN
08:41:04.959434 [0-0] * [HTTPS-CONNECT] connect -> 0, done=1
08:41:04.959441 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=1
08:41:04.959457 [0-0] * Established connection to kindleforpc.s3.us-east-1.amazonaws.com (16.15.188.88 port 443) from 187.224.2.21 port 21719
08:41:04.959464 [0-0] * [HTTPS-CONNECT] query ALPN
08:41:04.959471 [0-0] * using HTTP/1.x
08:41:04.959492 [0-0] > GET /70980/KindleForPC-installer-2.8.70980.exe HTTP/1.1
08:41:04.959492 [0-0] > Host: kindleforpc.s3.us-east-1.amazonaws.com
08:41:04.959492 [0-0] > User-Agent: curl/8.17.0
08:41:04.959492 [0-0] > Accept: */*
08:41:04.959492 [0-0] >
08:41:04.959528 [0-0] * Request completely sent off
08:41:05.078851 [0-0] < HTTP/1.1 200 OK
08:41:05.078863 [0-0] < x-amz-id-2: 3GV71S4y4G9s3w/HS3IFTdUoPJD7o7xfksXIHMhul9CIH1uOJsPnJm3A2mVIra6H6Zu5B38QL3HQ0dCqoIjkAeeO1tivc07n
08:41:05.078869 [0-0] < x-amz-request-id: 5CBHKGN17KSBEABE
08:41:05.078881 [0-0] < Date: Wed, 21 Jan 2026 14:41:06 GMT
08:41:05.078887 [0-0] < Last-Modified: Thu, 21 Aug 2025 10:56:57 GMT
08:41:05.078893 [0-0] < ETag: "2b756dcc3905a9ff3aef6a0a57dd7c09-18"
08:41:05.078898 [0-0] < x-amz-server-side-encryption: AES256
08:41:05.078904 [0-0] < Accept-Ranges: bytes
08:41:05.078910 [0-0] < Content-Type: application/octet-stream
08:41:05.078916 [0-0] < Content-Length: 298242024
08:41:05.078921 [0-0] < Server: AmazonS3
08:41:05.078927 [0-0] <
Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your
Warning: terminal anyway, or consider "--output <FILE>" to save to a file.
08:41:05.078952 [0-0] * client returned ERROR on write of 16384 bytes
08:41:05.078962 [0-0] * closing connection #0

If packets are not being modified nor fragmented by now, where else could the issue lie?

P.D. I'm not sure how relevant this could be, but the only reason I'm currently able to post to the forum is that I'm using a WireGuard road-warrior tunnel to the old site, in which regardless of the current site issues, encapsulating traffic through the VPN interface from the client device makes it able to go through and connect to problematic websites.