New site PPPoE PMTU woes

[ToasterPC]

Not quite. Obviously, you can currently use an MTU of 1492 bytes only according to your tests. That I read from your previous posts and it hold true unless you succeed in applying the method explained here to enlarge that WAN MTU to 1500 bytes. In order to do that on a Proxmox VM, the whole chain ISP -> physical NIC -> Proxmox bridge -> OS NIC -> OS VLAN -> OS PPPoE WAN Interface must be configured right and capable to support 1500 bytes MTU on the WAN interface.

Without that, at least on the WAN side, you obviously need a 1492 bytes MTU, probably because of PPPoE involved in your WAN setup.

From there, you have two options:

1. Use a LAN MTU of 1500 bytes and employ MSS clamping (Firewall: Settings: Normalization) to adapt the mismatch of WAN vs. LAN MTU.
2. (Better) Use a LAN MTU of 1492 bytes, too.

Okay, so considering that I'm indeed not able to go past an MTU of 1492 on the WAN, I should set every underlying bridge and interface in Proxmox to the default value of 1500, and then within OPNsense set both the LAN and WAN values to 1492, such as that there's no need for normalization to be involved.

Assuming that's the case, the TLS handshake in cURL still fails for downstream devices (even for the Debian CLI of the Proxmox host itself:

Code Select Expand

curl -vv "https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/unifi-os-server.sh"
08:40:10.365721 [0-0] * Host raw.githubusercontent.com:443 was resolved.
08:40:10.365823 [0-0] * IPv6: 2606:50c0:8002::154, 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8001::154
08:40:10.365842 [0-0] * IPv4: 185.199.109.133, 185.199.111.133, 185.199.110.133, 185.199.108.133
08:40:10.365868 [0-0] * [HTTPS-CONNECT] adding wanted h2
08:40:10.365884 [0-0] * [HTTPS-CONNECT] added
08:40:10.365902 [0-0] * [HTTPS-CONNECT] connect, init
08:40:10.365940 [0-0] *   Trying [2606:50c0:8002::154]:443...
08:40:10.366006 [0-0] * Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
08:40:10.366026 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366042 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366056 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366082 [0-0] *   Trying [2606:50c0:8003::154]:443...
08:40:10.366109 [0-0] * Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
08:40:10.366134 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366156 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366179 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366203 [0-0] *   Trying [2606:50c0:8000::154]:443...
08:40:10.366229 [0-0] * Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
08:40:10.366251 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366274 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366294 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0 socks
08:40:10.366320 [0-0] *   Trying [2606:50c0:8001::154]:443...
08:40:10.366345 [0-0] * Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
08:40:10.366372 [0-0] *   Trying 185.199.109.133:443...
08:40:10.366412 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.366429 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.366451 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.367839 [0-0] * ALPN: curl offers h2,http/1.1
08:40:10.368293 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
08:40:10.373316 [0-0] *  CAfile: /etc/ssl/certs/ca-certificates.crt
08:40:10.373335 [0-0] *  CApath: /etc/ssl/certs
08:40:10.373401 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.373418 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.373439 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.426522 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.426532 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.426543 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:10.566743 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:10.566760 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:10.566778 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:11.567835 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:11.567871 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:11.567886 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:12.569072 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:12.569102 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:12.569111 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:13.570290 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:13.570320 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:13.570330 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
08:40:14.571508 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:40:14.571540 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:40:14.571549 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
Attempting to do the same TLS handshake under these conditions does succeed for the OPNsense VM:

Code Select Expand

sudo curl -vv "https://kindleforpc.s3.us-east-1.amazonaws.com/70980/KindleForPC-installer-2.8.70980.exe"
08:41:04.788684 [0-0] * Host kindleforpc.s3.us-east-1.amazonaws.com:443 was resolved.
08:41:04.788742 [0-0] * IPv6: (none)
08:41:04.788748 [0-0] * IPv4: 16.15.188.88, 16.15.183.33, 52.216.114.22, 52.216.58.250, 16.15.203.218, 16.15.192.221, 3.5.24.218, 16.15.218.61
08:41:04.788755 [0-0] * [HTTPS-CONNECT] adding wanted h2
08:41:04.788762 [0-0] * [HTTPS-CONNECT] added
08:41:04.788770 [0-0] * [HTTPS-CONNECT] connect, init
08:41:04.788782 [0-0] *   Trying 16.15.188.88:443...
08:41:04.788818 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.788824 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.788832 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.851700 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.851716 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.851726 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.873268 [0-0] * ALPN: curl offers h2,http/1.1
08:41:04.873357 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
08:41:04.873384 [0-0] * SSL Trust Anchors:
08:41:04.873392 [0-0] *   CApath: /etc/ssl/certs
08:41:04.873400 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.873407 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.873415 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.957856 [0-0] * TLSv1.3 (IN), TLS handshake, Server hello (2):
08:41:04.957985 [0-0] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
08:41:04.957997 [0-0] * TLSv1.3 (IN), TLS handshake, Certificate (11):
08:41:04.958649 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
08:41:04.958659 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
08:41:04.958667 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
08:41:04.959173 [0-0] * TLSv1.3 (IN), TLS handshake, CERT verify (15):
08:41:04.959236 [0-0] * TLSv1.3 (IN), TLS handshake, Finished (20):
08:41:04.959258 [0-0] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
08:41:04.959274 [0-0] * TLSv1.3 (OUT), TLS handshake, Finished (20):
08:41:04.959305 [0-0] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
08:41:04.959315 [0-0] * ALPN: server accepted http/1.1
08:41:04.959323 [0-0] * Server certificate:
08:41:04.959334 [0-0] *   subject: CN=s3.amazonaws.com
08:41:04.959343 [0-0] *   start date: Jul 20 00:00:00 2025 GMT
08:41:04.959350 [0-0] *   expire date: Jun 25 23:59:59 2026 GMT
08:41:04.959359 [0-0] *   issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
08:41:04.959372 [0-0] *   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959382 [0-0] *   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959389 [0-0] *   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
08:41:04.959402 [0-0] *   subjectAltName: "kindleforpc.s3.us-east-1.amazonaws.com" matches cert's "*.s3.us-east-1.amazonaws.com"
08:41:04.959413 [0-0] * SSL certificate verified via OpenSSL.
08:41:04.959421 [0-0] * [HTTPS-CONNECT] connect+handshake h2: 170ms, 1st data: 169ms
08:41:04.959428 [0-0] * [SETUP] query ALPN
08:41:04.959434 [0-0] * [HTTPS-CONNECT] connect -> 0, done=1
08:41:04.959441 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=1
08:41:04.959457 [0-0] * Established connection to kindleforpc.s3.us-east-1.amazonaws.com (16.15.188.88 port 443) from 187.224.2.21 port 21719
08:41:04.959464 [0-0] * [HTTPS-CONNECT] query ALPN
08:41:04.959471 [0-0] * using HTTP/1.x
08:41:04.959492 [0-0] > GET /70980/KindleForPC-installer-2.8.70980.exe HTTP/1.1
08:41:04.959492 [0-0] > Host: kindleforpc.s3.us-east-1.amazonaws.com
08:41:04.959492 [0-0] > User-Agent: curl/8.17.0
08:41:04.959492 [0-0] > Accept: */*
08:41:04.959492 [0-0] >
08:41:04.959528 [0-0] * Request completely sent off
08:41:05.078851 [0-0] < HTTP/1.1 200 OK
08:41:05.078863 [0-0] < x-amz-id-2: 3GV71S4y4G9s3w/HS3IFTdUoPJD7o7xfksXIHMhul9CIH1uOJsPnJm3A2mVIra6H6Zu5B38QL3HQ0dCqoIjkAeeO1tivc07n
08:41:05.078869 [0-0] < x-amz-request-id: 5CBHKGN17KSBEABE
08:41:05.078881 [0-0] < Date: Wed, 21 Jan 2026 14:41:06 GMT
08:41:05.078887 [0-0] < Last-Modified: Thu, 21 Aug 2025 10:56:57 GMT
08:41:05.078893 [0-0] < ETag: "2b756dcc3905a9ff3aef6a0a57dd7c09-18"
08:41:05.078898 [0-0] < x-amz-server-side-encryption: AES256
08:41:05.078904 [0-0] < Accept-Ranges: bytes
08:41:05.078910 [0-0] < Content-Type: application/octet-stream
08:41:05.078916 [0-0] < Content-Length: 298242024
08:41:05.078921 [0-0] < Server: AmazonS3
08:41:05.078927 [0-0] <
Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your
Warning: terminal anyway, or consider "--output <FILE>" to save to a file.
08:41:05.078952 [0-0] * client returned ERROR on write of 16384 bytes
08:41:05.078962 [0-0] * closing connection #0
If packets are not being modified nor fragmented by now, where else could the issue lie?

P.D. I'm not sure how relevant this could be, but the only reason I'm currently able to post to the forum is that I'm using a WireGuard road-warrior tunnel to the old site, in which regardless of the current site issues, encapsulating traffic through the VPN interface from the client device makes it able to go through and connect to problematic websites.

[ToasterPC]

Bump

[meyergru]

I cannot test this, because I neither have the OpnSense VM on PVE nor an MTU of 1492, sorry.

[ToasterPC]

I cannot test this, because I neither have the OpnSense VM on PVE nor an MTU of 1492, sorry.

Don't worry
So far, you've helped me narrow down the issue a lot, so thanks for everything either way.
However, I'm not sure if this problem has gotten to a point where perhaps a bug report or a different thread would be more appropriate.

MTU, MSS and PMTU do seem to be working correctly now, it's just the downstream devices that seem to still need something to get in line, and I'm honestly not sure where to begin looking for alternate solutions.

Assuming that either of those other options were viable, where would you begin and with which one would you pick?

[meyergru]

I would probably first try to make sure that the problematic downstream devices also use an MTU of 1492 bytes.

[ToasterPC]

I would probably first try to make sure that the problematic downstream devices also use an MTU of 1492 bytes.

Okay, sounds reasonable.

At the moment, I haven't tried setting DHCP option 26, but trying to ping while explicitly setting the ping from the VirtIO bridge is successful until an MTU of 1464:20/80.294/0.787 ms

Code Select Expand

ping -c 10 -M do -s 1464 kindleforpc.s3.us-east-1.amazonaws.com
PING kindleforpc.s3.us-east-1.amazonaws.com (52.217.120.18) 1464(1492) bytes of data.
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=1 ttl=246 time=78.8 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=2 ttl=246 time=79.1 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=3 ttl=246 time=78.1 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=4 ttl=246 time=79.4 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=5 ttl=246 time=80.3 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=6 ttl=246 time=79.7 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=7 ttl=246 time=79.7 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=8 ttl=246 time=78.0 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=9 ttl=246 time=78.0 ms
1472 bytes from s3-us-east-1-r-w.amazonaws.com (52.217.120.18): icmp_seq=10 ttl=246 time=78.2 ms

--- kindleforpc.s3.us-east-1.amazonaws.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9002ms
rtt min/avg/max/mdev = 77.965/78.920/80.294/0.787 ms
Attempting to do the same from a Windows client yields the following results:

Code Select Expand

ping -n 10 -l 1464 8.8.8.8                                                                                                                                                                                  ─╯

Pinging 8.8.8.8 with 1464 bytes of data:
Reply from 8.8.8.8: bytes=1464 time=11ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=9ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=9ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=8ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=8ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=9ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=8ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=8ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=12ms TTL=119
Reply from 8.8.8.8: bytes=1464 time=9ms TTL=119

Ping statistics for 8.8.8.8:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 8ms, Maximum = 12ms, Average = 9ms
Though neither of them is receiving any instructions to set the MTU to anything in particular, and trying to set the NIC in Windows to any of the aforementioned values doesn't make a difference.

Assuming then that the problem is in communicating the proper MTU to every device, what would be the proper place to set this network-wide (e.g. in OPNsense, for example)?

[meyergru]

If reducing the MTU size on your Windows client does not fix the problem, them maybe the MTU size is not the problem after all?

Did you try the ping to your OpnSense instance itself, too?

For modern Windows versions, I think they automagically set the MTU size - IDK how they do that exactly, however. I do not have the problem, as both my WAN and LAN MTUs are 1500 bytes.

[ToasterPC]

If reducing the MTU size on your Windows client does not fix the problem, them maybe the MTU size is not the problem after all?

Honestly that's quite likely, though I'm still unsure on how to test for such a scenario.

Did you try the ping to your OpnSense instance itself, too?

Yes, and it seems getting to the firewall itself has no issues with employing packets way above the interface MTU

Code Select Expand

Pinging 10.10.1.1 with 10000 bytes of data:
Reply from 10.10.1.1: bytes=10000 time=2ms TTL=64
Reply from 10.10.1.1: bytes=10000 time=32ms TTL=64
Reply from 10.10.1.1: bytes=10000 time=14ms TTL=64
Reply from 10.10.1.1: bytes=10000 time=2ms TTL=64
Reply from 10.10.1.1: bytes=10000 time=2ms TTL=64
Ping statistics for 10.10.1.1:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 32ms, Average = 10ms

For modern Windows versions, I think they automagically set the MTU size - IDK how they do that exactly, however. I do not have the problem, as both my WAN and LAN MTUs are 1500 bytes.

Tbh neither do I, but I do know how to set it manually if it's ever needed (thanks to this GitHub Gist):

From an Administrative Command Prompt/PowerShell session, use the following command to list the system's available interfaces and their current MTU values:

Code Select Expand

netsh interface ipv4 show subinterfaces
Which in my case outputs the following:

Code Select Expand

       MTU  MediaSenseState      Bytes In     Bytes Out  Interface
----------  ---------------  ------------  ------------  -------------
4294967295                1             0        169774  Loopback Pseudo-Interface 1
      1500                5             0             0  Onboard GbE
      1464                1    1872595478      40841253  WiFi
      1500                5             0             0  Local Area Connection* 1
      1500                5             0             0  USB 2.5GbE
      1280                1             0         17580  Tailscale
      1500                5             0             0  Local Area Connection* 2
     65535                5             0             0  Local Area Connection
      1500                1             0        120046  vEthernet (Default Switch)
      1500                1        189514        974851  vEthernet (WSL (Hyper-V firewall))
      1500                1          1968        151879  VMware Network Adapter VMnet1
      1500                1          1968        150820  VMware Network Adapter VMnet8
      1500                5             0             0  Bluetooth Network Connection
As such, after identifying the interface needing the change, the MTU can be set by using this other command:

Code Select Expand

netsh interface ipv4 set subinterface "WiFi" mtu=1464
If everything went as expected, the output will be:

Code Select Expand

Ok.