rsync/scp over IPv4 much slower than over IPv6

[_nwb]

This is perhaps the strangest thing I've yet to figure out. I just narrowed it down to it being an issue with IPv4.

If I attempt to rsync any data to a server (or SSH into it and retrieve from here), whether directly over SSH or via WireGuard, it rarely surpasses 2MB/s, usually around 1.5MB/s. If I rsync by the IPv6 address to the remote server, I can get 30-40 MB/s. If the WireGuard link is connected by IPv6, then which IP type I use works regardless, since it's all being routed over an IPv6 network.

I thought it was an MTU issue maybe or firewall? I'm unsure, because the issue persists regardless of WireGuard. I just mentioned WireGuard as one of the things I've tried, and as long as the connection is made by IPv6, whether by a WireGuard bridge or by rsync directly, it works correctly.

I have a 1 Gb/s symmetrical fiber connection. Traffic shaping is enabled, but after having disabled it, it made no difference.

Supplementary information:
CPU type: Intel(R) Pentium(R) CPU G4400 @ 3.30GHz (2 cores, 2 threads)
Version: OPNsense 22.7.10_2-amd64
Driver: igb (Am using an Dell-branded Intel i350-T2 NIC)

WAN is set to DHCP and DHCPv6. There is a DHCPv6 config override, because AT&T's gateway doesn't hand out IPv6 correctly, and it's a workaround https://github.com/lilchancep/att-pfsense-ipv6

DNS servers are set to 1.1.1.1, 1.0.0.1, 2606:4700:4700::1111, and 2606:4700:4700::1001. Although I don't think that matters much given I was not using domains to test.

Essentially what I don't understand is why my transfers over IPv6 are sufficiently fast (~30-40+ MB/s), but the IPv4 connection only sustains <2MB/s transfer. I have also tried with

Code: [Select]

rsync -e "ssh -o Compression=off" which doesn't seem to make a difference.

Further, I have also tested using

Code: [Select]

wget -6 https://speed.hetzner.de/1GB.bin and

Code: [Select]

wget -4 https://speed.hetzner.de/1GB.binIn the case of wget, it works properly.

Please let me know if any logs or whatnot are needed. I'm pulling my hair out. thanks

edit: I want to add--retrieving files by IPv4 from the remote servers works fine. Only uploading data from this side is slow over IPv4. The endpoint is irrelevant. It happens for any remote server of mine. Make it make sense. I'm pulling my hair out

I further confirm it's a local issue, because the issue is non-existent when making a transfer over my IPv4-only AT&T hotspot.

Slow NAT??

[_nwb]

I have since discovered that IPv4 speed tests to ovh.us and with the Ookla Speedtest app are much slower on the upload, around 150 Mbit/s, verus ~900 Mbit/s upload on an IPv6-only transfer.

Further, the issue only exists on IPv4 TCP upload tests (as done in iperf3, like ~20 Mbit/s), while IPv6 TCP upload tests are >600Mbit/s.

Rsync/SSH is TCP, which makes sense as to why I first discovered the issue on IPv4 rsync.

I've been trying to Google this issue, but perhaps my Google-fu isn't good enough. Because I can't find much of anything, much less any solution.

edit: installed iperf3 directly on OPNsense firewall. Same issue.

edit 2: Somehow it's the AT&T modem. *shrug* No clue where to start now.