VPN performance, MTU sizes and Firewall Scrubbing

[Hildi]

I have the following setup:

- 2 Firewall computers, both with good performance (I am getting 300Mbit AES256 when directly connected with a switch).
- Internet Office: 200MBit down, 50MBit up, directly connected to the internet
- Internet Home: 1000MBit down, 50MBit up, but behind provider NAT

So I can only create an IPSec tunnel from Home->Office using tunneling, not routing at the moment.
Connection is up and stable.

I have no special setup for ICMP handling.

I had a bad performance from Office->Home with 2MBit and 30MBit in the other direction.

I tried to test the MTU size using `ping` but that did not work. So I used `iperf` on both sides of the tunnel. Found out that I get full 50MBit in both directions when I use an MTU of 1422.

I also had to set `NAT Traversal: Force` on the office side.

My first idea was to get the LAN MTU to 1422 and everything should work... but did not.
So my next try was setting the MTU through DHCP. Works for Linux. Does not work on macOS (the Macs simply ignore MTU settings in DHCP). It also seems that several mobile devices (connected to the firewall now have issues accessing some webpages).
I had no issues here when the MTU size was 1500.

I've also learned that the problem with testing the MTU size comes from the fact that OpnSense by default does "Interface scrubbing" which kills the "Do not fragment" flag from the IP packages and allows fragmented packages instead to allow access to servers which set do not fragment.

What I'd like to understand is

1) Do I have to allow ICMP in the firewall? Or is OpnSense already accepting the "packet too big" messages by default?

2) Should I set Firewall / Normalization / Disable interface scrub? Or better: What should I set or not?

3) Would using a routed tunnel solve the issue? Can I set the MTU on the tunnel interface instead?

4) Other ideas?


---

Update:

I've disabled "Interface Scrubbing" and set MTUs back to default. It seems my mobile devices and computers connected over the wifi can finally access all internet pages again.