IPSec routed mode - fragmentation issues

[pmladenov]

I have a very basic setup:

host1 <-> (em0) opnsense1 (em1) <----- ipsec routed mode -----> (em1) opnsense2 (em0) <-> host2

host1 is able to ping host2 will small packets but not with 1500bytes packet.
MTU size is default everywhere (1500 bytes and ipsec interface has MTU 1400 by default).
When I disable "interface scrub" (Firewall -> Settings -> Normalization) on opnsense1 firewall (ONLY!) everything starts working. Strange thing is that I don't touch that setting on opnsense2 firewall (we have scrub enabled there, as per default).

When host 1 send 1500 bytes packet it's received by opnsense1, fragmented to 2 packets (1400 bytes and 100 bytes) and send over the ipsec interface. It's getting received on opnsense2, re-assembled and forwarded to host2 as 1500bytes packet.
The problem is in the opposite direction - ICMP echo reply from Host2 is received by FW2, fragmented and sent via the ipsec interface to FW1. FW1 received both fragments on ipsec interface, combine them into a single 1500 byte packet and send it to Host1.  And here is the problem:

According to tcpdump on em0 interface of FW1, that ICMP 1500byte packet has a WRONG checksum (and at the end host1 is not receiving the replies from host2).

I've spent 2 full days in troubleshooting that, simplifying the setup the to above one. Both opnsense1 and 2 initially were 20.7.5, I upgraded the opnsense1 to the latest 20.7.X (no luck), then I completely reinstalled (from scratch without importing configs) opnsense1 VM, it didn't help at all, after that I deleted it and install 21.1 image but it didn't help either...
I think I'm missing something, it's really strange that in the other direction (host1->host2) everything is working (no bad checksums for re-assembled packets), no need to disable PF scrub on opnsense2 fw.

Any idea what I'm missing because I'm completely out of ideas anymore?
Anything else I can try to troubleshoot that problem?

If I have to disable scrub at all - what will happen with TCP MSS? I don't think it will be negotiated to 1360bytes which will definitely break many apps.




 

[pmladenov]

And the other strange thing:

When I ping host1 (10.70.10.100) from host2 (10.30.10.100) with 1500 byte packet - only the first one gets reply:

17:29:30.832068 IP (tos 0x0, ttl 253, id 8022, offset 0, flags [none], proto ICMP (1), length 1500)
    10.30.10.100 > 10.70.10.100: ICMP echo request, id 39, seq 0, length 1480
17:29:30.833003 IP (tos 0x0, ttl 254, id 8022, offset 0, flags [none], proto ICMP (1), length 1500)
    10.70.10.100 > 10.30.10.100: ICMP echo reply, id 39, seq 0, length 1480
17:29:30.834352 IP (tos 0x0, ttl 253, id 8023, offset 0, flags [none], proto ICMP (1), length 1500, bad cksum 5006 (->6f9e)!)
    10.30.10.100 > 10.70.10.100: ICMP echo request, id 39, seq 1, length 1480
17:29:32.835360 IP (tos 0x0, ttl 253, id 8024, offset 0, flags [none], proto ICMP (1), length 1500, bad cksum 5005 (->6f9d)!)
    10.30.10.100 > 10.70.10.100: ICMP echo request, id 39, seq 2, length 1480

This is again captured on opnsense-fw1 port towards host1.

[mimugmail]

This is a known bug/limitation of FreeBSD

[pmladenov]

This is a known bug/limitation of FreeBSD

Thanks for letting me know mimugmail.

So in that case I guess the workaround will be to simply to disable "scrub" on interfaces AND create a new scrub rule to set the TCP MSS to 1300 (for example).
I haven't read the last sentence of help

"Disable all default interface scrubing rules, mss clamping will also be disabled when you check this. Detailed settings specified below will still be used."

and was with wrong impression that disabling scrub will disable it completely

Meanwhile I found out the reason for "asymmetrical" ping behavior in my test lab. During my previous tests I have left a "Passthrough networks" subnet in the advanced VPN config on opnsense-FW2. I'm not sure how this is related, but when I add Host1 subnet in FW1 as a Passthrough networks and Host2 subnet in FW2 I can successfully send 1500 bytes ICMP packets (in both directions) without deactivating interface scrub, however no more than 1500 bytes..
 
Regards,
Plamen

[Ricardo]

This is a known bug/limitation of FreeBSD

Hmm, is this documented somewhere?

[Patrick M. Hausen]

Hmm, is this documented somewhere?

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744

[Ricardo]

Thanks. Another nail in the "ipsec transport mode on fbsd" coffin.