MTU size/fragmentation issue

[richardd]

Hi,
I have a strange issue with Opnsense in terms of fragmenation/MTU size.
Following setup

Host (192.168.44.32)
         |
internal router (MTU on physical int 9216, ip mtu on sub-int 1500)
         |
Switch (MTU 9198) - conenction between router and opnsense handled in VLAN 1131
         |
opnsense (igb3 MTU 9216, VLAN int: mtu 1500)
         |
IPSEC Tunnel to remote customer site
         |
Host (192.168.90.9)

Everything is fine with a MTU up to 1500 (even with DF set). In my further test with MTU 1501 of course no DF is set!
If I ping with MTU 1501 (Cisco notation including IP header size) I do not get an answer any more.
I did a capture on the VLAN between router and opnsense, on the VLAN interface directly on opnsense and on the IPSEC interface.
icmp request: Router is fragmenting, Opnsense is forwarding - everything is ok
icmp reply: fragmented icmp reply appears on IPSEC, packet is obviously reassembled and not forwarded any more: I see packet on capture of VLAN interface of FW but it never appears on switch although L2 MTU is much bigger!
Unfortunately I cannot attach traces due to filesize.
Is it normal that a device within routing path is reassembling?
Is it a problem for opnsense if mtu on main interface is not the same as on vlan interface?

running following version: OPNsense 21.7.7-amd64

any idea any answers are appreciated.
Thanks Richard

[richardd]

Hi,
no one with this problem? I have this issue on several interfaces as well as on a different opnsense box.
Regards Richards

[richardd]

ttt

[franco]

Hi Richard,

Not sure if this is the solution, but:

PMTU for IPsec was added to 23.7, yet you need to enable it.

https://docs.opnsense.org/manual/vpnet.html#path-mtu-discovery


Cheers,
Franco

[richardd]

Hi,
I did now the upgrade to 23.7.3 but still I have the issue. When a packet arrives through an IPSEC tunnel which originally was > 1500 Byte it comes fragmented but before Opnsense is forwarding it to the out interface it reassembles the packet and then silently drops it on the out interface instead of fragmenting it.
I have tested this as well with a normal Ethernet connection instead of an IPSEC tunnel and there it is ok.
Somehow I do not understand this reassembling of the payload packet when packet arrives through IPSEC tunnel.
Any further idea?
Thx Richard

[franco]

Well did you enable it?

[richardd]

Sorry, but I cannot find this possiblity.
I think it should be in system settings tuneable but there it is not.

These are the only ipsec tuneables:
net.enc.in.ipsec_bpf_mask    IPsec input bpf mask    runtime    default (2)    
net.enc.in.ipsec_filter_mask    IPsec input firewall filter mask    runtime    default (2)    
net.enc.out.ipsec_bpf_mask    IPsec output bpf mask    runtime    default (1)    
net.enc.out.ipsec_filter_mask    IPsec output firewall filter mask

Any hint for me?
Thx Richard

[Maurice]

You can add additional tunables with the + button.

Cheers
Maurice

[richardd]

Hi,
many thanks. I added the tunables but this did not change the strange behaviour. I think it was a try but PMTU is in my mind a different story as I am always testing without DF bit set. So it is still a topic of fragmentation and reassembling. Still do not get the point why after receiving the FW is reassembling packet.
Here the packets received from IPSEC tunnel:
two packets because fragmented:
Frame 14: 1432 bytes on wire (11456 bits), 1432 bytes captured (11456 bits)
Enc IPv4, SPI 0xc1f65146
Internet Protocol Version 4, Src: 192.168.90.9, Dst: 192.168.44.32
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 1420
    Identification: 0x0541 (1345)
    001. .... = Flags: 0x1, More fragments
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..1. .... = More fragments: Set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 61
    Protocol: ICMP (1)
    Header Checksum: 0x4bb6 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.90.9
    Destination Address: 192.168.44.32
    [Reassembled IPv4 in frame: 15]
Data (1400 bytes)

Frame 15: 113 bytes on wire (904 bits), 113 bytes captured (904 bits)
Enc IPv4, SPI 0xc1f65146
Internet Protocol Version 4, Src: 192.168.90.9, Dst: 192.168.44.32
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 101
    Identification: 0x0541 (1345)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 1010 1111 = Fragment Offset: 1400
    Time to Live: 61
    Protocol: ICMP (1)
    Header Checksum: 0x702e [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.90.9
    Destination Address: 192.168.44.32
    [2 IPv4 Fragments (1481 bytes): #14(1400), #15(81)]
Internet Control Message Protocol

and here the packet before leaving the FW on a LAN (with default MTU of 1500):
Frame 12: 1515 bytes on wire (12120 bits), 1515 bytes captured (12120 bits)
Ethernet II, Src: Deciso_00:34:60 (f4:90:ea:00:34:60), Dst: Cisco_9f:f4:6b (00:00:0c:9f:f4:6b)
Internet Protocol Version 4, Src: 192.168.90.9, Dst: 192.168.44.32
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 1501
    Identification: 0x0540 (1344)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 60
    Protocol: ICMP (1)
    Header Checksum: 0x6c66 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.90.9
    Destination Address: 192.168.44.32
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xb8b6 [correct]
    [Checksum Status: Good]
    Identifier (BE): 573 (0x023d)
    Identifier (LE): 15618 (0x3d02)
    Sequence Number (BE): 3 (0x0003)
    Sequence Number (LE): 768 (0x0300)
    [Request frame: 11]
    [Response time: 6,693 ms]
    Data (1473 bytes)

If FW is reassenmbling it then FW should fragment it when sending it out but this packet never is seen on the wire

regards Richard