reordering packets under higher traffic

[Andreas_]

I'm running OpnSense 19.1 on Xen, connecting a DMZ host to its file server via NFS.
On rare occasions, when a big file is transferred, the nfs connection is broken, and a new tcp connection has to be started.

I've been tcpdumping the traffic in and out of the firewall (TCP segment offloading is disabled on all interfaces to avoid driver trouble), and found the following explanation:
Sometimes, a big PDU sent from the fileserver (split into 364 segments within 9.5ms) isn't forwarded to the destination DMZ host in-order, but instead in the middle of the flow segments are forwarded out-of-order, provoking out-of-order acks and resends, apparently driving the tcp stack mad and ultimately breaking the connection.

The server is a Xeon E5-2620V3, with 4 CPUs assigned to the firewall (low single digit cpu utilization, load rarely reaching 1), and no other machines running on the host. Typical state table size is 450, mbuf usage 800.

While the usage pattern of the system and general load hasn't changed over the last year, the problem started some months ago, which kind of coincides with the upgrade to 19.1 and the hardened kernel.

Why does the firewall start reordering, what can I do to prevent that?

Regards
Andreas

[mimugmail]

Do you have the chance to disable shared forwarding for testing?

[Andreas_]

Shared forwarding under Firewall/Settings/Advanced isn't enabled.
No traffic shaping, routing groups, advanced rules or other fancy stuff configured.