Packets dropped seemingly at random

[Soudini]

Hi,

I am running a proxmox server with currently a container exposed to the internet acting as a webserver and doing some NAT to access services behind it.
I am in the process of replacing my proxy by a vm running OPNsense but I am pulling my hair at some issues. 

NAT is working kinda well but I have a big issue where packets seems to be dropped at random : see attached screenshot.
It is breaking my nextcloud server by breaking the file transfers each time I try one.

Also I have an issue where I NAT the traffic on port 22 to my proxy to ssh into the others services but when I do, I have timeouts for my ssh connections.

Do you have any idea what I can do to fix that ?