I was able to narrow down the cause of the crash. I have openwrt router as next hop for some traffic and by default it rejects traffic instead of drop.
So whenever it rejects the traffic, it causes opnsense to crash. Changing it to drop stops it from crashing.
So whenever it rejects the traffic, it causes opnsense to crash. Changing it to drop stops it from crashing.