I finally figured this out this week and was able to validate it, I hope this helps someone in the future. The Qotom box doesn't supply the correct voltages, and the distance of the DAC cable was causing my issue. I was using a 1-meter DAC cable from 10Gtek and that caused the issue. Once I tested with a 0.3-meter DAC cable my speeds were normal on 1Gb. I ordered a 1 meter AOC (Active) DAC cable and I've installed the router today and so far it's stable. Time will tell, though from testing on multiple ports and multiple reboots, it's been stable. With the unstable setup it would have already failed.
This thread saved me: https://forums.servethehome.com/index.php?threads/qotom-denverton-fanless-system-with-4-sfp.41331/page-14#post-419760
This thread saved me: https://forums.servethehome.com/index.php?threads/qotom-denverton-fanless-system-with-4-sfp.41331/page-14#post-419760