How to track down performance issues?

[Kreeblah]

I'm in the process of exploring a switch from pfSense to OPNsense, but I'm running into a really weird issue with my speeds that I'm having trouble tracking down.  Part of the complication here is that I purchased new hardware for this, so I'm not sure how much of it might be a hardware issue.

Hardware-wise, I'm running a Qotom Q20342G9-S20 (which has an Atom C3808 in it) with a pair of A-Tech SODIMMs (ECC, 32GB each, PC4-21300), a pair of mirrored 64GB Optane M10 SSDs for storage, and a pair of FS.com SFP-10GB-T-30 SFP+ modules for my LAN and WAN connections.  On paper, that should be fine for my connection, I'd think.  It's currently a symmetric gigabit connection, but I was thinking about upgrading to 5 gig once I have this set up.

The problem is, I've been noticing downloads are pretty slow.  And, I was able to verify that when I run a speed test using Ookla's CLI speedtest app from a server on my network, I get ~70-100mbps down, and 900+mbps up.  Running that from the OPNsense firewall itself (after having forcibly installed the FreeBSD 13 package that Ookla offers) gets me the opposite results: 900+mbps down, and ~70-100 up.  I find that to be really strange, and don't really know what to make of it.  Also, I can tell that it's not my connection, as if I connect my laptop directly to my ONT with a cat5e cable, I can get 900+mbps in both directions running the same tool.

So, something's up with either my hardware or my OPNsense configuration, and I'm not sure what.  I'm not running Suricata or any shapers, I turned off OpenVPN, and I'm using my connection much when I run these tests (maybe a few KB/s from other stuff).  I've tried with both offloading checksums and VLAN filtering to the NICs and leaving offloading disabled (with rebooting between changing settings).  None of it's really made a difference.

I can't see anything obvious in my resource use, either.  I don't ever have any cores that get past a few percent (unless I run a speedtest directly on the firewall, at which point one might get up to 70%), and I've got over 60 gigs of RAM free.

Does anybody have any thoughts as to what I might be missing here, or where to investigate from this point?  I'm pretty much out of ideas.

Edit: Oh, and I get my connection via DHCP from my ISP, so no bottlenecking on PPPoE to worry about.

[Kreeblah]

It figures I'd think of something right after I posted this.  It turns out this hardware is really picky (and I mean really picky) about SFP modules.  I had another couple of them from a different manufacturer that I was able to try, and they work . . . but only with specific modules (of nominally the same model number from the same manufacturer) in specific ports.  If I swap them around, I see the same behavior on my network, though the firewall is fine.

So, I guess I just have to hope neither of these two SFP modules or SFP ports dies.  That . . . doesn't feel great.

[newsense]

There's nothing extraordinary about Qotom, never was. There will always be better and cheaper alternatives as time goes by.

If you're so worried about SFP+ modules dying simply be proactive and have a few verified spares ready for an emergency.