DEC750 realistic 10G expectations

[ou1]

Hello community

I have a DEC750, a 2-year old model with 2.5G ports.

On the WAN side, I have a 10G XGS-PON module, 10G internet plan from my provider. The WAN interface requires a non-default VLAN tag.

For testing, I plugged my PC directly into a 10GBe transceiver in the DEC750 using default VLAN 1.

Doing internet speedtests, the highest speed I can achieve is 5.2Gbps (iperf3 with parallel streams, speedtest net, cnlab). Is this to be expected? I don't want to spend too much time trying to increase this if it's already maxing out the firewall throughput of the DEC750. What I haven't tried is disabling the firewall on the DEC750, but this falls into the "spending too much time" category since I'd never use it in that configuration.

Using the router provided by my ISP, I can reach just over 8Gbps in speed tests.

[OPNenthu]

I don't own one but the published port-to-port throughput for DEC750v1 and v2 (as tested with TCP, full-duplex, unknown # of streams) is 8.5Gbps, so for a pure firewall application without IDS/IDP I don't see why you couldn't get similar results.

Aside from that, I have doubts about the SFP+ modules and VLAN setup you mentioned.  A lot of variables there.

1) Is it reasonable to compare your XGS-PON module to the ISP modem and expect similar results?  Have you been able to separately test and verify the module on another router box and confirm that it could hit 8Gbps with your ISP?  (I'm not saying it doesn't- just that we can't assume.)

2) Do both of the transceivers negotiate properly?

3) Do you need any specific NIC and VLAN offload settings or tunables (e.g. RSS) that are recommended for the DEC750?  Maybe your WAN connection is also impacted by some overhead due to VLAN tagging.

4) AFAIK, there's no "default" VLAN in OPNsense.  Did you explicitly create one with VLAN ID 1?  Not that it matters really, but could be some VLAN filtering overhead issue as well (?)

--

Update: Just in case...

https://docs.opnsense.org/hardware/defaults.html

Looking trough the default config, I don't see any interface offload settings or tunables that aren't already default in OPNsense, except for maybe the ones related to Meltdown & Spectre mitigation.  However there have been threads here recommending to at least enable RSS.

https://forum.opnsense.org/index.php?topic=24409.0
https://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling

--

UPDATE 2:

Interesting recent thread here: https://forum.opnsense.org/index.php?topic=49030.0

Presumably both of your SFP+ modules are using the same speed and not mixing (you'll need to check) but I wonder if maybe you are hitting a thermal limit with XGS-PON on the DEC750.

[ou1]

Thanks for the reply

It seems I've gone a bit down the rabbit hole and decided to try something else, to remove the XGS-PON SFP+ adapter from the equation.

I've connected my old ONT (XGS-PON, 10Gbe) to the ISP fiber and connected my PC directly to the ONT.
set up VLAN 10 on my PC, got a provider non-routable IP, provisioned the new ONT, restarted it, got a public IP.

speed test: 8Gbps
https://www.speedtest.net/result/18844302123

disconnected PC from ONT, removed VLAN 10 from PC, plugged my DEC750 into the ONT using a 10Gbe adapter, plugged my PC into another 10GBe adapter. Repeated the same test (this time through the DEC750 and the ONT)
speed test: 4.3Gbps
https://www.speedtest.net/result/18844315297

These tests were 5min apart, there wasn't much chance for hitting thermal limits on the SFP+ modules, they were only both connected for about 2 minutes.

Answer to your questions:

1) I now removed the module completely.

2) I am assuming so, it's definitely going over 5Gbps sometimes, but just barely. That's with protocol overheads too, I'm only looking at data throughput.

3) VLAN 10 on WAN interface could play a role, but impossible to test without. I don't have any other 10G routers that I could use for internal testing. I've followed all the multi-gig guides, including all tunables. Hardware offloading is all disabled, but enabling made no performance improvements, only corrupted UDP packets.

4) I don't have a VLAN id 1, I just meant that ISP needed VLAN tag 10, and on the LAN interface I didn't set up any VLAN.

The last thing to test will be turning off the firewall on the DEC750 and using it as a router. I'm holding back a bit on that because I'm worried of locking myself out, hate the thought of restoring a config through the console. But this will be a last-resort test.

I will look through your links later today, thank you.

I don't own one but the published port-to-port throughput for DEC750v1 and v2 (as tested with TCP, full-duplex, unknown # of streams) is 8.5Gbps, so for a pure firewall application without IDS/IDP I don't see why you couldn't get similar results.

Aside from that, I have doubts about the SFP+ modules and VLAN setup you mentioned.  A lot of variables there.

1) Is it reasonable to compare your XGS-PON module to the ISP modem and expect similar results?  Have you been able to separately test and verify the module on another router box and confirm that it could hit 8Gbps with your ISP?  (I'm not saying it doesn't- just that we can't assume.)

2) Do both of the transceivers negotiate properly?

3) Do you need any specific NIC and VLAN offload settings or tunables (e.g. RSS) that are recommended for the DEC750?  Maybe your WAN connection is also impacted by some overhead due to VLAN tagging.

4) AFAIK, there's no "default" VLAN in OPNsense.  Did you explicitly create one with VLAN ID 1?  Not that it matters really, but could be some VLAN filtering overhead issue as well (?)

--

Update: Just in case...

https://docs.opnsense.org/hardware/defaults.html

Looking trough the default config, I don't see any interface offload settings or tunables that aren't already default in OPNsense, except for maybe the ones related to Meltdown & Spectre mitigation.  However there have been threads here recommending to at least enable RSS.

https://forum.opnsense.org/index.php?topic=24409.0
https://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling

--

UPDATE 2:

Interesting recent thread here: https://forum.opnsense.org/index.php?topic=49030.0

Presumably both of your SFP+ modules are using the same speed and not mixing (you'll need to check) but I wonder if maybe you are hitting a thermal limit with XGS-PON on the DEC750.

[OPNenthu]

I should have mentioned that you can confirm #2 with 'ifconfig' (the line starting with 'media:' on the interface).

Also from the GUI if you go to Interfaces->Overview->[WAN/LAN]->Details and look for the same (Media) as well as Line Rate.

Sorry I don't have more helpful hints.  Hopefully someone with one of these devices has some more thoughts, or you might also e-mail Deciso support.  Will be good for us to know the outcome for future upgrade decisions.