DEC3840 - very slow throughput

[NW4FUN]

Hello,

I'm fairly new to OPNsense as it has been sitting in my home lab for just a few months. I've had the 22.1.x community version running on 2 bare metals and it was all working just fine to the point I've decided to buy an "official appliance" - DEC3840 - for my prod environment.

This is where things started going south...

In a nutshell, this is running on 21.10.3 Business edition (I've noticed this is based on FreeBSD12 rather than 13) and my WAN pppoe throughput reaches ~300M at its best. I have a symmetric 10G line which was just working at full speed with the previous - supposedly lowered spec - HW.

I supposed I must have done something wrong? (I manually copied and pasted all of the previous settings and double/triple checked to make sure nothing was being forgotten)

Anyone might help me in troubleshooting this as I start thinking I might have received a faulty unit from Deciso...

Thanks for your support,

NW4FUN

[Patrick M. Hausen]

I think the Deciso hardware is really fine but install the community edition, anyway. In your situation I would try that first. While you are at it you can install with ZFS.

[NW4FUN]

Hey,

Thanks for your reply.

Why installing community edition and migrating to ZFS?

Can you walk me through the advantages compared to business edition on UFS.

Cheers,

NW4FUN

[Patrick M. Hausen]

ZFS is way more resilient to unexpected outages. It's 2022 - I run ZFS literally everywhere. With the exception of virtual machines, because the copy-on-write nature of ZFS thwarts storage overprovisioning.

As for the community edition - get new features faster. E.g. the FreeBSD 13 kernel and vastly improved bridging performance among other things.

HTH,
Patrick

[NW4FUN]

Thanks Bud,

Much appreciated. I’ve upgraded to 22.1.4_1 following your suggestion, however:

1) I wasn’t able to run a fresh install as it kept freezing at the same point (the only image able to be launched was SERIAL as the deciso appliance lacks of video output). Therefore couldn’t switch to ZFS (any suggestion appreciated)
2) the throughput hasn’t improved at as Speedtest from LAN sits at ~300M max while ran from cli sits at 1.9G
3) I can’t perform iPerf testing as client times out
4) cpu goes up to 100% very often without really doing anything

I’m really disappointed with this appliance so far, I keep thinking I’ve got a faulty unit and thinking of returning it.

My previous HW was running on i7-9700 with 16G Ram and it was flying. I was expecting at the very least same performances with the deciso 3840. Not the case.

I wish I could be able to run a clean install to rule out any potential misconfiguration or anything.

[mfpck]

1) I wasn’t able to run a fresh install as it kept freezing at the same point (the only image able to be launched was

I had similar issues once, The issue was related to the sense image and or the usb stick...So i downloaded the image again, verified it + used a different usb stick !

[NW4FUN]

Hey,

Thanks for your suggestion. I've tried 3 different USB dongles, always getting stuck at the same point.
Below what I get from the console (I guess tomorrow I'm going to initiate return, I'm insanely disappointed!!!)

Perhaps any of you guys have any idea on what I might be doing wrong??

Cheers

NW4FUN

Last login: Sun Mar 27 15:05:45 on ttys001

umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:1:0: Attached to scbus1
Root mount waiting for: usbus0 CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS256GMTS952T2 02J0T4GB> ACS-2 ATA SATA 3.x device
ada0: Serial Number G821760019
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 244198MB (500118192 512 byte sectors)
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <TDKMedia TF 150 Drive PMAP> Removable Direct Access SCSI device
da0: Serial Number 07B3080795C7178F
da0: 40.000MB/s transfers
da0: 3812MB (7806976 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
GEOM: da0: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
mountroot: waiting for device /dev/ufs/OPNsense_Install...
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled

[Patrick M. Hausen]

You are using the serial image?

[franco]

Should be serial already, the bigger question is if it was a 22.1.2 image or not (22.1 is buggy with FreeBSD 13) and on the device BIOS you need to disable legacy UART. It's all rather basic information that the existing customer support channel can also offer. 


Cheers,
Franco

[NW4FUN]

@franco you're spot on there!

Following the super caring advise of Deciso's support, I've disabled legacy UART in the BIOS and managed to run a fresh install. At least I've got that one down...

I'm still trying to get to the bottom of the very bizarre performances behaviour my machine is having.

I'm going to post a detailed report once I've finished troubleshooting so the community may benefit from my experience.

[NW4FUN]

alright....as promised, after a very long period of testing, troubleshooting, network changes, etc... I've found a configuration that is consistently solid in terms of performances.

Apparently, having WAN configured on igb(x) and LAN on ax(x) delivers very satisfactory performances in terms of throughput. The drawback is with the above config I limit my link to just 1G instead of 10G.

What I've found interesting tho (already informed the DECISO's guys) is that everything breaks if either:

1. WAN goes over ax0/1

OR

2. LAN goes over LACP ax0,ax1

Cheers

[Berzerker]

alright....as promised, after a very long period of testing, troubleshooting, network changes, etc... I've found a configuration that is consistently solid in terms of performances.

Apparently, having WAN configured on igb(x) and LAN on ax(x) delivers very satisfactory performances in terms of throughput. The drawback is with the above config I limit my link to just 1G instead of 10G.

What I've found interesting tho (already informed the DECISO's guys) is that everything breaks if either:

1. WAN goes over ax0/1

OR

2. LAN goes over LACP ax0,ax1

Cheers

I'm also on a DEC3840 and I've been going through the same issues with you, apparently. For your console issue, you need to disable Legacy UART in the BIOS https://docs.opnsense.org/hardware/serial_connectivity.html#legacy-uart-vs-uefi-serial. That solved my console output issues.

I have multi-gig WAN as well and having WAN on ax1 and LAN on ax0 causes poor routing performance for me, I can't seem to get past 2-3Gbps when traffic comes *into* the box first. iperf tests from an interface across VLANs is always maxing out 10Gb.

Seems to maybe be an issue if both ax0 and ax1 are in use?

[NW4FUN]

Interesting. I always thought this box wasn’t fit for purpose and tried to explain that to deciso.
I wonder whether the CPU is the culprit here?

[Berzerker]

Interesting. I always thought this box wasn’t fit for purpose and tried to explain that to deciso.
I wonder whether the CPU is the culprit here?

I don't think so, their tests clearly show >=10Gb throughput through the firewall. And doing iPerf from one VLAN to a device on a different VLAN from the opnsense CLI shows full 10Gb speed, so clearly the power is there, it's just when the traffic is coming into the box first is where I have performance issues.

Code: [Select]

[  5] local 10.0.30.1 port 16402 connected to 10.0.70.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.17 Gbits/sec    0   1.82 MBytes       
[  5]   1.00-2.00   sec  1.08 GBytes  9.24 Gbits/sec    0   2.53 MBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    0   3.00 MBytes       
[  5]   3.00-3.64   sec   701 MBytes  9.23 Gbits/sec    0   3.00 MBytes

[NW4FUN]

How are you measuring that?

IPerf plug-in on FW or IPerf.server-DEC3840-IPerf.client?