[Solved] fitlet2 (celeron J3455): throughput down to 30% after 5 minutes

[meepmeep]

I just bought a fitlet2 box beause it looked like a small device perfect for handling opnsense at home.
I have a 300/300 connection, I don't need very high intensive cpu tasks (flat lan network, one wireguard outbound connection and some basic nat/pf filtering).

However, for an unknown reason, my throughput goes instable / very low after 5min of uptime.
At the beginning (after a reboot), I can easily reach 100% of my bandwidth (iperf3 test with public server / curl with 1G to /dev/null).

After about 5min, the throughput decrease : iperf test around 150, and few minutes after, 70~90mbps. And it stays that low until the next reboot.

There is no IDS/IPS services, the cpu usage is very low (around 10~20% max), I have no error / alert in the system log.

I tried with PowerD set to maximum and disabling it all together : same result.
I tried to change some bios settings (MWait disabled) : same result
I tried some tunables (see below to increase speed on igb nic) : same result

Where could I find some information which could explain this speed problem ? I don't understand why the decrease only happened after few minutes. (some queue full ? some cpu throttle ?)

Configuration :
- OPNsense 20.1.7-amd64 - fresh install
- Fitlet2 (celeron j3455) with 4 nic
-- Nic 1 : WAN Connection 300/300Mbps
-- Nic 2 : LAN network
- No ids/ips services
- No additional services / plugin / package[/li]

- tunables :

Code Select Expand

hint.hpet.0.clock 0
hint.ahci.0.msi 2
hint.ahcich.0.sata_rev 2
hint.sdhci_pci.0.disabled 1
hint.sdhci_pci.1.disabled 1
hint.sdhci_pci.2.disabled 1
hw.igb.rx_process_limit -1
hw.igb.tx_process_limit -1
legal.intel_ipw.license_ack 1
legal.intel_igb.license_ack 1

Edit;
- I also tried to activate / deactivate all hardware checksum (crc, two, etc.) : Same result
- all iperf / curl test are made from a server on lan subnet ( going through Nat)

[meepmeep]

New information : I discover that upload is even lower : around 4mbps (instead of 300Mbps).
And as soon as i started to upload a file, the download speed crashs (from 280mbps to 20mbps max).

There is clearly something wrong in my configuration / drivers / bios but I don't know what it is :(

edit : in netstat -s I have this :
3619604 packets forwarded (651210 packets fast forwarded)

is that normal the so few packet are fast forwarded ?

[meepmeep]

Update : the issue came from a mis-configuration from the dhcp vlan and priority. Nothing link to the hardware or configuration.

[klausdude]

Update : the issue came from a mis-configuration from the dhcp vlan and priority. Nothing link to the hardware or configuration.

I think I have a quite similar setup (fitlet2). What were your mis-configs? I am a real newbie to the DHCP, clan and priority stuff ...

[meepmeep]

Hi Klausdude

What is you ISP ? Mine is orange France with fiber connection.
The issue was link to the vlan priority (6).
I set it on the interface but the priority was only needed on the DHCP request (options modifiers on dhcp advanced configuration : vlan-parent "igb1",vlan-id 832, vlan-pcp 6)

[klausdude]

Hi meepmeep,

unfortunately, no fiber, just a cable connection with max 1 GB down and only 50 MB up.
I have not yet started with VLAN segmentation. It is good to know that I do not need to set these priorities on the interface. Do you use them for certain traffic types?