Possible PF Software Bug Causing Slowness [Solved, Lower the ARP Expiration]

[eneerge]

Hello All,

After a couple days of testing, I think there may be a software issue with PF. I've tried various configurations over a couple different hardware setups and keep experiencing the exact same issue. Full details below.

I currently have a fiber link of 1gbit up/1gbit down. Connecting directly to the fiber modem without going through the firewall, my speed tests run 950mbit up and 950mbit down. I have run the tests multiple times over a period of an hour to verify there's no deviation in that speed when connected directly to the modem.

Now, I have built a couple firewalls during my tests:
1) Intel Xeon E3-1220 v3 @ 3.10GHz, 250GB Samsung SSD, 8GB ram, 2x Intel CT Desktop NICs (Intel EXPI9301CTBLK)
2) Virtual Machine, AMD Ryzen 5950x. Assigned 8 threads to the HyperV VM. 128gb HD on 4x raid 10 SSDs. 8gb ram. Intel NICs

On both machines, I have the exact same experience:
1) I power up the firewall
2) I do a speedtest. The speedtests are 950mbit up/down like they are when directly plugged in
3) I watch some YouTube videos for about 5 minutes.
4) I do another speedtest. The speedtests are 600-700mbit down and only 60mbit up.
5) I reboot
6) Speedtest returns back to normal

I have performed the speedtests using a Windows 10 Machine and also a Windows Server 2019 machine. I have plugged the Ethernet cables directly from the test machines directly into the firewall LAN port (no middle switches).

Additionally, I have also tried installing pfsense on the same machines to see if it was something to do with opnsense. I experienced the exact same issue. Speed drops after the firewall has been online for a few minutes.

The performance seems to deviate. Occasionally it will come back up to the 950mbit, but the majority of the time the speed is slower. The upload rate is the primary issue. It is always below 100mbit for some reason.

I have tried enabling RSS in the tuneables. That did not help. I tried disabling Spectre and Meltdown mitigations. Disabling the meltdown mitigations for some reason causes it to run slower on the Xeon processor - Download never goes above 600mbit, but the upload seems to be a little faster than 60mbit when it goes into slow down mode.

I've tried enabling and disabling the "Hardware CRC", "Hardware TSO", "Hardware LRO" in the interface settings. I tried enabling/disabling interface scrubbing.

When performing speedtests, I watch the opnsense interface statistics to make sure the speeds match what the speedtests shows. They are very close to each other. This shows that there's no background activity occurring other than the speedtest.

Since I have this issue with opnsense and pfsense, the only thing that makes sense to me is an issue with PF. Anyone have a similar issue?

[eneerge]

When it enters "slow down" mode. The CPU doesn't even go over 1%. IE, upload is 60mbit and cpu is 1%. Prior to going into slow down mode, the CPU hits 25% when downloading and uploading. It's like there's a bottleneck somewhere.

[Patrick M. Hausen]

Have you tried disabling powerd? And disable all sleep states in the BIOS if that option available.

[eneerge]

I just swapped to 2 completely different nics in the 5950x box. I'm still experiencing the same slowness.

When I make a configuration change on the WAN interface (any change that reloads it), the speed returns back to normal.

PowerD is not enabled. I have tried enabling it and setting to conservative and max and that didn't change anything.

I am not using Suricata. I basically have a vanilla install, but configured 10.0.1.0/24 for the LAN.

I just bought an intel x540 dual lan nic, but I doubt this will help at this point since I've been through 4 NICs now that did not have any issue in the past.

[eneerge]

When I made LAN interface config changes, it reloaded the LAN interface, but the speed was still 60mbit upload.

I do the same with the WAN interface, speed goes back up to 1gbit/second.

Any config change on the WAN interface allows the speed to return to normal. That's the only thing I know for sure at this point.  Across 4 NICS (2 different intel nics, a realtek, and the hyperv virtual nics). Make any WAN config change and click apply, speed returns.

[eneerge]

I feel absolutely dingy now. I just setup a netfilter based distro (IPFire) on the same hardware. I do not have any performance slow downs now. Everything is 900mbit+ on up/down.

I really prefer OPNSense, but the slow downs are making it unusable at the moment.

[mimugmail]

You say you use vm? What happens when you install opnsense on the Hardware itself?

[eneerge]

Exactly the same thing. I've tried it on bare metal and got the exact same issue. Since changing to netfilter, it's been consistently 900mbps+.

[mimugmail]

Can you try setting MSS to 1300 in Interface : LAN?

[eneerge]

Could this be related? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268490

[eneerge]

After testing MSS settings. It had no effect. Tried on WAN and LAN.

[eneerge]

I will say changing the MSS settings on the LAN, reduced max throughput to about 700mbit up/down, but the throughput eventually dropped to 150mbit down and 60mbit up.

[dinguz]

Could you elaborate a bit more on the network topology? Are there things like PPPoE involved?

[eneerge]

I am using a very basic configuration at the moment while troubleshooting.

Fiber Modem -> Firewall WAN. Fiber modem gives OPNSense a WAN IP using DHCP. No PPOE. Just basic DHCP.
Firewall LAN -> Network Device. Firewall gives device an IP via DHCP (10.0.1.0/24)

I have tried the following NICs:

• Intel Gigabit CT Desktop Adapter (x2)
• Intel i210 Gigabit Adapter
• Realtek PCIe 2.5GbE Family Controller RTL8125

For every NIC, network device gets 900mbit+ for about 5-10 minutes before dropping to about 600mbps down and 60mbps up.

I have also tried different network cables (CAT8).

At the moment, I am running this in a HyperV virtual machine with 8 cores assigned with 4GB RAM. Before virtualizing the setup, I was running bare metal on a Xeon E3-1220 v3 @ 3.10GHz Quadcore with 128GB SSD and 8gb RAM. The exact same experience occurred on that hardware. However, I was using the Intel Gigabit CT Desktop NICs.

Now that I have everything virtualized, I've setup new VMs with the following:

• IPFire - Consistent 900mbps up/down with no performance degradation
• Endian- Consistent 900mbps up/down with no performance degradation
• Pfsense - 900mbps for 5-10mins before dropping to 600mbps down and 60mbps up. If the connection remains idle for a short period, performance returns for a few minutes before dropping again
• OPNSense - 900mbps for 5-10mins before dropping to 600mbps down and 60mbps up. If the connection remains idle for a short period, performance returns for a few minutes before dropping again

[mimugmail]

I cant imagine this happens on bare metal too