Messages

For reference. I have a Calix Gigapoint 803g OTP that my fiber runs into.
Calix 803g -> Opnsense -> Switch -> Devices

Anyone that could enlighten me as to why this fixed the issue, please feel free to do so. This just seems odd that deleting a cached entry and creating the exact same entry every 540 seconds instead of every 1200 seconds fixes the issue.

I do not want to "jenks" myself, but holy f-ing s, it seems to be fixed after changing the ARP expiration timeout.

When using OpenWrt, I never experienced any slow downs. Linux by default has an ARP expiration of 60 seconds. Pf/Opnsense has a default expiration of 20 minutes. At exactly 600 seconds into the ARP response was when I started experiencing the slow downs. With pfsense, I was able to manually remove the individual cached entry for the gateway. I removed the ARP entry for my gateway and that instantly restored my speeds (and also caused it to instantly create a new ARP entry). So now I can apply the same to Opnsense.

I  just added this to tuneables:

• net.link.ether.inet.max_age = 540

This should set the ARP cache to expire every 9 minutes instead of 20.

I don't understand why this works. The MAC address of my gateway is the exact same even after the expiration and renew. Anyone have any idea why an old ARP cache entry (which is actually still valid) would cause this issue?

So, I just stumbled upon this post on reddit that mentioned the EXACT issue I described (speed dropping to the exact same rate). https://www.reddit.com/r/HomeNetworking/comments/p63zbo/calix_ont_to_3rd_party_router_not_working/

Last post mentions that it's caused by the ARP timeout. Maybe this needs to be statically assigned to resolve. I will test at some point.

I had performance issues as well (https://forum.opnsense.org/index.php?topic=31680.0). I've since moved to a Linux based product. I experienced the same issue on pfSense as well.

My issue occurs only with 1gbit up/down. Initially, I get full 1gbit up/down. However, it will eventually go into "slow mode" and the download will be like 500-600mbit and the upload only 50mbit. I only get this when I use a pf based firewall. Updated all ethernet firmware as well. Now that I've swapped to something netfilter based, I have 0 hickups.

The best way I could reproduce the issue was to go to youtube and then click on several videos in quick succession for about 5-10 minutes. Eventually, it would just slow down. If I rebooted the system, speed would come back. If I disabled the interface and reenabled it, the speed would come back. Tried various tweaks/tunes to no avail.

What I'm running now doesn't have the features that OpnSense does. I miss it, but I can't deal with the major slow down.

Just installed the Intel x540-t2. Exact same issue.

Tried using the tunables Kirk recommended in that second post. Didn't make any difference, unfortunately. :(

I tried these settings to no avail.   :(

I am using a very basic configuration at the moment while troubleshooting.

Fiber Modem -> Firewall WAN. Fiber modem gives OPNSense a WAN IP using DHCP. No PPOE. Just basic DHCP.
Firewall LAN -> Network Device. Firewall gives device an IP via DHCP (10.0.1.0/24)

I have tried the following NICs:

• Intel Gigabit CT Desktop Adapter (x2)
• Intel i210 Gigabit Adapter
• Realtek PCIe 2.5GbE Family Controller RTL8125

For every NIC, network device gets 900mbit+ for about 5-10 minutes before dropping to about 600mbps down and 60mbps up.

I have also tried different network cables (CAT8).

At the moment, I am running this in a HyperV virtual machine with 8 cores assigned with 4GB RAM. Before virtualizing the setup, I was running bare metal on a Xeon E3-1220 v3 @ 3.10GHz Quadcore with 128GB SSD and 8gb RAM. The exact same experience occurred on that hardware. However, I was using the Intel Gigabit CT Desktop NICs.

Now that I have everything virtualized, I've setup new VMs with the following:

• IPFire - Consistent 900mbps up/down with no performance degradation
• Endian- Consistent 900mbps up/down with no performance degradation
• Pfsense - 900mbps for 5-10mins before dropping to 600mbps down and 60mbps up. If the connection remains idle for a short period, performance returns for a few minutes before dropping again
• OPNSense - 900mbps for 5-10mins before dropping to 600mbps down and 60mbps up. If the connection remains idle for a short period, performance returns for a few minutes before dropping again

I will say changing the MSS settings on the LAN, reduced max throughput to about 700mbit up/down, but the throughput eventually dropped to 150mbit down and 60mbit up.

After testing MSS settings. It had no effect. Tried on WAN and LAN.

Could this be related? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268490

Exactly the same thing. I've tried it on bare metal and got the exact same issue. Since changing to netfilter, it's been consistently 900mbps+.

I just posted this same issue.

What did you mean by dupe? Did you find another post where this happened (duplicate)?

I feel absolutely dingy now. I just setup a netfilter based distro (IPFire) on the same hardware. I do not have any performance slow downs now. Everything is 900mbit+ on up/down.

I really prefer OPNSense, but the slow downs are making it unusable at the moment.

When I made LAN interface config changes, it reloaded the LAN interface, but the speed was still 60mbit upload.

I do the same with the WAN interface, speed goes back up to 1gbit/second.

Any config change on the WAN interface allows the speed to return to normal. That's the only thing I know for sure at this point.  Across 4 NICS (2 different intel nics, a realtek, and the hyperv virtual nics). Make any WAN config change and click apply, speed returns.