Messages

I ended up switching to WireGuard, however I also noticed it being very slow when copying numerous files from the NAS to the local host (~3kb/s).
After adjusting MTU to around 1300 and attempting to download a 700Mb file, I was getting around 25-30Mb/s (250Mbps).

So in practice it looks like everything's working as expected; I'm guessing that listing files in SMB over VPN is just one of those heavy tasks to process likely causing overhead in different places (mounted NAS drive on Windows).

The other thing I noticed is that while I do get 3Gbps up/down and 1Gbps up/down at home and here in Europe, after trying a speedtest between Europe and a home based speed test server, I could only get 250Mbps down, which is consistent with the large file transfer speed I'm getting on download.

Bottom line, it looks like my expectation of speed was skewed by the local speedtests, when in fact I should have considered the full scope speed test to a distant server.

I haven't checked the OpenVPN speeds, but I expect they'd align with what I'm getting on WireGuard right now, given the bandwidth limitation is between distant hops.

Try setting the OpenVPN port to 443 UDP and see if you can get lost in the static.

I tried setting the internet facing port to 443 (router port forwarding) on the modem, it made no difference, speeds are the same. OPNSense is set up on the DMZ, so there's no filtering between the modem and OPNSense.

On what port speed is your WAN connecting to the ISP modem/router ? 3Gb is not a standard port speed, so I'm wondering if ISPs give out now modems with 5Gb or 10GB NICs for their residential clients.

The modem has a 10Gb port, which is connected to a matching 10Gb NIC on OPNSense, I consistently get 3Gbps measured twice a day. I don't think raw available bandwidth is an issue.

Hello,

I've been trying just about anything I can think of to solve this issue.

I have set both a legacy and a new instance of OpenVPN, I am able to connect to both instances without issues using Viscosity, however when it comes to transferring data, the transfer speeds are incredibly slow, around 3KB/S (0.024Mbps).
I only realized this was an issue when I connected to the VPN and attempted to do some work.

My Home connection is 3Gbps up/down (measured daily using using the speedtest plugin), the internal network is 10Gbps.

My home servers are in Canada, I'm currently in Europe on a 1Gbps (up/down) connection.

The OPNsense box runs on an Intel i7-7700K with 16Gb of RAM and a bunch of 10Gb NICs.

I've mainly tried adjusting MTU/MSS with no change; tried some of this too: https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

I've primarily tested different configs on the Instance type server (not on the legacy), since I don't want to lose connectivity to the main server and not be able to connect.

I can't think of anything that would be causing such a drastic drop in speed or preventing OpenVPN to be a lot faster than 3kb/s...

Any help with this would be greatly appreciated!

Thanks!

Yes, I ended up using the speedtest-cli on the second post in this thread, I'm still seeing a 400Mbps drop on download while getting the full 3.2Gbps on the upload.

Not sure what kind of overhead would be causing that, iperf3 between the win machine and opnsense is around 9.5+ Gbps

My opnsense setup is as follows:

Internet [10Gbe] <=> [10Gbe ix1] opnsense [10Gbe ix2] <=> [10Gbe Onboard] Win 11

Hello,

I just experienced a strange issue today.
My opnsense froze, I noticed a message `ketn.ipc.nmbjumbop` limit reached on the screen.

At the time this happened, I was running a download at around 3.2Gbps (~400Mb/s) through opnsense on one of my servers (both with 10Gbe, 9000MTU, iperf3 clocking at 9.89Gbps between opnsense and the server).
Due to a completely unrelated issue (different problem altogether associated with the ISP - due to some throttling), the connection dropped a couple of times and then opnsense froze (no direct input on console and I noticed the message I mentioned above).
The ISP throttling/connection dropoff lasts for about 6min before restoring the internet. So it goes around 5min on, 6 min off.

I tried to change the tunables to:
- kern.ipc.nmbclusters = 1048576
- kern.ipc.nmbjumbop = 524288
- kern.ipc.nmbjumbo9 = 524288

This didn't have any effect, whenever the ISP connection would drop after a couple of drops the system would freeze, this time with no message on screen.

The opnsense hardware is as follows:
- CPU: i7-7700K
- RAM: 16Gb

This is the first time I've experienced with issue. It started happening with the optimized MTU for the NICs in order to reach 10Gbps on the network.

Any help with this would be greatly appreciated.

Using the speedtest CLI on windows seems to make things a lot better, however I'm still seeing a 400Mbps difference on download whereas I'm getting the full speed on upload (see screenshot)

Hello,

I'm trying to figure out if I'm doing something wrong in my environment.

Before I describe the issue, here is some context:
- I'm running opnsense as the gateway to the internet
- The appliance has 10Gbge interfaces
- iperf3 testing between opnsense and my windows machine runs at around 9.2Gbps up/down
- Internet runs at 3gbps, opensense speedtest measures 3.2Gbps up/down (daily measurements) through a 10Gbe interface

The issue that I'm experiencing is that when I run a speed test on the windows machine (AV and firewall disabled), I'm only seeing around 2.4Gbps even though the throughput on the firewall seems to be capable of handling the entire 3Gbps (based on iperf3 results).
I'm running speedtest through the browser (same server as the one used by opnsense).

I'm not sure what could be causing the drop, or whether I'm performing the test incorrectly.

Any insight would be greatly appreciated.

Thanks!

** Originally posted here: https://forum.bell.ca/t5/Internet/Home-Hub-4000-bridge-drops-internet-when-using-the-full-3Gbps/m-p/5821

Hello,

Yesterday I spent a bit over 1h45 with a Bell Internet support rep and we weren't able to find a concrete solution.

I'm curious to see if anyone else has experienced something similar and trying to get some insight into the issue.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense linux system using the 10G WAN port and going into a 10G NIC on the Opnsense firewall. The internet uplink is achieved using PPPoE (with Bell login info - b1 id and password) through the 10Gb WAN port.

I have a Bell Fibe Internet 3Gbps.

Up to here, everything works flawlessly:



My speed tests on the opnsense box are consistently around 3Gbps, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.
Other than small iot devices, I don't have anything other than the opnsense firewall connected to the WAN 10G port of the HH4k.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the data transfer process from AWS. This command is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access). Everything sits on a rack with shielded CAT6 copper wire. Transfer speeds are within the expected parameters across the internal network.

The first time I did the file transfer attempt, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.



In my experience, this looks a lot like some kind IPS/active defense/firewall rule blocking the connection after a certain sustained download threshold for speed, transfer size and duration.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream. Additionally, under any other circumstances, the connection does not drop.

Eventually I managed to talk to a Level 2 Tech support rep, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the original problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets/data flow and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result (using the firewall Shaper and a download pipe).

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 4.5Tb, the full set is around 60Tb. The connection dropped after around ~50-100Gb, and kept dropping around that mark.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions or blocks.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mb/s (~500-900Mbps). These downloads typically last 5-10min. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers on Bell Fibe with a 3Gbps residential line. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts or ideas greatly appreciated.

Thanks!