Messages

1

22.7 Legacy Series / Re: Speedtest Bandwidth drop through opnsense

« on: January 13, 2023, 06:19:41 pm »

Yes, I ended up using the speedtest-cli on the second post in this thread, I'm still seeing a 400Mbps drop on download while getting the full 3.2Gbps on the upload.

Not sure what kind of overhead would be causing that, iperf3 between the win machine and opnsense is around 9.5+ Gbps

My opnsense setup is as follows:

Internet [10Gbe] <=> [10Gbe ix1] opnsense [10Gbe ix2] <=> [10Gbe Onboard] Win 11

2

22.7 Legacy Series / opensense crash 9000MTU during transfer dropoff

« on: January 12, 2023, 03:12:54 am »

Hello,

I just experienced a strange issue today.
My opnsense froze, I noticed a message `ketn.ipc.nmbjumbop` limit reached on the screen.

At the time this happened, I was running a download at around 3.2Gbps (~400Mb/s) through opnsense on one of my servers (both with 10Gbe, 9000MTU, iperf3 clocking at 9.89Gbps between opnsense and the server).
Due to a completely unrelated issue (different problem altogether associated with the ISP - due to some throttling), the connection dropped a couple of times and then opnsense froze (no direct input on console and I noticed the message I mentioned above).
The ISP throttling/connection dropoff lasts for about 6min before restoring the internet. So it goes around 5min on, 6 min off.

I tried to change the tunables to:
- kern.ipc.nmbclusters = 1048576
- kern.ipc.nmbjumbop = 524288
- kern.ipc.nmbjumbo9 = 524288

This didn't have any effect, whenever the ISP connection would drop after a couple of drops the system would freeze, this time with no message on screen.

The opnsense hardware is as follows:
- CPU: i7-7700K
- RAM: 16Gb

This is the first time I've experienced with issue. It started happening with the optimized MTU for the NICs in order to reach 10Gbps on the network.

Any help with this would be greatly appreciated.

3

22.7 Legacy Series / Re: Speedtest Bandwidth drop through opnsense

« on: January 11, 2023, 05:21:38 pm »

Using the speedtest CLI on windows seems to make things a lot better, however I'm still seeing a 400Mbps difference on download whereas I'm getting the full speed on upload (see screenshot)

4

22.7 Legacy Series / Speedtest Bandwidth drop through opnsense

« on: January 11, 2023, 04:59:39 pm »

Hello,

I'm trying to figure out if I'm doing something wrong in my environment.

Before I describe the issue, here is some context:
- I'm running opnsense as the gateway to the internet
- The appliance has 10Gbge interfaces
- iperf3 testing between opnsense and my windows machine runs at around 9.2Gbps up/down
- Internet runs at 3gbps, opensense speedtest measures 3.2Gbps up/down (daily measurements) through a 10Gbe interface

The issue that I'm experiencing is that when I run a speed test on the windows machine (AV and firewall disabled), I'm only seeing around 2.4Gbps even though the throughput on the firewall seems to be capable of handling the entire 3Gbps (based on iperf3 results).
I'm running speedtest through the browser (same server as the one used by opnsense).

I'm not sure what could be causing the drop, or whether I'm performing the test incorrectly.

Any insight would be greatly appreciated.

Thanks!

5

General Discussion / Bell Fibe Home Hub 4000 bridge drops internet when using the full 3Gbps pipe

« on: July 01, 2022, 06:47:20 pm »

** Originally posted here: https://forum.bell.ca/t5/Internet/Home-Hub-4000-bridge-drops-internet-when-using-the-full-3Gbps/m-p/5821

Hello,

Yesterday I spent a bit over 1h45 with a Bell Internet support rep and we weren't able to find a concrete solution.

I'm curious to see if anyone else has experienced something similar and trying to get some insight into the issue.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense linux system using the 10G WAN port and going into a 10G NIC on the Opnsense firewall. The internet uplink is achieved using PPPoE (with Bell login info - b1 id and password) through the 10Gb WAN port.

I have a Bell Fibe Internet 3Gbps.

Up to here, everything works flawlessly:



My speed tests on the opnsense box are consistently around 3Gbps, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.
Other than small iot devices, I don't have anything other than the opnsense firewall connected to the WAN 10G port of the HH4k.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the data transfer process from AWS. This command is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access). Everything sits on a rack with shielded CAT6 copper wire. Transfer speeds are within the expected parameters across the internal network.

The first time I did the file transfer attempt, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.



In my experience, this looks a lot like some kind IPS/active defense/firewall rule blocking the connection after a certain sustained download threshold for speed, transfer size and duration.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream. Additionally, under any other circumstances, the connection does not drop.

Eventually I managed to talk to a Level 2 Tech support rep, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the original problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets/data flow and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result (using the firewall Shaper and a download pipe).

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 4.5Tb, the full set is around 60Tb. The connection dropped after around ~50-100Gb, and kept dropping around that mark.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions or blocks.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mb/s (~500-900Mbps). These downloads typically last 5-10min. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers on Bell Fibe with a 3Gbps residential line. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts or ideas greatly appreciated.

Thanks!