Bell Fibe Home Hub 4000 bridge drops internet when using the full 3Gbps pipe

[ninja0n3]

** Originally posted here: https://forum.bell.ca/t5/Internet/Home-Hub-4000-bridge-drops-internet-when-using-the-full-3Gbps/m-p/5821

Hello,

Yesterday I spent a bit over 1h45 with a Bell Internet support rep and we weren't able to find a concrete solution.

I'm curious to see if anyone else has experienced something similar and trying to get some insight into the issue.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense linux system using the 10G WAN port and going into a 10G NIC on the Opnsense firewall. The internet uplink is achieved using PPPoE (with Bell login info - b1 id and password) through the 10Gb WAN port.

I have a Bell Fibe Internet 3Gbps.

Up to here, everything works flawlessly:



My speed tests on the opnsense box are consistently around 3Gbps, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.
Other than small iot devices, I don't have anything other than the opnsense firewall connected to the WAN 10G port of the HH4k.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the data transfer process from AWS. This command is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access). Everything sits on a rack with shielded CAT6 copper wire. Transfer speeds are within the expected parameters across the internal network.

The first time I did the file transfer attempt, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.



In my experience, this looks a lot like some kind IPS/active defense/firewall rule blocking the connection after a certain sustained download threshold for speed, transfer size and duration.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream. Additionally, under any other circumstances, the connection does not drop.

Eventually I managed to talk to a Level 2 Tech support rep, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the original problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets/data flow and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result (using the firewall Shaper and a download pipe).

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 4.5Tb, the full set is around 60Tb. The connection dropped after around ~50-100Gb, and kept dropping around that mark.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions or blocks.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mb/s (~500-900Mbps). These downloads typically last 5-10min. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers on Bell Fibe with a 3Gbps residential line. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts or ideas greatly appreciated.

Thanks!