Comcast gigabit and OpnSense slow speeds.

[LostnIL]

Hi Everyone.

I am new to OpnSense and I was wondering if it is possible to achieve gigabit downloads with Comcast (DOCSIS 3.1 modem) with OpnSense?

Comcast > My Own Modem (DOCSIS 3.1) > Proxmox VM

i7-7700K CPU @ 4.20GHz
NVME
64GB of RAM, 8 dedicated to OpnSense VM.

Testing from Comcast Cable (xx.xxx.xx.xx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by i3 Broadband () [1.86 km]: 35.827 ms
Testing download speed................................................................................
Download: 619.69 Mbit/s
Testing upload speed...
.Upload: 39.37 Mbit/s


I've read a lot on the internet but I can't seem to get my speeds above 620 Mbit/s. Has anyone had any luck with similar setups?

[bunchofreeds]

I'm getting Gigabit (900/400 is my internet) through my OPNsense running as a VM on Proxmox.
My Proxmox host hardware is less than yours but I do have an additional dual port 82576EB Intel card. These external Intel ports are attached to two linux bridges, one for WAN and one for LAN.
Pretty simple and straight forward setup.
I'm using virtio NIC's for OPNsense.
I have not applied any tunables to OPNsense.
OPNsense has 8GB RAM, 4vCPU, 10GB Disk backed by SSD.

Another VM on the same host and connected to LAN (vmbr2) blasts 900/400 using the speedtest.net CLI tool.
This is a Windows server 2019 guest.

A Physical Desktop connected to a switch and then to the Proxmox host via LAN also tests 900/400.

So it certainly is possible.

Are you using Intel NIC's? I hear anything else is problematic but can't confirm this.
Are you testing using the Speedtest.net CLI tool and not the standard web browsed method?
Have you tried iPerf3?


   Speedtest by Ookla

     Server: Xtreme Networks
        ISP: 2degrees Mobile
    Latency:     1.81 ms   (0.24 ms jitter)
   Download:   915.50 Mbps (data used: 460.4 MB)
     Upload:   495.86 Mbps (data used: 396.5 MB)
Packet Loss:     0.0%

[LostnIL]

I picked up a dual NIC locally. Thank you for sharing your setup and experience. It is nice knowing it is possible, I really want to stick with OpnSense but if I can't get my full speeds, I'm going to have to find another solution.

The NIC I am using:
10Gtek
82576-GE-2T-X1
Intel 82576
PCIe x1

I am using the python3 speedtest cli script.

[bunchofreeds]

I have a fresh install of the latest OPNsense 21.7.3-amd64
Make sure you are not running any additional services like IPS or even config like DynDNS initially. Just as vanilla as possible.

My OPNsense was installed as 'Other' (Not Linux etc within Proxmox)
I have selected QEMU guest agent and run this in the OPNsense guest through plugins
It only has two interfaces being WAN and LAN

I'm not sure how much you know about virtualisation, but make sure you are dedicating the NIC's to OPNsense as much as possible to ensure it has physical bandwidth to achieve what you want. Less sharing basically. Thinking about how much each VM is sending and receiving during your test within your host and on what network paths.

Make sure you have setup your disks correctly (I use 'discard' with an older SSD).
VirtIO SCSI and the disks are SCSI.
Make sure they are targeting your NVME and its actually giving you the IOPS you expect when tested at the host.
I use I/O Thread also as an option when selecting disk type.

Virtio seems best for NIC interfaces.

I tested using this
https://www.speedtest.net/apps/cli

OH and make sure your OPNsense is not running/showing Reporting>Traffic Graphs. This slows my throughput considerably!

[LostnIL]

bunchofreeds

I don't have IPS on. It looks like I do have reports running. I am trying to find out how to shut those off.

I have dedicated a dual intel NIC to OpnSense:

OpnSense (vmbr1)vtnet0 = WAN (vmbr2)vtnet1 = LAN

and then my VMs are using a separate Linux (vmbr0)bridge to another physical nic.

I have selected QEMU guest agent and run this in the OPNsense guest through plugins

In Proxmox, I created an actual VM for it, not a LXC container. Is that what you did?

[bunchofreeds]

Sounds like my setup.
For the reports just don't have the screen open on OPNsense that shows the realtime graph of the network bandwidth, be on the Dashboard instead for example.

Yes I run a VM for OPNsense also
My onboard NIC is what I use to connect to Proxmox for admin etc this is vmbr0 and has the proxmox IP address.
Like you my vmbr1 is WAN and vmbr2 is LAN, these are my two intel ports on my add in card.

If your VM's are on the same host as your OPNsense then you 'could' connect them to your vmbr2 LAN. The same virtual LAN switch your OPNsense is connected to, this would keep the traffic between them and OPNsense within the host and not having to trombone up to the switch and back down again.

Is your physical switch that your LAN ports are connected to capable of multiple gigabit (Can it move multiple gigibit simultaneously between multiple ports. Most enterprise switches will be able to but some consumer ones are not great. Keeping OPNsense LAN and all VM's that need LAN on the same virtual switch should mitigate this.

Do you run any VLANs or multiple internal subnets?

Do you have a way of confirming you can actually get the gigabit speeds you are after with another router?

[LostnIL]

No VLANs (yet).

I have a Cisco Small Business gigabit switch. I've ordered a new switch. I was getting gigabit speeds with my TPLINK DeCo router. Actually, I was able to pull 1,400mbps as Comcast boosted their gigabit speeds.

[LostnIL]

New switch in. No effect.

I did however launch OpenWRT and IPFIRE (both linux-based) and my connection STILL doesn't exceed 670mpbs.

Maybe Comcast is limiting the connection for some odd reason, or maybe it is because I am doing it in a VM and/or some BIOS setting on the hypervisor.

[bunchofreeds]

Does iperf3 provide any better results?
This can be installed as a plugin on OPNsense then also on a device somewhere in your LAN.
This might help you to track down the bottleneck.
We seem to have a similar setup so it should be possible for you.

But like you say, it would also be sensible to ensure that comcast is reliably providing you gigabit.

Are you able to have another router AND your proxmox host connected to a set of ports on your switch that then connects upstream to comcast? Switching between them using port enable/disable or powering up/down etc.

I VLAN'd off some ports on my switch for WAN so I could switch between devices directly connected to my ISP.
This allows me to switch between a linksys router and my OPNsense running on proxmox for testing.

[darp12345]

I'm on Xfinity's Gigabit Xtra plan which is supposed to be 1200Mbps. With 40Mbps upload speed. I have codel_fq on the egress to keep the latency in check as the outbound is easy to saturate. I don't have traffic shaper running on the ingress. The download speed maxes out at around 1400Mbps. Note the high latency which means that the modem link is saturated. The connection between the firewall and the modem is 2.5Gbps so it still has capacity.

Code Select Expand


speedtest

   Speedtest by Ookla

      Server: Race Communications - San Francisco, CA (id: 8228)
         ISP: Comcast Cable
Idle Latency:    18.01 ms   (jitter: 1.18ms, low: 17.09ms, high: 19.35ms)
    Download:  1423.36 Mbps (data used: 2.2 GB)                                                   
                 33.47 ms   (jitter: 4.75ms, low: 9.96ms, high: 159.69ms)
      Upload:    38.87 Mbps (data used: 37.6 MB)                                                   
                 17.45 ms   (jitter: 2.68ms, low: 11.94ms, high: 24.88ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/1bd53d7b-803b-40f4-ad0d-40bc9ce9838a


[xPliZit_xs]

Check with XFinity or your cable modem web interface if your cable modem has bonded correctly with all 32 the download channels.

I am running opense on proxmox with XFinity gigabit service and no issues with download or upload speed.
I do use 2.5 Gbit/s from cable modem to dual channel 10Gbit SFP card in proxmox/opnsense.