Optimize network for Fios - settings, switches and WAP

[FlightService]

Hey y'all.  I am new to the whole commercial grade router bit but I keep frying consumer stuff.  So I decided to build my own and have been happy with it so far.  My upgrade to 20.7 was bumpy but it seems to be rolling along now.  Time to optimize for my Fios Gigabit and network setup.  Here is my current set-up

Fios modem (just the modem, no Fios router or moca bridge) to the Opnsense router (Lenovo M91P USFF) motherboard's network port.

Opnsense router consists of a Lenovo M91p USFF with a Intel Core i5-2400S CPU @ 2.50GHz (4 cores), 8 gb ram, 120 gb SSD, with a Jeirdus (Intel Chipset 82546) dual-port gigabit 8492MT PCI Server NIC which is connected to a TP-Link switch. (Yes, the M91p USFF is weird with a PCI instead of PCI-e, I know I am close to saturating the PCI bus with a gigabit connection but Fios doesn't seem to be able to deliver over 1.1 gigabit to my house and never consistently.)

The TP-Link TL-SG108 switch has attached a TP-Link EAP225 V3 AC1350 WAP, Netgear GSS-108E managed switch (set to dumb), and 2 other PCs. 

The Netgear GSS-108E is behind my TV and is feeding my Roku, StreamLink (only one of these can be in use at one time as they are all to the same TV), PS4 and laptop.  Eventually this may see both the laptop and a PC.

I would like to optimize the Opnsense settings and the network set-up in general.  I tried to connect the Netgear switch directly to the other open port on the NIC with no luck.  So I just put it in the TP-Link router which has everything running.  I am running basically out of the box 20.7, no WLAN for the WAP or anything.

Any advice and guidance is appreciated.  Thanks in advance.

[ligand]

I would love to hear back from some experts on their fios configuration.  The max I've been able to achieve is 800/800 Mbps.  I haven't quite gotten to a gig yet...

[FlightService]

I hope so.  I have searched the forums a bit and believe I have put enough information in to where the knowledgeable ones can respond.

[FlightService]

This is just a bump.  Just a little bump.  Let's bump.

[littlepepper]

As long as you get your full connection speed, any optimization on the OS level is very minimal. Opnsense just see the any network port as a network port. I assume AES NI and temp settings are all Intel and turned on.

Depending on how often your swap gets filled up (I suspect not often) you can consider disabling swap to eliminate a potential point of failure (SSD). You might want to experiment with turning on / off suricata and see if you and get a bit more out of your CPU cycles.

You can also consider putting your iot devices on one network port and your more personal devices on another in case iot devices snitch.

[FlightService]

Thanks.  I did not have the temp switched on.  I have never seen the ssd over 10% full.  I don’t keep massive logs or anything.

Other than my phones, I don’t have any IoTs.  No smart anything.  Which is like their owner!

Sent from my iPhone using Tapatalk

[packetmangler]

If you're getting 1.1 gigabit you're already getting more than FiOS officially offers.

Taken directly from their website: "Up to 940/880 Mbps"

I typically am in the 930 - 950 range when testing via speedtest.net and haven't done any tuning specifically for FiOS. The connection does sometimes test faster using fast.com.

If you're using Suricata, Sensei or the like, I'd disable those and see where things stand.

[FlightService]

The one time I was seeing full gigabit speeds was when I connected directly to their modem with my laptop.  I have never seen that through any router.  Also I have never seen that speed more than once and that was with Fios on the phone.  Now I have seen uploads massively outpace what they say with a direct connection.  (1.3 gb up).  I mentioned 1.1 gb because that is the max a PCI lane (remember I am running a PCI NIC not a PCI-e) is rated to hold.  Testing to their servers (which is another asterisk on their service) I have seen through other routers a stable 800. 

What’s interesting is they say if I use their router they say my speed will increase.  That was a fun conversation when I asked how that worked.

And I’m not using either of those services. 

Sent from my iPhone using Tapatalk

[FlightService]

So after everyone’s advice I am still seeing the same ~600mb/s down and ~800mb/s up as before.  Anyone have any other ideas?   


Sent from my iPhone using Tapatalk

[littlepepper]

From 800 to 900 Gbps is more than 10% improvement. That is hard to eke out that kind of improvement purely from software alone.

You can move to another location closer to Verizon's data center (assuming that's your FIOS provider). Yes even with fiber, # of hops still matters, just not as much as before.
and/or you can upgrade your router to something newer that has PCI-E or something like this https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E301-9D-8CN8TP.cfm especially if you run CPU intensive packet inspection like Sensei or Suricata.

[FlightService]

I am roughly 5 miles from their data center and 3 miles from there to the main internet backbone. 
I am going to do some comparison tests this weekend late between going through the router and straight modem connection and try to give more accurate 1 to 1 comparison. 

I have never seen the processor exceed 20% load. 

Given my upload is significantly faster than my download, do you think I am saturating the PCI lane on the router?


Sent from my iPhone using Tapatalk

[cmmh]

You're doing better than me.  I have built an OPNSense router using an HP610t thin client with a quad-port Intel GigE PCIe card.  I'm using two ports on that card during this test.  I believe that it has 8x lane PCIe interface.

Retrieving speedtest.net configuration...
Testing from Verizon Fios (74.96.xxx.xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Xiber LLC (Washington, DC) [21.78 km]: 6.96 ms
Testing download speed................................................................................
Download: 333.60 Mbit/s
Testing upload speed................................................................................................
Upload: 538.73 Mbit/s
Keep in mind that FIOS bandwidth is shared with your neighbors.   If others are heavily using FIOS your bandwidth will be affected. 

[spetrillo]

FIOS is not shared with others. It is FTTH, which is fiber to the home. Unlike COAX, which is a shared technology, you are not sharing.

I am curious...@FlightService you mentioned a FIOS modem. There is no modem. Are you speaking about the ONT, or optical network terminal? You might want to check your ethernet cable from the ONT to your firewall. I just recently swapped mine out and picked up 10%. Both Verizon and I were noticing some oddities and timeouts, so I swapped out the ethernet cable and Verizon swapped out my ONT.

[FlightService]

I can double check but it is a Cat 6.  Although you are technically correct Verizon refers to it as a modem if you call them.


Sent from my iPhone using Tapatalk

[cmmh]

Fios connections are indeed shared by multiple users.  Usually with smaller ratios than Cable. 
From the FIOS FAQ at broadbandreports.com

FiOS networks are built upon Passive Optical Networks.

BPON conforms to the ITU-T G983.1 specification which is capable of 622 Mbps download and 155 Mbps upload. Each BPON fiber is split using an optical splitter to serve 16 or 32 users.

GPON conforms to the ITU-T G984.1 specification. Verizon's GPON implementation uses Gig-E instead of ATM that was used with BPON. A 2.4 Gbps download speed coupled with a 1.2 Gbps upload speed. Each GPON fiber is split to serve 16, 32, or 64 users per fiber.

BPON ONT users have a max speed of 50/50-75/75, and are switched to GPON when a faster speed is ordered. All new installs are done with GPON.

I did have success doubling my speed by turning on the hardware offloading to 600 Mbps up and down.  Still shy of the 800 MBps I was able to get with my edgerouter X.  MTU changes had not speed change