[Help] troubleshooting high latency issues

[tswalker]

[In short]:  I'm looking for suggestions on how to determine what is causing high latency when multiple HTTPs streams are occurring via my ISP.

---- details ----

I've been using OPNsense for a couple months now and have an issue that I've been trying to determine what the actual cause.

I have 200/10 (Mbps) service, which has been reasonably stable; however, when multiple users begin streaming (mostly over HTTPs), and bandwidth of those stream is greater than a total of 10Mbps, I begin seeing high latency with all other traffic.  My sons like to play online games too, and these spikes are really noticeable so i've been trying to determine what the actual problem might be.

To 'curb' the issues, I created some limiters and dump all HTTP and HTTPs traffic into a single pipe and queued with CoDel of only 10Mbps.  When more than one or two streamings begin, they get buffering... and other protocol transfer will be fine (ie: gaming with UDP or such).

Is this normal to see such an issue?  I would think I could break each MAC or IP into its' own pipe of say 5Mbps, and utilize more of my available bandwidth, but my attempts to do this previously have failed and continue to cause high amounts of latency and will lead to dropped packets.

Could this be a problem with my firewall hardware?

I'm listing the general specs here, but I just can't figure out what to do or how to address this problem.

CPU:
hw.machine: amd64
hw.model: Intel(R) Atom(TM) CPU D525   @ 1.80GHz
hw.ncpu: 4
hw.machine_arch: amd64

MEMORY: 4GB DDR2
# sysctl hw.realmem
hw.realmem: 4294967296

NICS (2 identical):
em0@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L1)
    cap 11[a0] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 xxxxxxxxxxxxxxx


em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xd000-0xd01f mem 0xfe780000-0xfe79ffff,0xfe740000-0xfe77ffff,0xfe7a0000-0xfe7a

HARDWARE (dump):
hostb0@pci0:0:0:0:      class=0x060000 card=0xa0108086 chip=0xa0008086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor D4xx/D5xx/N4xx/N5xx DMI Bridge'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:     class=0x030000 card=0xa0118086 chip=0xa0018086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor D4xx/D5xx/N4xx/N5xx Integrated Graphics Controller'
    class      = display
    subclass   = VGA
vgapci1@pci0:0:2:1:     class=0x038000 card=0xa0118086 chip=0xa0028086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor D4xx/D5xx/N4xx/N5xx Integrated Graphics Controller'
    class      = display
uhci0@pci0:0:26:0:      class=0x0c0300 card=0x29378086 chip=0x29378086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:26:1:      class=0x0c0300 card=0x29388086 chip=0x29388086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:26:2:      class=0x0c0300 card=0x29398086 chip=0x29398086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:26:7:      class=0x0c0320 card=0x293c8086 chip=0x293c8086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
pcib1@pci0:0:28:0:      class=0x060400 card=0x29408086 chip=0x29408086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) PCI Express Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:3:      class=0x060400 card=0x29468086 chip=0x29468086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) PCI Express Port 4'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:4:      class=0x060400 card=0x29488086 chip=0x29488086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) PCI Express Port 5'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:28:5:      class=0x060400 card=0x294a8086 chip=0x294a8086 rev=0x02 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) PCI Express Port 6'
    class      = bridge
    subclass   = PCI-PCI
uhci3@pci0:0:29:0:      class=0x0c0300 card=0x29348086 chip=0x29348086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci4@pci0:0:29:1:      class=0x0c0300 card=0x29358086 chip=0x29358086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci5@pci0:0:29:2:      class=0x0c0300 card=0x29368086 chip=0x29368086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci1@pci0:0:29:7:      class=0x0c0320 card=0x293a8086 chip=0x293a8086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
pcib5@pci0:0:30:0:      class=0x060401 card=0x244e8086 chip=0x244e8086 rev=0x92 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:      class=0x060100 card=0x29168086 chip=0x29168086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR (ICH9R) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:      class=0x010601 card=0x29228086 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
none0@pci0:0:31:3:      class=0x0c0500 card=0x29308086 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
xhci0@pci0:2:0:0:       class=0x0c0330 card=0x00021b96 chip=0x01941033 rev=0x04 hdr=0x00
    vendor     = 'NEC Corporation'
    device     = 'uPD720200 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
em0@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:4:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet

[tswalker]

I believe i may have improved on this situation by following some of the guides available to adjust tunables for em(4).

I have applied the following values:

Property	Description
machdep.hyperthreading_allowed="0"	#(default 1, allow Hyper Threading (HT))
hw.em.eee_setting="1"	#Energy Efficient   Ethernet (Default 1 (disabled))
hw.em.rx_process_limit="-1"	#Remove limit of packets to manage at once (default 100)
hw.em.tx_process_limit="-1"	#Remove limit of packets to manage at once (default 100)
net.inet.ip.intr_queue_maxlen="2048"	#Interface Maximum Queue Length (net.inet.ip.intr_queue_maxlen)
net.inet.tcp.soreceive_stream="1"	#Enable the optimized version of soreceive() for stream (TCP) sockets
net.isr.maxthreads="-1"	#(default 1 [single threaded]) CPU affinity for all cores & threads
net.isr.bindthreads="1"	#(default 0 [runs randomly on any one cpu core], 1 binds thread to CPU)
dev.em.0.fc="0"	#Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
dev.em.1.fc="0"	# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
net.inet.tcp.tso="0"	#TPC Segmentation Offload (disable = 0, enable = 1)

Most of which came from recommendations here: https://calomel.org/freebsd_network_tuning.html
(which was a very good article)

I've had reasonable results today but have yet to fully test things. 

note: some of the property settings must be done in loader.conf.local