OPNsense Forum
English Forums => Hardware and Performance => Topic started by: somesysadmin on April 30, 2022, 07:31:18 am
-
I get constant latency spikes in Tabletop Simulator and about 3 times so far in a month I have had my firewall drop all packets for up to 30 seconds. Enabling hardware offload makes the latency worse. I am not sure what steps to take next or if I simply have a common problem. Still new to opnsense. I would love to solve this problem as a custom opnsense box is the only way I can get above 1Gbps on my network
I have a HP NC365T and a Lenovo 00MM861 Intel X550-T2 Dual Port 10GBase-T Adapter running in a Dell PowerEdge T30
This thread (https://forum.opnsense.org/index.php?topic=6590.0) discussed performance tuning but not with my specific hardware. The part about pause frames was particularly interesting, that seemed to line up with my problem
This page (https://calomel.org/freebsd_network_tuning.html) also mentioned how to do low-level BSD tuning but I don't know if that is handled at a different level for opnsense.
Here is the output of pciconf -lbcevV
root@OPNsense:~ # pciconf -lbcevV
hostb0@pci0:0:0:0: class=0x060000 rev=0x07 hdr=0x00 vendor=0x8086 device=0x 1918 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = 'Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/ DRAM Registers'
class = bridge
subclass = HOST-PCI
cap 09[e0] = vendor (length 16) Intel cap 0 version 1
PCI errors = Received Master-Abort
pcib1@pci0:0:1:0: class=0x060400 rev=0x07 hdr=0x01 vendor=0x8086 device=0x 1901 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '6th-10th Gen Core Processor PCIe Controller (x16)'
class = bridge
subclass = PCI-PCI
cap 0d[88] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message
cap 10[a0] = PCI-Express 2 root port max data 256(256)
max read 128
link x4(x16) speed 5.0(8.0) ASPM disabled(L0s/L1)
slot 1 power limit 75000 mW
ecap 0002[100] = VC 1 max VC0
ecap 0005[140] = Root Complex Link Declaration 1
ecap 0001[1c0] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0019[d94] = PCIe Sec 1 lane errors 0
vgapci0@pci0:0:2:0: class=0x030000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x 191d subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = 'HD Graphics P530'
class = display
subclass = VGA
bar [10] = type Memory, range 64, base 0xde000000, size 16777216, enabled
bar [18] = type Prefetchable Memory, range 64, base 0xc0000000, size 26843 5456, enabled
bar [20] = type I/O Port, range 32, base 0xf000, size 64, enabled
cap 09[40] = vendor (length 12) Intel cap 0 version 1
cap 10[70] = PCI-Express 2 root endpoint max data 128(128) FLR
max read 128
cap 05[ac] = MSI supports 1 message
cap 01[d0] = powerspec 2 supports D0 D3 current D0
ecap 001b[100] = Process Address Space ID 1
ecap 000f[200] = ATS 1
ecap 0013[300] = Page Page Request 1
xhci0@pci0:0:20:0: class=0x0c0330 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a12f subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller'
class = serial bus
subclass = USB
bar [10] = type Memory, range 64, base 0xdf430000, size 65536, enabled
cap 01[70] = powerspec 2 supports D0 D3 current D0
cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
pchtherm0@pci0:0:20:2: class=0x118000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a131 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family Thermal Subsystem'
class = dasp
bar [10] = type Memory, range 64, base 0xdf44f000, size 4096, enabled
cap 01[50] = powerspec 3 supports D0 D3 current D0
cap 05[80] = MSI supports 1 message
none0@pci0:0:22:0: class=0x078000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a13a subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family MEI Controller'
class = simple comms
bar [10] = type Memory, range 64, base 0xdf44e000, size 4096, enabled
cap 01[50] = powerspec 3 supports D0 D3 current D0
cap 05[8c] = MSI supports 1 message, 64 bit
uart2@pci0:0:22:3: class=0x070002 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a13d subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family KT Redirection'
class = simple comms
subclass = UART
bar [10] = type I/O Port, range 32, base 0xf0a0, size 8, enabled
bar [14] = type Memory, range 32, base 0xdf44d000, size 4096, enabled
cap 05[40] = MSI supports 1 message, 64 bit enabled with 1 message
cap 01[50] = powerspec 3 supports D0 D3 current D0
ahci0@pci0:0:23:0: class=0x010400 rev=0x31 hdr=0x00 vendor=0x8086 device=0x 2822 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = 'SATA Controller [RAID mode]'
class = mass storage
subclass = RAID
bar [10] = type Memory, range 32, base 0xdf448000, size 8192, enabled
bar [14] = type Memory, range 32, base 0xdf44c000, size 256, enabled
bar [18] = type I/O Port, range 32, base 0xf090, size 8, enabled
bar [1c] = type I/O Port, range 32, base 0xf080, size 4, enabled
bar [20] = type I/O Port, range 32, base 0xf060, size 32, enabled
bar [24] = type Memory, range 32, base 0xdf44b000, size 2048, enabled
cap 05[80] = MSI supports 1 message enabled with 1 message
cap 01[70] = powerspec 3 supports D0 D3 current D0
cap 12[a8] = SATA Index-Data Pair
pcib2@pci0:0:28:0: class=0x060400 rev=0xf1 hdr=0x01 vendor=0x8086 device=0x a110 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family PCI Express Root Port'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled
max read 128
link x1(x1) speed 2.5(8.0)
slot 2 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
cap 01[a0] = powerspec 3 supports D0 D3 current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[140] = ACS 1
ecap 0019[220] = PCIe Sec 1 lane errors 0
pcib4@pci0:0:29:0: class=0x060400 rev=0xf1 hdr=0x01 vendor=0x8086 device=0x a118 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family PCI Express Root Port'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128) ARI disabled
max read 128
link x4(x4) speed 5.0(8.0)
slot 3 power limit 250 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
cap 01[a0] = powerspec 3 supports D0 D3 current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[140] = ACS 1
ecap 0019[220] = PCIe Sec 1 lane errors 0
isab0@pci0:0:31:0: class=0x060100 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a149 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = 'C236 Chipset LPC/eSPI Controller'
class = bridge
subclass = PCI-ISA
none1@pci0:0:31:2: class=0x058000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a121 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family Power Management Control ler'
class = memory
bar [10] = type Memory, range 32, base 0xdf444000, size 16384, enabled
hdac0@pci0:0:31:3: class=0x040300 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a170 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family HD Audio Controller'
class = multimedia
subclass = HDA
bar [10] = type Memory, range 64, base 0xdf440000, size 16384, enabled
bar [20] = type Memory, range 64, base 0xdf420000, size 65536, enabled
cap 01[50] = powerspec 3 supports D0 D3 current D0
cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
ichsmb0@pci0:0:31:4: class=0x0c0500 rev=0x31 hdr=0x00 vendor=0x8086 device=0x a123 subvendor=0x1028 subdevice=0x07c5
vendor = 'Intel Corporation'
device = '100 Series/C230 Series Chipset Family SMBus'
class = serial bus
subclass = SMBus
bar [10] = type Memory, range 64, base 0xdf44a000, size 256, enabled
bar [20] = type I/O Port, range 32, base 0xf040, size 32, enabled
em0@pci0:0:31:6: class=0x020000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x 15b7 subvendor=0x1028 subdevice=0x06b7
vendor = 'Intel Corporation'
device = 'Ethernet Connection (2) I219-LM'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xdf400000, size 131072, enabled
cap 01[c8] = powerspec 3 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP
ix0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 sub vendor=0x8086 subdevice=0x0022
vendor = 'Intel Corporation'
device = 'Ethernet Controller 10G X550T'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xd0400000, size 41943 04, enabled
bar [20] = type Prefetchable Memory, range 64, base 0xd0804000, size 16384 , enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 1 endpoint max data 256(512) FLR RO
max read 512
link x4(x8) speed 5.0(5.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 9fe2d0ffff740000
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x1565
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4 194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1
ecap 0018[1c0] = LTR 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
ix1@pci0:1:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 sub vendor=0x8086 subdevice=0x0022
vendor = 'Intel Corporation'
device = 'Ethernet Controller 10G X550T'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xd0000000, size 41943 04, enabled
bar [20] = type Prefetchable Memory, range 64, base 0xd0800000, size 16384 , enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 1 endpoint max data 256(512) FLR RO
max read 512
link x4(x8) speed 5.0(5.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 9fe2d0ffff740000
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x1565
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4 194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
pcib3@pci0:2:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x104c device=0x 8240 subvendor=0x0000 subdevice=0x0000
vendor = 'Texas Instruments'
device = 'XIO2001 PCI Express-to-PCI Bridge'
class = bridge
subclass = PCI-PCI
cap 0d[40] = PCI Bridge subvendor=0x0000 subdevice=0x0000
cap 01[48] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 05[50] = MSI supports 16 messages, 64 bit
cap 10[70] = PCI-Express 2 PCI bridge max data 256(512)
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM enable d
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
igb0@pci0:4:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x 150e subvendor=0x103c subdevice=0x1780
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdf180000, size 524288, enabled
bar [20] = type Memory, range 64, base 0xdf20c000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
max read 512
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
cap 03[e0] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 f4ce46ffffa8a897
ecap 0017[1a0] = TPH Requester 1
ecap 0018[1c0] = LTR 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
igb1@pci0:4:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x 150e subvendor=0x103c subdevice=0x1780
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdf100000, size 524288, enabled
bar [20] = type Memory, range 64, base 0xdf208000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
max read 512
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
cap 03[e0] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 f4ce46ffffa8a897
ecap 0017[1a0] = TPH Requester 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
igb2@pci0:4:0:2: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x 150e subvendor=0x103c subdevice=0x1780
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdf080000, size 524288, enabled
bar [20] = type Memory, range 64, base 0xdf204000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
max read 512
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
cap 03[e0] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 f4ce46ffffa8a897
ecap 0017[1a0] = TPH Requester 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
igb3@pci0:4:0:3: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x 150e subvendor=0x103c subdevice=0x1780
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdf000000, size 524288, enabled
bar [20] = type Memory, range 64, base 0xdf200000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
max read 512
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
cap 03[e0] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 f4ce46ffffa8a897
ecap 0017[1a0] = TPH Requester 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
root@OPNsense:~ #
-
Fixed this problem by upgrading to PFsense. It's a shame, loved the fact that OPNsense seemed to have more features, but if you don't have stability with rock-solid Intel server NICs then it's useless.
-
Bummer you had some issues. Unfortunately in this case there wasn't enough information to troubleshoot.
What is happening on the network with the "latency spikes" are occurring? Is there some other traffic generation happening? What else was the OPNsense install doing (traffic shaping? IPS/IDS? VPN? etc. etc.)
It may not specifically be a hardware issue but some combination of environmental factors based on the config.
Also I presume this was the latest OPNsense release available, 22.1.6?
-
I did a clean install and then found the fix for an issue - it was disabling flow control on a per-interface basis using the following system tunables written below (System --> Settings --> Tunables) in the web UI. FC was set to 3 or full, by default. I rebooted after setting the tunables. My guess is that FC being fully enabled dropped frames periodically, manifesting as latency spikes in game.
This might possibly be related to the fact my Intel 10Gb card supports 2.5Gbps and 5Gbps but FreeBSD/Opnsense does not offer those as negotiated speed/duplex options or show it in system output, but that would only be my guess. I was a bit more interested in getting my 2.5Gb firewall up and running than deep diving into low level FreeBSD/Intel driver issues. Also learned that in FreeBSD the driver name is in the interface name, which helped a lot.
Hopefully this helps someone else out in the future!
Name Description Type Value
dev.igb.0.fc Flow Control runtime 0
dev.igb.1.fc Flow Control runtime 0
dev.igb.2.fc Flow Control runtime 0
dev.igb.3.fc Flow Control runtime 0
dev.ix.0.fc runtime 0
dev.ix.1.fc runtime 0
-
I think its awesome you found a fix. Also thanks for the followup here for others to try in case they are having a similar issue. :)