Latency with Tabletop Simulator

[somesysadmin]

I get constant latency spikes in Tabletop Simulator and about 3 times so far in a month I have had my firewall drop all packets for up to 30 seconds. Enabling hardware offload makes the latency worse. I am not sure what steps to take next or if I simply have a common problem. Still new to opnsense. I would love to solve this problem as a custom opnsense box is the only way I can get above 1Gbps on my network

I have a HP NC365T and a Lenovo 00MM861 Intel X550-T2 Dual Port 10GBase-T Adapter running in a Dell PowerEdge T30

This thread discussed performance tuning but not with my specific hardware. The part about pause frames was particularly interesting, that seemed to line up with my problem

This page also mentioned how to do low-level BSD tuning but I don't know if that is handled at a different level for opnsense.

Here is the output of pciconf -lbcevV

root@OPNsense:~ # pciconf -lbcevV
hostb0@pci0:0:0:0:      class=0x060000 rev=0x07 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             1918 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/                                                                                                                                                             DRAM Registers'
    class      = bridge
    subclass   = HOST-PCI
    cap 09[e0] = vendor (length 16) Intel cap 0 version 1
    PCI errors = Received Master-Abort
pcib1@pci0:0:1:0:       class=0x060400 rev=0x07 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                             1901 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '6th-10th Gen Core Processor PCIe Controller (x16)'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[88] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 1 message
    cap 10[a0] = PCI-Express 2 root port max data 256(256)
                 max read 128
                 link x4(x16) speed 5.0(8.0) ASPM disabled(L0s/L1)
                 slot 1 power limit 75000 mW
    ecap 0002[100] = VC 1 max VC0
    ecap 0005[140] = Root Complex Link Declaration 1
    ecap 0001[1c0] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0019[d94] = PCIe Sec 1 lane errors 0
vgapci0@pci0:0:2:0:     class=0x030000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             191d subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = 'HD Graphics P530'
    class      = display
    subclass   = VGA
    bar   [10] = type Memory, range 64, base 0xde000000, size 16777216, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xc0000000, size 26843                                                                                                                                                             5456, enabled
    bar   [20] = type I/O Port, range 32, base 0xf000, size 64, enabled
    cap 09[40] = vendor (length 12) Intel cap 0 version 1
    cap 10[70] = PCI-Express 2 root endpoint max data 128(128) FLR
                 max read 128
    cap 05[ac] = MSI supports 1 message
    cap 01[d0] = powerspec 2  supports D0 D3  current D0
    ecap 001b[100] = Process Address Space ID 1
    ecap 000f[200] = ATS 1
    ecap 0013[300] = Page Page Request 1
xhci0@pci0:0:20:0:      class=0x0c0330 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a12f subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xdf430000, size 65536, enabled
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
pchtherm0@pci0:0:20:2:  class=0x118000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a131 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family Thermal Subsystem'
    class      = dasp
    bar   [10] = type Memory, range 64, base 0xdf44f000, size 4096, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[80] = MSI supports 1 message
none0@pci0:0:22:0:      class=0x078000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a13a subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family MEI Controller'
    class      = simple comms
    bar   [10] = type Memory, range 64, base 0xdf44e000, size 4096, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[8c] = MSI supports 1 message, 64 bit
uart2@pci0:0:22:3:      class=0x070002 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a13d subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family KT Redirection'
    class      = simple comms
    subclass   = UART
    bar   [10] = type I/O Port, range 32, base 0xf0a0, size 8, enabled
    bar   [14] = type Memory, range 32, base 0xdf44d000, size 4096, enabled
    cap 05[40] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 01[50] = powerspec 3  supports D0 D3  current D0
ahci0@pci0:0:23:0:      class=0x010400 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             2822 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = 'SATA Controller [RAID mode]'
    class      = mass storage
    subclass   = RAID
    bar   [10] = type Memory, range 32, base 0xdf448000, size 8192, enabled
    bar   [14] = type Memory, range 32, base 0xdf44c000, size 256, enabled
    bar   [18] = type I/O Port, range 32, base 0xf090, size 8, enabled
    bar   [1c] = type I/O Port, range 32, base 0xf080, size 4, enabled
    bar   [20] = type I/O Port, range 32, base 0xf060, size 32, enabled
    bar   [24] = type Memory, range 32, base 0xdf44b000, size 2048, enabled
    cap 05[80] = MSI supports 1 message enabled with 1 message
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 12[a8] = SATA Index-Data Pair
pcib2@pci0:0:28:0:      class=0x060400 rev=0xf1 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                             a110 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled
                 max read 128
                 link x1(x1) speed 2.5(8.0)
                 slot 2 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
    cap 01[a0] = powerspec 3  supports D0 D3  current D0
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 000d[140] = ACS 1
    ecap 0019[220] = PCIe Sec 1 lane errors 0
pcib4@pci0:0:29:0:      class=0x060400 rev=0xf1 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                             a118 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128) ARI disabled
                 max read 128
                 link x4(x4) speed 5.0(8.0)
                 slot 3 power limit 250 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1028 subdevice=0x07c5
    cap 01[a0] = powerspec 3  supports D0 D3  current D0
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 000d[140] = ACS 1
    ecap 0019[220] = PCIe Sec 1 lane errors 0
isab0@pci0:0:31:0:      class=0x060100 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a149 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = 'C236 Chipset LPC/eSPI Controller'
    class      = bridge
    subclass   = PCI-ISA
none1@pci0:0:31:2:      class=0x058000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a121 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family Power Management Control                                                                                                                                                             ler'
    class      = memory
    bar   [10] = type Memory, range 32, base 0xdf444000, size 16384, enabled
hdac0@pci0:0:31:3:      class=0x040300 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a170 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family HD Audio Controller'
    class      = multimedia
    subclass   = HDA
    bar   [10] = type Memory, range 64, base 0xdf440000, size 16384, enabled
    bar   [20] = type Memory, range 64, base 0xdf420000, size 65536, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
ichsmb0@pci0:0:31:4:    class=0x0c0500 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             a123 subvendor=0x1028 subdevice=0x07c5
    vendor     = 'Intel Corporation'
    device     = '100 Series/C230 Series Chipset Family SMBus'
    class      = serial bus
    subclass   = SMBus
    bar   [10] = type Memory, range 64, base 0xdf44a000, size 256, enabled
    bar   [20] = type I/O Port, range 32, base 0xf040, size 32, enabled
em0@pci0:0:31:6:        class=0x020000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             15b7 subvendor=0x1028 subdevice=0x06b7
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection (2) I219-LM'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xdf400000, size 131072, enabled
    cap 01[c8] = powerspec 3  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 13[e0] = PCI Advanced Features: FLR TP
ix0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 sub                                                                                                                                                             vendor=0x8086 subdevice=0x0022
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller 10G X550T'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xd0400000, size 41943                                                                                                                                                             04, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xd0804000, size 16384                                                                                                                                                             , enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 1 endpoint max data 256(512) FLR RO
                 max read 512
                 link x4(x8) speed 5.0(5.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 9fe2d0ffff740000
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x1565
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4                                                                                                                                                             194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1
    ecap 0018[1c0] = LTR 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
ix1@pci0:1:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 sub                                                                                                                                                             vendor=0x8086 subdevice=0x0022
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller 10G X550T'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xd0000000, size 41943                                                                                                                                                             04, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xd0800000, size 16384                                                                                                                                                             , enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 1 endpoint max data 256(512) FLR RO
                 max read 512
                 link x4(x8) speed 5.0(5.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 9fe2d0ffff740000
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x1565
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4                                                                                                                                                             194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
pcib3@pci0:2:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x104c device=0x                                                                                                                                                             8240 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Texas Instruments'
    device     = 'XIO2001 PCI Express-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[40] = PCI Bridge subvendor=0x0000 subdevice=0x0000
    cap 01[48] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 16 messages, 64 bit
    cap 10[70] = PCI-Express 2 PCI bridge max data 256(512)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM enable                                                                                                                                                             d
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
igb0@pci0:4:0:0:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             150e subvendor=0x103c subdevice=0x1780
    vendor     = 'Intel Corporation'
    device     = '82580 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdf180000, size 524288, enabled
    bar   [20] = type Memory, range 64, base 0xdf20c000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
                 max read 512
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 f4ce46ffffa8a897
    ecap 0017[1a0] = TPH Requester 1
    ecap 0018[1c0] = LTR 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
igb1@pci0:4:0:1:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             150e subvendor=0x103c subdevice=0x1780
    vendor     = 'Intel Corporation'
    device     = '82580 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdf100000, size 524288, enabled
    bar   [20] = type Memory, range 64, base 0xdf208000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
                 max read 512
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 f4ce46ffffa8a897
    ecap 0017[1a0] = TPH Requester 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
igb2@pci0:4:0:2:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             150e subvendor=0x103c subdevice=0x1780
    vendor     = 'Intel Corporation'
    device     = '82580 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdf080000, size 524288, enabled
    bar   [20] = type Memory, range 64, base 0xdf204000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
                 max read 512
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 f4ce46ffffa8a897
    ecap 0017[1a0] = TPH Requester 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
igb3@pci0:4:0:3:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                             150e subvendor=0x103c subdevice=0x1780
    vendor     = 'Intel Corporation'
    device     = '82580 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdf000000, size 524288, enabled
    bar   [20] = type Memory, range 64, base 0xdf200000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
                 max read 512
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 f4ce46ffffa8a897
    ecap 0017[1a0] = TPH Requester 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
root@OPNsense:~ #

[somesysadmin]

Fixed this problem by upgrading to PFsense. It's a shame, loved the fact that OPNsense seemed to have more features, but if you don't have stability with rock-solid Intel server NICs then it's useless.

[opnfwb]

Bummer you had some issues. Unfortunately in this case there wasn't enough information to troubleshoot.

What is happening on the network with the "latency spikes" are occurring? Is there some other traffic generation happening? What else was the OPNsense install doing (traffic shaping? IPS/IDS? VPN? etc. etc.)

It may not specifically be a hardware issue but some combination of environmental factors based on the config.

Also I presume this was the latest OPNsense release available, 22.1.6?

[somesysadmin]

I did a clean install and then found the fix for an issue - it was disabling flow control on a per-interface basis using the following system tunables written below (System --> Settings --> Tunables) in the web UI. FC was set to 3 or full, by default. I rebooted after setting the tunables. My guess is that FC being fully enabled dropped frames periodically, manifesting as latency spikes in game.

This might possibly be related to the fact my Intel 10Gb card supports 2.5Gbps and 5Gbps but FreeBSD/Opnsense does not offer those as negotiated speed/duplex options or show it in system output, but that would only be my guess. I was a bit more interested in getting my 2.5Gb firewall up and running than deep diving into low level FreeBSD/Intel driver issues. Also learned that in FreeBSD the driver name is in the interface name, which helped a lot.

Hopefully this helps someone else out in the future!


Name   Description   Type   Value   
dev.igb.0.fc   Flow Control   runtime   0   
dev.igb.1.fc   Flow Control   runtime   0   
dev.igb.2.fc   Flow Control   runtime   0   
dev.igb.3.fc   Flow Control   runtime   0   
dev.ix.0.fc                              runtime   0   
dev.ix.1.fc                              runtime   0

[opnfwb]

I think its awesome you found a fix. Also thanks for the followup here for others to try in case they are having a similar issue.