Retired, but managed pfsense during most of my career at an ISP. For home I was using an APU model with pfsense but when it died I needed something quick so got a ubiquiti edgerouter X that I've been using a year. I finally got around to replacing it and several friends said I should look at opnsense before installing pfsense. From spending a few days with it, I'm happier with opnsense. One problem though....
WAN connection is to ATT fiber via a BGW320-500 set to IP passthrough, speed 1gbps, static public IP.
When I first hooked up the ubiquiti edgerouter X to the above connection, my throughput via several online broadband speed tests showed around 250mbps, well short of 1gbps. The moment I turned on hardware nat offload, same speed tests showed around 950mbps and I've been getting that speed since.
I've built a box to replace the edgerouter X; i7-3770S (4 cores 8 threads at 3.1ghz), ASUS P8B75-M, 16gb ram, 240gb ssd, and two dual port Intel Pro1000 pci nics (4 ports total, 2 on each card) using em driver, and a built-in re0 that I may use for management or not at all. LAN is em0, WAN is em2, so each is on a different card.
When I hook this up in place of the edgerouter X, all speed tests from lan clients show about 250mbps. I tried turning on offloading for CRC, TSO, and LRO. Retesting speeds shows throughput is unchanged. I put the edgerouter X back in place of the new server and I'm back to about 950mbps, so it's something with the opnsense machine.
The setup is very simple - just nat from lan to wan, a handful of static dhcp mappings, and a couple port forwards. No other software is chewing up resources.
I saw this article https://jeffmbelt.com/opnsense-1g-throughput.html that may offer help, but I noticed several of the tunables dont exist so perhaps it is an old outdated article.
Can anyone point me in the right direction to begin troubleshooting why my throughput is tanking? I'd really love to stick with opnsense if possible.
Best,
J
What are you using to test? The ISP site, speedtest.net, fast.com?
Can you put the new machine in between two computers and do an iperf3 test?
Can you do a vanilla install and test that without any changes?
I have tried several different speed test sites, the ones you list included. The results are all nominally comparative indicating it's unlikely to be related to one test site vs another. iperf3 is of course the definitive test, but I'm reasonably confident the above test is indicative of the same issue.
Testbed locally - I can do this when I get back in town next week, but the above test shows a marked difference between the two hosts, regardless of how its plumbed.
Well, not quite vanilla, as I'll have to assign some IP's :D The only thing I've added are static dhcp mappings and a few port forwards. I can reinstall to do this; just to cross it off the list.
When I get back, I'll also verify if the cards are getting assigned different interrupts or are they sharing one.
Best,
J
Soon as I have those results I'll post them here, thx.
Have you tried disabling all hardware offloading? Verify with ifconfig ...
Hi Patrick - I think I may have seen you over on a NAS forum, if same person good to see you here as well :D
I believe by default all hardware offloading is already turned off, or are you referring to anything besides the 3 gui selections for TSO, LRO, and CRC? If there's a different spot in the gui or tunable that I should check, will do so.
Best,
J
Check with ifconfig if the UI actually did disable TXCSUM and RXCSUM. I had abysmal performance on DigitalOcean droplets with FreeBSD and pf NAT until I disabled that. If I remember correctly, because I ended up just disabling everything.
Sorry it took me so long to get back to this, life intervened....
To try and get to the bottom of this, I put a fresh opnsense install on the i7/new router and ran an iperf test between it and a windows client on the same LAN (all ports are set to auto, all ports negotiated 1000MF). The windows client is 172.30.30.40, and the opnsense router is 172.30.30.1. Results:
C:\Users\Admin\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3 -c 172.30.30.1 -p 34102
Connecting to host 172.30.30.1, port 34102
[ 4] local 172.30.30.40 port 51088 connected to 172.30.30.1 port 34102
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 64.2 MBytes 539 Mbits/sec
[ 4] 1.00-2.00 sec 67.4 MBytes 565 Mbits/sec
[ 4] 2.00-3.00 sec 59.8 MBytes 501 Mbits/sec
[ 4] 3.00-4.00 sec 58.6 MBytes 491 Mbits/sec
[ 4] 4.00-5.00 sec 67.5 MBytes 567 Mbits/sec
[ 4] 5.00-6.00 sec 68.1 MBytes 571 Mbits/sec
[ 4] 6.00-7.00 sec 68.1 MBytes 572 Mbits/sec
[ 4] 7.00-8.00 sec 58.5 MBytes 491 Mbits/sec
[ 4] 8.00-9.00 sec 68.0 MBytes 570 Mbits/sec
[ 4] 9.00-10.00 sec 60.2 MBytes 506 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 640 MBytes 537 Mbits/sec sender
[ 4] 0.00-10.00 sec 640 MBytes 537 Mbits/sec receiver
iperf Done.
This would not seem to be correct throughput for a 1gb lan link. I also tried with and without hardware offloading per Patricks suggestion, UI changes are reflected in ifconfig output but bandwidth tests via 'broadband speed test site' still show about 250mbps. I'm not sure where to go with this other than replace hardware, but the nics are intel pro1000 (emX driver) so I'd think they are the most likely to work. These are plugged into standard PCI (not pcie, but regular pci) slots, maybe thats the ceiling I'm butting my head against?
Any advice is most appreciated!
PCI is very bandwidth constrained compared to PCIe - and with 4 gbe nics on PCI you're almost certainly running into those constraints
I'd replace those old pro1000 cards with either a couple of iintel i21x series cards (if you only need a couple of ports ) or if you really need 4 ports something like an i350-t4 - again well supported NICs for freeBSD.
you'll also probably want to look at enabling RSS once you've got the cards replaced - the i7-3770 has pretty low single-core perf by modern standards and without RSS you can end up limited by that
found a block diagram for a 'typical' h77/z77 board - the 2 pci slots are off a pci/pcie bridge chip and the upstream link is pcie3x1 - that should be heaps for 4 x gbe links ( presuming the bridge chip isn't utter rubbish )
I'd check core loading on the firewall when running a speed test from a lan client to wan and see if you're saturating a single core ( ssh in to opnsense, run 'top -P' and watch the 'interrupt and 'idle' column ).
I dug up the pro/1000 mt dual specs pdf - no mention of rss on it at all that I can see, freedBSD used to have a separate 'emx' driver with rss support for some of those older cards but it doesn't list the 82546, just newer variants ( https://man.dragonflybsd.org/?command=emx§ion=4 ). Suspect you need a new NIC
Could you please reverse the iperf3 connection with '-R' so the server sends, I'm curious about the speed.
iperf3 -R -c 172.30.30.1 -p 34102
I looked at photos of the mainboard and the PCI slots seem to be PCI 2.3 | 32 Bit | 33 MHz | 0.133 GByte/s | 5V while the NICs probably are PCI-X 1.0 | 64 Bit | 66 MHz | 0.533 GByte/s | 3.3V.
PCI-X is backward compatible, that's why it works, but there must be some overhang from the PCI-X cards at the end of the slot, could you please make a photo?
So the theoretical bandwidth (unidirectional) = 0.133 GByte/s * 8 = 1.064 Gbit/s shared by 2 ports = 0.532 GBit/s.
A PCI Express to PCI Extended adapter would be nice but this only seems to exist as PCIe to PCI 32 Bit 5 V adapter.
I seem to recall that all PCI slots shared the same bandwidth instead of having separate lanes like PCIe. So this problem would remain regardless of whether using PCI or PCI-X NICs.
I'm running a 4th gen i5 using a PCIe quad NIC along with 10g and 2.5g NICs and I'm able to pull full gig speeds(higher when applicable), so I don't think the i7 is the bottleneck unless there's IDS/IPS, etc.
Quote from: johndchch on September 18, 2023, 11:10:13 AM
PCI is very bandwidth constrained compared to PCIe - and with 4 gbe nics on PCI you're almost certainly running into those constraints
I scrapped this from google, and then did the math to clean it up and include all cases. Supposedly:
PCI 32-bit, 33 MHz: 1067 Mbit/s or 133 MB/s
PCI 32-bit, 66 MHz: 2128 Mbit/s or 266 MB/s
PCI 64-bit, 33 MHz: 2128 Mbit/s or 266 MB/s
PCI 64-bit, 66 MHz: 4264 Mbit/s or 533 MB/s
What Gig-E (1000base-X) should be: 1000 Mbit/s or 125 MB/s
What I'm getting: 537 Mbits/s or 67 MB/sSo if 1gb is 125MB/s, worst case scenario in above (admittedly max best case), 133MB/s is what the slot can deliver. So I'd not think I'm running into that. Even less likely if it's a 66mhz or 64bit slot.
More below....
Quote from: johndchch on September 18, 2023, 11:25:48 AM
found a block diagram for a 'typical' h77/z77 board - the 2 pci slots are off a pci/pcie bridge chip and the upstream link is pcie3x1 - that should be heaps for 4 x gbe links ( presuming the bridge chip isn't utter rubbish )
I'd check core loading on the firewall when running a speed test from a lan client to wan and see if you're saturating a single core ( ssh in to opnsense, run 'top -P' and watch the 'interrupt and 'idle' column ).
I dug up the pro/1000 mt dual specs pdf - no mention of rss on it at all that I can see, freedBSD used to have a separate 'emx' driver with rss support for some of those older cards but it doesn't list the 82546, just newer variants ( https://man.dragonflybsd.org/?command=emx§ion=4 ). Suspect you need a new NIC
My board is using B75 chipset. Slots are as follows:
1 PCI Express 3.0/2.0 x16 slot (pcie 3 speed is only supported by intel 3rd gen core processors)
1 PCI Express 2.0 x4 slot
2 PCI slots (dont know if these are 32 or 64 bit, and if they are 33 or 66 mhz)Perhaps important detail:
Due to the 2U chassis I'm using, expansion cards can only be put into a 2 slot riser card horizontally. This riser card is bolted to the case in such a way that the only possible slot I can use is PCI2. Below is a picture - riser card is in pci2 and on the riser card are two pci slots (standard pci, not pcie). In each of those two slots is a 2 port intel 1000 nic (more on those below). In addition, there is a cable ending in a paddleboard from the riser card that plugs into PCI1 on the mainboard, only necessary if using two cards in the riser. I haven't rung out the pins, but I suspect that secondary cable/paddleboard is primarily for additional power. I would expect the two cards to share an interrupt, but perhaps the paddleboard takes care of that too.
The nic cards I am using are intel PRO/1000 MT, and are recognized by freebsd as:
em0: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xc0c0-0xc0ff mem 0xf7c60000-0xf7c7ffff irq 19 at device 0.0 on pci7
em1: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xc080-0xc0bf mem 0xf7c40000-0xf7c5ffff irq 16 at device 0.1 on pci7
em2: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xc040-0xc07f mem 0xf7c20000-0xf7c3ffff irq 16 at device 1.0 on pci7
em3: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xc000-0xc03f mem 0xf7c00000-0xf7c1ffff irq 17 at device 1.1 on pci7I will do the additional iperf test you mention as well in a subsequent post. See attached picture below where the paddleboard is in PCIE1.
Thanks!
Quote from: vpx on September 19, 2023, 03:23:49 PM
Could you please reverse the iperf3 connection with '-R' so the server sends, I'm curious about the speed.
iperf3 -R -c 172.30.30.1 -p 34102
Per your request, I've added an iperf with the -R option:
C:\Users\Admin\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3 -R -c 172.30.30.1 -p 23175
Connecting to host 172.30.30.1, port 23175
Reverse mode, remote host 172.30.30.1 is sending
[ 4] local 172.30.30.40 port 56999 connected to 172.30.30.1 port 23175
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 30.2 MBytes 254 Mbits/sec
[ 4] 1.00-2.00 sec 30.1 MBytes 253 Mbits/sec
[ 4] 2.00-3.00 sec 30.1 MBytes 253 Mbits/sec
[ 4] 3.00-4.00 sec 30.1 MBytes 253 Mbits/sec
[ 4] 4.00-5.00 sec 30.1 MBytes 252 Mbits/sec
[ 4] 5.00-6.00 sec 30.2 MBytes 254 Mbits/sec
[ 4] 6.00-7.00 sec 30.1 MBytes 253 Mbits/sec
[ 4] 7.00-8.00 sec 30.1 MBytes 252 Mbits/sec
[ 4] 8.00-9.00 sec 30.1 MBytes 252 Mbits/sec
[ 4] 9.00-10.00 sec 30.9 MBytes 259 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 302 MBytes 254 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 302 MBytes 254 Mbits/sec receiver
iperf Done.
Quote from: vpx23 on September 19, 2023, 07:07:30 PM
I looked at photos of the mainboard and the PCI slots seem to be PCI 2.3 | 32 Bit | 33 MHz | 0.133 GByte/s | 5V while the NICs probably are PCI-X 1.0 | 64 Bit | 66 MHz | 0.533 GByte/s | 3.3V.
PCI-X is backward compatible, that's why it works, but there must be some overhang from the PCI-X cards at the end of the slot, could you please make a photo?
So the theoretical bandwidth (unidirectional) = 0.133 GByte/s * 8 = 1.064 Gbit/s shared by 2 ports = 0.532 GBit/s.
A PCI Express to PCI Extended adapter would be nice but this only seems to exist as PCIe to PCI 32 Bit 5 V adapter.
Yes, there is overhang from stuffing two PCIX cards into two std PCI slots. See above post where I describe the riser card and the paddleboard. Photo included there.
That's a crazy setup, 2 PCI-X dual port cards in one PCI slot.
But the PCI slot on the board is definitely 32 bit because this is shorter than a 64 bit port, so the max. speed would be 0.266 GByte/s.
Did you try the speed tests with only one NIC plugged?
To find out the PCI port speed we could try these commands in OPNsense:
pciconf -lvc
lspci -vvv
https://unix.stackexchange.com/questions/541688/how-to-see-pcie-device-info-on-freebsd
What do you mean with paddleboard, could you show a photo please?
Yes, it's a crazy setup but it's the case, cards, and mainboard I had sitting around already. Didn't have to buy anything except a new cpu fan.
I did not try with only one NIC plugged, but can easily do so.
pciconf output is:
Enter an option: 8
root@OPNsense:~ # pciconf -lvc
hostb0@pci0:0:0:0: class=0x060000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0150 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = 'Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller'
class = bridge
subclass = HOST-PCI
cap 09[e0] = vendor (length 12) Intel cap 0 version 1
pcib1@pci0:0:1:0: class=0x060400 rev=0x09 hdr=0x01 vendor=0x8086 device=0x0151 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = 'Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port'
class = bridge
subclass = PCI-PCI
cap 0d[88] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message
cap 10[a0] = PCI-Express 2 root port max data 256(256)
max read 128
link x0(x16) speed 0.0(8.0) ASPM disabled(L0s/L1)
slot 1 power limit 75000 mW
ecap 0002[100] = VC 1 max VC0
ecap 0005[140] = Root Complex Link Declaration 1
ecap 0019[d94] = PCIe Sec 1 lane errors 0
vgapci0@pci0:0:2:0: class=0x030000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0162 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = 'IvyBridge GT2 [HD Graphics 4000]'
class = display
subclass = VGA
cap 05[90] = MSI supports 1 message
cap 01[d0] = powerspec 2 supports D0 D3 current D0
cap 13[a4] = PCI Advanced Features: FLR TP
none0@pci0:0:22:0: class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c3a subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family MEI Controller'
class = simple comms
cap 01[50] = powerspec 3 supports D0 D3 current D0
cap 05[8c] = MSI supports 1 message, 64 bit
ehci0@pci0:0:26:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
class = serial bus
subclass = USB
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
cap 13[98] = PCI Advanced Features: FLR TP
hdac0@pci0:0:27:0: class=0x040300 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c20 subvendor=0x1043 subdevice=0x841b
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family High Definition Audio Controller'
class = multimedia
subclass = HDA
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR NS
max read 128
ecap 0002[100] = VC 1 max VC1
ecap 0005[130] = Root Complex Link Declaration 1
pcib2@pci0:0:28:0: class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c10 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family PCI Express Root Port 1'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x0(x4) speed 0.0(5.0) ASPM disabled(L0s/L1)
slot 0 power limit 250 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[a0] = powerspec 2 supports D0 D3 current D0
pcib3@pci0:0:28:4: class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c18 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family PCI Express Root Port 5'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
slot 4 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[a0] = powerspec 2 supports D0 D3 current D0
pcib4@pci0:0:28:5: class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1a subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family PCI Express Root Port 6'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
slot 5 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[a0] = powerspec 2 supports D0 D3 current D0
pcib5@pci0:0:28:6: class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1c subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family PCI Express Root Port 7'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
slot 6 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[a0] = powerspec 2 supports D0 D3 current D0
pcib6@pci0:0:28:7: class=0x060401 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '82801 PCI Bridge'
class = bridge
subclass = PCI-PCI
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
slot 7 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
cap 01[a0] = powerspec 2 supports D0 D3 current D0
ehci1@pci0:0:29:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
class = serial bus
subclass = USB
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
cap 13[98] = PCI Advanced Features: FLR TP
isab0@pci0:0:31:0: class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c4a subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = 'H67 Express Chipset LPC Controller'
class = bridge
subclass = PCI-ISA
cap 09[e0] = vendor (length 12) Intel cap 1 version 0
features: AMT, 4 PCI-e x1 slots
atapci1@pci0:0:31:2: class=0x01018f rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c00 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 0-3)'
class = mass storage
subclass = ATA
cap 01[70] = powerspec 3 supports D0 D3 current D0
cap 13[b0] = PCI Advanced Features: FLR TP
ichsmb0@pci0:0:31:3: class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c22 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family SMBus Controller'
class = serial bus
subclass = SMBus
atapci2@pci0:0:31:5: class=0x010185 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c08 subvendor=0x1043 subdevice=0x844d
vendor = 'Intel Corporation'
device = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 4-5)'
class = mass storage
subclass = ATA
cap 01[70] = powerspec 3 supports D0 D3 current D0
cap 13[b0] = PCI Advanced Features: FLR TP
atapci0@pci0:3:0:0: class=0x010185 rev=0x00 hdr=0x00 vendor=0x1106 device=0x0415 subvendor=0x1043 subdevice=0x838f
vendor = 'VIA Technologies, Inc.'
device = 'VT6415 PATA IDE Host Controller'
class = mass storage
subclass = ATA
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 05[70] = MSI supports 1 message, 64 bit, vector masks
cap 10[90] = PCI-Express 1 legacy endpoint max data 128(128)
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[130] = Serial 1 004063ffff634000
xhci0@pci0:4:0:0: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x1042 subvendor=0x1043 subdevice=0x8488
vendor = 'ASMedia Technology Inc.'
device = 'ASM1042 SuperSpeed USB Host Controller'
class = serial bus
subclass = USB
cap 05[50] = MSI supports 8 messages, 64 bit
cap 11[68] = MSI-X supports 8 messages, enabled
Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
cap 01[78] = powerspec 3 supports D0 D3 current D0
cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) NS
max read 512
link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
ecap 0002[100] = VC 1 max VC0
re0@pci0:5:0:0: class=0x020000 rev=0x06 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8432
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet
cap 01[40] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit
cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(256)
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM disabled
cap 11[b0] = MSI-X supports 4 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x800]
cap 03[d0] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0002[140] = VC 1 max VC0
ecap 0003[160] = Serial 1 14000000684ce000
pcib7@pci0:6:0:0: class=0x060401 rev=0x01 hdr=0x01 vendor=0x1b21 device=0x1080 subvendor=0x1043 subdevice=0x8489
vendor = 'ASMedia Technology Inc.'
device = 'ASM1083/1085 PCIe to PCI Bridge'
class = bridge
subclass = PCI-PCI
cap 0d[c0] = PCI Bridge subvendor=0x1043 subdevice=0x8489
em0@pci0:7:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
vendor = 'Intel Corporation'
device = '82546EB Gigabit Ethernet Controller (Copper)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
em1@pci0:7:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
vendor = 'Intel Corporation'
device = '82546EB Gigabit Ethernet Controller (Copper)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
em2@pci0:7:1:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
vendor = 'Intel Corporation'
device = '82546EB Gigabit Ethernet Controller (Copper)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
em3@pci0:7:1:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
vendor = 'Intel Corporation'
device = '82546EB Gigabit Ethernet Controller (Copper)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
root@OPNsense:~ #
lspci isn't on stock freebsd, and I don't see it in the plugins or packages. I'd prefer to keep things "per ui".
The paddleboard - you can see it in the previous picture. In PCI1 (left one of the two pci slots) is the riser card, with the two cards plugged into the left. At the right side rear/bottom you can see a few (around six or eight) multicolored wires coming off the riser card. These go to a black single row edge connector, that you can see going to PCI2 (the right slot of the two pci slots). If you look at the right slot closely you can see there is a paddleboard plugged into the slot, it brings those six or eight signals from pci2 to the riser. I dont know if it's just getting additional power from the other slot, or if it provides more than that (addressing, etc). The paddleboard is long and thin, pushed into the entire pci2, with only a fraction of an inch sticking above so the pin connector can attach.
OK, now I saw the extra board, it's probably only for power because I don't see any data cable.
This must be the PCIe to PCI Bridge Controller on your mainboard:
https://www.asmedia.com.tw/product/E8AyQ44sX3zPbruF/2f2yq89xz7URBZqD
It converts PCIe 1.1 x1 (0.250 GByte/s) to 2x PCI 32-bit with 33 MHz. I don't know why they made it so slow because the board is from 2012 when PCI 3.0 was already 8 years old.
According to your output the Intel PRO/1000 MT 82546EB are PCI-X 1.0 | 64 Bit | 133 MHz | 1.066 GByte/s | 3.3V. So they are not the problem, they should easily reach 940 MBit/s. Even with both ports and full-duplex at the same time (3,760 MBit/s) they wouldn't saturate the possible bandwidth of PCI-X. Only the 4 port cards with 7,520 Mbit/s + overhead could saturate the max. 8,528 MBit/s bandwidth.
If it's not the pci slot and it's not the pcix cards, must be the riser lash-up. Changing that is a mechanical problem. I'll need a different case and likely a different mainboard.
I need 3 ports, so I'm rounding to 4. I was hoping to use off the shelf stuff but that may not be possible.
I don't understand why we're still toying around with 20 year old PCI cards? Based on the picture provided it looks like there is a blue PCIe x16 slot on that board. Can you purchase a low profile PCIe quad port card? This will immediately fix the issue and you'll get full gigabit easily. Also, a quad port i340 or i350 card will use the newer igb driver as well with more queues.
I suspect I could do that, I thought there was something in the mainboard manual that said that particular slot could *only* be used for video cards. I may be wrong about that. But....
As I said several times above including the very previous post, the only case I had available does not allow that. Hence the reason I said I'd have to get a new case for sure and possibly a mainboard.
As a side note, its interesting to find and understand the problem rather than just sweep it under the rug.
Admittedly I'm being a bit tongue in cheek here as I was looking at the riser splitting 1 PCI 32bit slot for 2x 64bit cards, running in 32bit mode. I didn't realize it was even possible to jerry rig PCI this much. ;)
For example, you could keep the case and use one of these ribbon extensions to mount a PCIe card horizontally in one of those slots.
https://www.ebay.com/itm/403649610768 (I don't have direct experience with this seller, just an example that won't constrain you to PCI slots and you can still keep that case)
Agreed. Tomorrow I'm going to take a second hard look at the metalwork and see if this could be done.
Plugging it into the pciex16 would be easy with a ribbon extension but 1) The mainboard manual seems to say 'video card only' but I'm not sure thats what they really mean, and 2) one end of the card bracket would be held captive by the screw, but I'm not sure the bottom would be. I'd certainly prefer a i340 to the pro1000.
I just didn't want to go down this path if it was something else causing the issue. I do sincerely appreciate the input.
At this point we're slapping stuff together. I don't say this in a bad way, I love slapping rigs together and making it work so don't take that as a negative. ;)
I would think you would still have some good support for a horizontal mounted card even without it plugged in to a PCB riser. You'll have the screw clamping the top of the card to the case and most of the cases I've worked with have a small t-slot at the bottom for the end of the expansion card to nest in. Once you have your ethernet cables plugged in, those too will stabilize the slot somewhat. It should be workable. If all else fails, duct tape the end of the expansion card to the inside of the case and it should be serviceable. It won't be pretty but it should work and get you symmetric gigabit throughput.
This is a genuine Intel I340 for not much money, I've used this seller before as they have been good to deal with: https://www.ebay.com/itm/235149064664
If you have a Micro Center near by they also should sell PCIe ribbon extenders and they may be higher quality than the ones you can find on ebay or amazon.
Let us know how you end up.
I will, thx. I looked at specs on the i340, and if I'm reading it right that card supports pcie 2.0 x4.
In addition to the pcie 3.0/2.0 x16 slot we've been discussing, the board does have a pcie 2.0 x4 slot at the far left, so I might go that route insted of using the x16 slot.
I think the bracket arrangement had no support at the bottom of the bracket, all cards were in line and the bottoms in a continuous slot. In other words, they relied on the card slot on the bracket for support. I could be wrong but will definitely look tomorrow. If this gets me close to 1gb/port, I'd be very happy. Thanks again for the advice.
Just an update.... arrived from amazon is a pciex16 riser cable and from ebay an intel i340-t4. They arrived while we are out of town though, should be back there this coming saturday.
Soon as I get back I'll see about metalwork (if any) required to get the card to fit properly and run the cable to the pciex16 port. I'll re-run iperf3 and we'll see if the performance goes as expected for 1gb. Will advise & thx!
Tangentially - for the farm house I got a supermicro SYS-5018A-FTN4 which contains an A1SRI-2758F (atom 2758). I immediately ran an iperf3 from a local windows machine to that servers LAN port and got 300mbps, far below what I'd expect out of a 1gbps interface. So I've got two different machines at two different locations with (what I suspect is) low speed on the ethernet ports. This makes me wonder if I'm running the tests wrong or interpreting the results wrong.... on both systems. The wan connection there is only 25mbps (rural internet service), so it's not a problem but still would like to solve it. One at a time :D
I guess you don't have an ideal setup to do these tests.
iperf3 client<--->test object<--->iperf3 server
And ideally both client and server are very powerful and only run dedicated iperf3, so no other process will disturb the throughput. The test object should always be the bottleneck.
When you have 3 systems with weak CPUs in the setup you will have 3 potential bottlenecks. Even worse when the test object itself is the iperf3 client or server.
I agree with what vpx23 mentioned for iperf testing, don't host a server or client session on the router itself. Its job is to route, so have a server sitting on one side of the router and a client sitting on another side and run the tests pushing/pulling traffic through the router.
It's also worth noting any other services that may be configured on OPNsense. Any IPS/IDS services can also slow down throughput. Ideally the tests should be done with a bare minimal install, slowly configure and turn on components to see where the bottleneck is happening. If you're running IDS/IPS usually the CPU will be the limiting factor first.
Another thing worth checking is cable quality and the length of the runs. You mention this is at a farmhouse so are we using any kind of powerline over ethernet components or other oddities? Those can have an impact as well.
C:\Users\Admin\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3 -c 172.30.30.1 -p 51310
Connecting to host 172.30.30.1, port 51310
[ 4] local 172.30.30.40 port 61282 connected to 172.30.30.1 port 51310
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 113 MBytes 951 Mbits/sec
[ 4] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 4] 2.00-3.00 sec 113 MBytes 949 Mbits/sec
[ 4] 3.00-4.00 sec 113 MBytes 949 Mbits/sec
[ 4] 4.00-5.00 sec 113 MBytes 949 Mbits/sec
[ 4] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 4] 6.00-7.00 sec 113 MBytes 949 Mbits/sec
[ 4] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 4] 8.00-9.00 sec 113 MBytes 949 Mbits/sec
[ 4] 9.00-10.00 sec 113 MBytes 949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.11 GBytes 949 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.11 GBytes 949 Mbits/sec receiver
iperf Done.
These speeds are 4x what I was getting before, and right at what I'd expect for 1gb copper. YAY! In addition, speakeasy.net speed test now shows 950mbps instead of 250mbps i was getting in previous iperf3 posts in this thread.
The change made: switched from two dual port pcie-x cards plugged into a single pci (non Express) slot via a riser card, to a single i340 4 port card plugged into the pcieX16 slot normally reserved for addon video. Easy :D
I can't begin to express how much I appreciate the input from everyone steering me towards the solution. THANK YOU!
PS - totally understand the comments about best iperf3 testbed. In this case, swapping routers made it slow, so pretty easy to see after swapping cards in new router that problem is solved. If it had NOT solved the problem, I would have headed down the path of setting up a testbed with router inbetween sender/receiver for iperf.
Nice! always fun when a cheap fix does the trick.
I've used those i340 server card pulls for years in various routers. They're quite reliable, I haven't had one fail yet. Hopefully it gives you many years of service.
I did purchase from the ebay seller you linked.... https://www.ebay.com/itm/235149064664
Item was well priced (a bit low I thought), well packed, shipped quick, great condition. Am happy with them, thank you for that.
At those prices, it would be silly not to buy a spare for each location just to have a hot spare on hand whether it ever gets used or not.