Suggestions for troubleshooting slow nat performance (throughput)?

[jwest]

I looked at photos of the mainboard and the PCI slots seem to be PCI 2.3 | 32 Bit | 33 MHz | 0.133 GByte/s | 5V while the NICs probably are PCI-X 1.0 | 64 Bit | 66 MHz | 0.533 GByte/s | 3.3V.

PCI-X is backward compatible, that's why it works, but there must be some overhang from the PCI-X cards at the end of the slot, could you please make a photo?

So the theoretical bandwidth (unidirectional) = 0.133 GByte/s * 8 = 1.064 Gbit/s shared by 2 ports = 0.532 GBit/s.

A PCI Express to PCI Extended adapter would be nice but this only seems to exist as PCIe to PCI 32 Bit 5 V adapter.

Yes, there is overhang from stuffing two PCIX cards into two std PCI slots. See above post where I describe the riser card and the paddleboard. Photo included there.

[vpx23]

That's a crazy setup, 2 PCI-X dual port cards in one PCI slot.

But the PCI slot on the board is definitely 32 bit because this is shorter than a 64 bit port, so the max. speed would be 0.266 GByte/s.

Did you try the speed tests with only one NIC plugged?

To find out the PCI port speed we could try these commands in OPNsense:

Code: [Select]

pciconf -lvc

Code: [Select]

lspci -vvv
https://unix.stackexchange.com/questions/541688/how-to-see-pcie-device-info-on-freebsd

What do you mean with paddleboard, could you show a photo please?

[jwest]

Yes, it's a crazy setup but it's the case, cards, and mainboard I had sitting around already. Didn't have to buy anything except a new cpu fan.

I did not try with only one NIC plugged, but can easily do so.

pciconf output is:

Code: [Select]

Enter an option: 8

root@OPNsense:~ # pciconf -lvc
hostb0@pci0:0:0:0:      class=0x060000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0150 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
    cap 09[e0] = vendor (length 12) Intel cap 0 version 1
pcib1@pci0:0:1:0:       class=0x060400 rev=0x09 hdr=0x01 vendor=0x8086 device=0x0151 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[88] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 1 message
    cap 10[a0] = PCI-Express 2 root port max data 256(256)
                 max read 128
                 link x0(x16) speed 0.0(8.0) ASPM disabled(L0s/L1)
                 slot 1 power limit 75000 mW
    ecap 0002[100] = VC 1 max VC0
    ecap 0005[140] = Root Complex Link Declaration 1
    ecap 0019[d94] = PCIe Sec 1 lane errors 0
vgapci0@pci0:0:2:0:     class=0x030000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0162 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'IvyBridge GT2 [HD Graphics 4000]'
    class      = display
    subclass   = VGA
    cap 05[90] = MSI supports 1 message
    cap 01[d0] = powerspec 2  supports D0 D3  current D0
    cap 13[a4] = PCI Advanced Features: FLR TP
none0@pci0:0:22:0:      class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c3a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family MEI Controller'
    class      = simple comms
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[8c] = MSI supports 1 message, 64 bit
ehci0@pci0:0:26:0:      class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
hdac0@pci0:0:27:0:      class=0x040300 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c20 subvendor=0x1043 subdevice=0x841b
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR NS
                 max read 128
    ecap 0002[100] = VC 1 max VC1
    ecap 0005[130] = Root Complex Link Declaration 1
pcib2@pci0:0:28:0:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c10 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x0(x4) speed 0.0(5.0) ASPM disabled(L0s/L1)
                 slot 0 power limit 250 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib3@pci0:0:28:4:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c18 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 5'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 4 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib4@pci0:0:28:5:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 6'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
                 slot 5 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib5@pci0:0:28:6:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1c subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 7'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 6 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib6@pci0:0:28:7:      class=0x060401 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 7 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ehci1@pci0:0:29:0:      class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
isab0@pci0:0:31:0:      class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c4a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'H67 Express Chipset LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
    cap 09[e0] = vendor (length 12) Intel cap 1 version 0
                 features: AMT, 4 PCI-e x1 slots
atapci1@pci0:0:31:2:    class=0x01018f rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c00 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 0-3)'
    class      = mass storage
    subclass   = ATA
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 13[b0] = PCI Advanced Features: FLR TP
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c22 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
atapci2@pci0:0:31:5:    class=0x010185 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c08 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 4-5)'
    class      = mass storage
    subclass   = ATA
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 13[b0] = PCI Advanced Features: FLR TP
atapci0@pci0:3:0:0:     class=0x010185 rev=0x00 hdr=0x00 vendor=0x1106 device=0x0415 subvendor=0x1043 subdevice=0x838f
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT6415 PATA IDE Host Controller'
    class      = mass storage
    subclass   = ATA
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[70] = MSI supports 1 message, 64 bit, vector masks
    cap 10[90] = PCI-Express 1 legacy endpoint max data 128(128)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[130] = Serial 1 004063ffff634000
xhci0@pci0:4:0:0:       class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x1042 subvendor=0x1043 subdevice=0x8488
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1042 SuperSpeed USB Host Controller'
    class      = serial bus
    subclass   = USB
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
    ecap 0002[100] = VC 1 max VC0
re0@pci0:5:0:0: class=0x020000 rev=0x06 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8432
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(256)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM disabled
    cap 11[b0] = MSI-X supports 4 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x800]
    cap 03[d0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0002[140] = VC 1 max VC0
    ecap 0003[160] = Serial 1 14000000684ce000
pcib7@pci0:6:0:0:       class=0x060401 rev=0x01 hdr=0x01 vendor=0x1b21 device=0x1080 subvendor=0x1043 subdevice=0x8489
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1083/1085 PCIe to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[c0] = PCI Bridge subvendor=0x1043 subdevice=0x8489
em0@pci0:7:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em1@pci0:7:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em2@pci0:7:1:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em3@pci0:7:1:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
root@OPNsense:~ #
lspci isn't on stock freebsd, and I don't see it in the plugins or packages. I'd prefer to keep things "per ui".

The paddleboard - you can see it in the previous picture. In PCI1 (left one of the two pci slots) is the riser card, with the two cards plugged into the left. At the right side rear/bottom you can see a few (around six or eight) multicolored wires coming off the riser card. These go to a black single row edge connector, that you can see going to PCI2 (the right slot of the two pci slots). If you look at the right slot closely you can see there is a paddleboard plugged into the slot, it brings those six or eight signals from pci2 to the riser. I dont know if it's just getting additional power from the other slot, or if it provides more than that (addressing, etc). The paddleboard is long and thin, pushed into the entire pci2, with only a fraction of an inch sticking above so the pin connector can attach.

[vpx23]

OK, now I saw the extra board, it's probably only for power because I don't see any data cable.

This must be the PCIe to PCI Bridge Controller on your mainboard:

https://www.asmedia.com.tw/product/E8AyQ44sX3zPbruF/2f2yq89xz7URBZqD

It converts PCIe 1.1 x1 (0.250 GByte/s) to 2x PCI 32-bit with 33 MHz. I don't know why they made it so slow because the board is from 2012 when PCI 3.0 was already 8 years old.

According to your output the Intel PRO/1000 MT 82546EB are PCI-X 1.0 | 64 Bit | 133 MHz | 1.066 GByte/s | 3.3V. So they are not the problem, they should easily reach 940 MBit/s. Even with both ports and full-duplex at the same time (3,760 MBit/s) they wouldn't saturate the possible bandwidth of PCI-X. Only the 4 port cards with 7,520 Mbit/s + overhead could saturate the max. 8,528 MBit/s bandwidth.

[jwest]

If it's not the pci slot and it's not the pcix cards, must be the riser lash-up. Changing that is a mechanical problem. I'll need a different case and likely a different mainboard.

I need 3 ports, so I'm rounding to 4. I was hoping to use off the shelf stuff but that may not be possible.

[opnfwb]

I don't understand why we're still toying around with 20 year old PCI cards? Based on the picture provided it looks like there is a blue PCIe x16 slot on that board. Can you purchase a low profile PCIe quad port card? This will immediately fix the issue and you'll get full gigabit easily. Also, a quad port i340 or i350 card will use the newer igb driver as well with more queues.

[jwest]

I suspect I could do that, I thought there was something in the mainboard manual that said that particular slot could *only* be used for video cards. I may be wrong about that. But....

As I said several times above including the very previous post, the only case I had available does not allow that. Hence the reason I said I'd have to get a new case for sure and possibly a mainboard.

As a side note, its interesting to find and understand the problem rather than just sweep it under the rug.

[opnfwb]

Admittedly I'm being a bit tongue in cheek here as I was looking at the riser splitting 1 PCI 32bit slot for 2x 64bit cards, running in 32bit mode. I didn't realize it was even possible to jerry rig PCI this much.

For example, you could keep the case and use one of these ribbon extensions to mount a PCIe card horizontally in one of those slots.
https://www.ebay.com/itm/403649610768 (I don't have direct experience with this seller, just an example that won't constrain you to PCI slots and you can still keep that case)

[jwest]

Agreed. Tomorrow I'm going to take a second hard look at the metalwork and see if this could be done.

Plugging it into the pciex16 would be easy with a ribbon extension but 1) The mainboard manual seems to say 'video card only' but I'm not sure thats what they really mean, and 2) one end of the card bracket would be held captive by the screw, but I'm not sure the bottom would be. I'd certainly prefer a i340 to the pro1000.

I just didn't want to go down this path if it was something else causing the issue. I do sincerely appreciate the input.

[opnfwb]

At this point we're slapping stuff together. I don't say this in a bad way, I love slapping rigs together and making it work so don't take that as a negative.

I would think you would still have some good support for a horizontal mounted card even without it plugged in to a PCB riser. You'll have the screw clamping the top of the card to the case and most of the cases I've worked with have a small t-slot at the bottom for the end of the expansion card to nest in. Once you have your ethernet cables plugged in, those too will stabilize the slot somewhat. It should be workable. If all else fails, duct tape the end of the expansion card to the inside of the case and it should be serviceable. It won't be pretty but it should work and get you symmetric gigabit throughput.

This is a genuine Intel I340 for not much money, I've used this seller before as they have been good to deal with: https://www.ebay.com/itm/235149064664

If you have a Micro Center near by they also should sell PCIe ribbon extenders and they may be higher quality than the ones you can find on ebay or amazon.

Let us know how you end up.

[jwest]

I will, thx. I looked at specs on the i340, and if I'm reading it right that card supports pcie 2.0 x4.

In addition to the pcie 3.0/2.0 x16 slot we've been discussing, the board does have a pcie 2.0 x4 slot at the far left, so I might go that route insted of using the x16 slot.

I think the bracket arrangement had no support at the bottom of the bracket, all cards were in line and the bottoms in a continuous slot. In other words, they relied on the card slot on the bracket for support. I could be wrong but will definitely look tomorrow. If this gets me close to 1gb/port, I'd be very happy. Thanks again for the advice.

[jwest]

Just an update.... arrived from amazon is a pciex16 riser cable and from ebay an intel i340-t4. They arrived while we are out of town though, should be back there this coming saturday.

Soon as I get back I'll see about metalwork (if any) required to get the card to fit properly and run the cable to the pciex16 port. I'll re-run iperf3 and we'll see if the performance goes as expected for 1gb. Will advise & thx!

Tangentially - for the farm house I got a supermicro SYS-5018A-FTN4 which contains an A1SRI-2758F (atom 2758). I immediately ran an iperf3 from a local windows machine to that servers LAN port and got 300mbps, far below what I'd expect out of a 1gbps interface. So I've got two different machines at two different locations with (what I suspect is) low speed on the ethernet ports. This makes me wonder if I'm running the tests wrong or interpreting the results wrong.... on both systems. The wan connection there is only 25mbps (rural internet service), so it's not a problem but still would like to solve it. One at a time

[vpx23]

I guess you don't have an ideal setup to do these tests.

iperf3 client<--->test object<--->iperf3 server

And ideally both client and server are very powerful and only run dedicated iperf3, so no other process will disturb the throughput. The test object should always be the bottleneck.

When you have 3 systems with weak CPUs in the setup you will have 3 potential bottlenecks. Even worse when the test object itself is the iperf3 client or server.

[opnfwb]

I agree with what vpx23 mentioned for iperf testing, don't host a server or client session on the router itself. Its job is to route, so have a server sitting on one side of the router and a client sitting on another side and run the tests pushing/pulling traffic through the router.

It's also worth noting any other services that may be configured on OPNsense. Any IPS/IDS services can also slow down throughput. Ideally the tests should be done with a bare minimal install, slowly configure and turn on components to see where the bottleneck is happening. If you're running IDS/IPS usually the CPU will be the limiting factor first.

Another thing worth checking is cable quality and the length of the runs. You mention this is at a farmhouse so are we using any kind of powerline over ethernet components or other oddities? Those can have an impact as well.

[jwest]

Code: [Select]

C:\Users\Admin\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3 -c 172.30.30.1 -p 51310
Connecting to host 172.30.30.1, port 51310
[  4] local 172.30.30.40 port 61282 connected to 172.30.30.1 port 51310
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   951 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   949 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
[  4]   6.00-7.00   sec   113 MBytes   949 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.11 GBytes   949 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.11 GBytes   949 Mbits/sec                  receiver

iperf Done.
These speeds are 4x what I was getting before, and right at what I'd expect for 1gb copper. YAY! In addition, speakeasy.net speed test now shows 950mbps instead of 250mbps i was getting in previous iperf3 posts in this thread.

The change made: switched from two dual port pcie-x cards plugged into a single pci (non Express) slot via a riser card, to a single i340 4 port card plugged into the pcieX16 slot normally reserved for addon video. Easy

I can't begin to express how much I appreciate the input from everyone steering me towards the solution. THANK YOU!

PS - totally understand the comments about best iperf3 testbed. In this case, swapping routers made it slow, so pretty easy to see after swapping cards in new router that problem is solved. If it had NOT solved the problem, I would have headed down the path of setting up a testbed with router inbetween sender/receiver for iperf.