Messages

1

24.1 Legacy Series / dhcp client looping, and unplanned reboot???

« on: June 03, 2024, 07:34:56 pm »

Two issues I'm not sure how to proceed on.

1) For a few months, I've noticed that (if I'm reading the logs right) my wan port which gets IP via dhcp from my provider, seems to keep renewing like every 2 minutes forever. These two lines repeat every couple of minutes in system->logFiles->General

2024-06-03T12:23:15-05:00   Notice   dhclient   dhclient-script: Creating resolv.conf   
2024-06-03T12:23:15-05:00   Notice   dhclient   dhclient-script: Reason RENEW on igb3 executing

But my ISP says their lease period is 24 hours. Is there any likely cause for this on my end or is it for sure on the ISP's side?

2) My router just spontaneously rebooted. I see no way to view system logs from just before the reboot. Am I missing something?

I'm using opnsense 24.1.8-amd64 on a supermicro 5018A-FTN4, with Intel(R) Atom(TM) CPU C2758 @ 2.40GHz (8 cores, 8 threads) and 8gb ram.

Any input appreciated, I'm at a loss......

2

General Discussion / WAN link light drops randomly

« on: May 07, 2024, 09:50:37 pm »

I just want to run this by folks here to see if anyone has more/better ideas than I on the issue.

My setup:
opnsense_24.1.6 wan port igb3 -> poe injector -> isp modem

Randomly (sometimes every 5 or 10 minutes, sometimes a few hours, sometimes a day) I will notice my internet is down for a minute or two or 5, and then comes back up. I noticed that when this is happening, the link light on my wan port goes out. After 10 seconds or so, the link light comes on, activity flashes for a bit, then link light goes off again. Rinse and repeat, after a number of times eventually the link light stays on and everything works again.

From looking at my dashboard, in the gateway section under wan_dhcp, the ip address disappears when the link light goes out. When link light comes back on the IP re-appears in the gatewaysection, then disappears and gateway shows 'offline'. After repeating this cycle, at some point after a handful of retries the ip shows, and then the block says "online". To me this would seem like carrier/link signal lost, so dhcp tries to re-get its settings.

My wan port is set to dhcp, and my isp provides a sticky static public IP. According to them, the dhcp lease time is 24 hours. However when I look in the general logs, I see a dhcp.client renew on igb3 message repeating about every 2 minutes forever. I'd have thought it would renew at the 24 hour lease, not every 2 minutes or so? Is this an issue?

My guess from the above is that the drops are caused by loss of link signal from whatever is connected to my wan port, which would be the poe injector (2 port model: ac plug, data in, and data+poe out, provided by isp), or their modem, or the cables from router to poe and/or poe to modem.

Does anyone see this is likely a problem with my router vs. their poe adapter and modem?

They say that on their end they show their modem negotiates half-duplex with my wan port, when it should be full-plex, and that their equipment will shut down the connection if it doesnt negotiate to full duplex.

I'd have thought that if link dropped, I'd be getting console messages or messages in the log about lost carrier. I do not.

The ISP is coming out to replace all cables, the poe adapter, and possibly the modem. Is there anything in my description above that may be something on my end instead? Any help or pointers highly appreciated!

3

Virtual private networks / Re: Wireguard site2site MTU setting advice?

« on: March 05, 2024, 09:20:02 pm »

Ah thanks for the link, read it all and I'm off and running. Thanks so much!

4

Virtual private networks / Wireguard site2site MTU setting advice?

« on: March 05, 2024, 08:05:46 pm »

I have two locations. For each one there is a roadwarrior (wireguard) setup which is instance 1. Then there is a site to site VPN set up between the two (wireguard) which is instance 2. All this works well, but I'm curious about a point in the instructions maybe someone can advise on.

In the official opnsense wireguard site2site instructions (https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html) under step 4A and 4B there is a section that mentions setting up a normalization rule to prevent wireguard from trying to stuff 1500 bytes inside a 1500 byte packet. I omitted that step on each side. Not the whole step, I did add the rule to allow traffic, just the normalization rule I skipped.

It seems to work fine, but I noticed that when I look at the site2site interfaces they list 1420 as the MTU. Would this not indicate that without the rule, something is already smart enough to reduce the MTU? Or am I missing something still necessary? 1420 sounds about right, but the doc page says use less than or equal to 1380. I feel like I'm missing some understanding.... Any thoughts MOST appreciated!

5

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: October 03, 2023, 12:53:19 am »

I did purchase from the ebay seller you linked.... https://www.ebay.com/itm/235149064664

Item was well priced (a bit low I thought), well packed, shipped quick, great condition. Am happy with them, thank you for that.

At those prices, it would be silly not to buy a spare for each location just to have a hot spare on hand whether it ever gets used or not.

6

General Discussion / Re: switching from dnsmasq to unbound, fail

« on: October 02, 2023, 11:57:02 pm »

Agreed. The current setup is the system points to 1.1.1.1/1.0.0.1 and unbound resolver set to use system servers.
I wanted to get the hardware up and running quickly and move on to a few downstream projects. But planning to return to it soon and finish 'flushing things out' - DNS over TLS is the first item on my list. Also captive portal at both locations, ids...

7

General Discussion / Re: switching from dnsmasq to unbound, fail

« on: October 02, 2023, 08:50:12 pm »

yeah, here's the odd thing. Yes, turning on query forwarding on the router in question immediately fixed the issue. I didn't notice I hadn't checked it because....

However, I was copying the configuration on this router from another router I have running at a different house with mostly the same setup. It was using unbound and did NOT have the query forwarding checked. But it definitely is/was working. I'm curious how the original router was working without query forwarding set.

All the rest of the config between the two routers is pretty much the same. I will stare at this a bit and see if I can't figure out how the original router is working without that set. If not, I'll come back and ask for guidance.

Thanks folks!

8

Hardware and Performance / SOLVED Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: October 02, 2023, 08:43:00 pm »

Code: [Select]

C:\Users\Admin\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3 -c 172.30.30.1 -p 51310
Connecting to host 172.30.30.1, port 51310
[  4] local 172.30.30.40 port 61282 connected to 172.30.30.1 port 51310
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   951 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   949 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
[  4]   6.00-7.00   sec   113 MBytes   949 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.11 GBytes   949 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.11 GBytes   949 Mbits/sec                  receiver

iperf Done.
These speeds are 4x what I was getting before, and right at what I'd expect for 1gb copper. YAY! In addition, speakeasy.net speed test now shows 950mbps instead of 250mbps i was getting in previous iperf3 posts in this thread.

The change made: switched from two dual port pcie-x cards plugged into a single pci (non Express) slot via a riser card, to a single i340 4 port card plugged into the pcieX16 slot normally reserved for addon video. Easy

I can't begin to express how much I appreciate the input from everyone steering me towards the solution. THANK YOU!

PS - totally understand the comments about best iperf3 testbed. In this case, swapping routers made it slow, so pretty easy to see after swapping cards in new router that problem is solved. If it had NOT solved the problem, I would have headed down the path of setting up a testbed with router inbetween sender/receiver for iperf.

9

General Discussion / switching from dnsmasq to unbound, fail

« on: September 26, 2023, 08:09:30 pm »

On an opnsense box at one location, I was using dnsmasq successfully for quite some time but decided to switch to unbound. To do the switch, I literally just took the host/domain overrides from dnsmasq and put them in unbound, disabled dnsmasq - enabled unbound - and it has been working for weeks no issues.

On an opnsense box at another location, I set it up with dnsmasq and it was definitely working for clients. Then 10 minutes later I decided "nah, I meant to use unbound". I had no host/domain overrides on that one, so just disabled dnsmasq and enabled unbound. Within a few seconds, no lan client was able to perform dns queries to that opnsense box.

I have compared all the settings between the two, and can't find anything obviously different that would make sense as impacting this. I know I have not provided great details on my setup, but I was wondering if anyone could suggest things I should check. I'd assume its not a firewall rule, as requests to dnsmasq get through.... and if I turn off unbound and turn dnsmasq back on - all clients can immediately get resolutions. Bizarre!

Any thoughts?

10

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 26, 2023, 07:39:38 pm »

Just an update.... arrived from amazon is a pciex16 riser cable and from ebay an intel i340-t4. They arrived while we are out of town though, should be back there this coming saturday.

Soon as I get back I'll see about metalwork (if any) required to get the card to fit properly and run the cable to the pciex16 port. I'll re-run iperf3 and we'll see if the performance goes as expected for 1gb. Will advise & thx!

Tangentially - for the farm house I got a supermicro SYS-5018A-FTN4 which contains an A1SRI-2758F (atom 2758). I immediately ran an iperf3 from a local windows machine to that servers LAN port and got 300mbps, far below what I'd expect out of a 1gbps interface. So I've got two different machines at two different locations with (what I suspect is) low speed on the ethernet ports. This makes me wonder if I'm running the tests wrong or interpreting the results wrong.... on both systems. The wan connection there is only 25mbps (rural internet service), so it's not a problem but still would like to solve it. One at a time

11

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 21, 2023, 06:06:41 am »

I will, thx. I looked at specs on the i340, and if I'm reading it right that card supports pcie 2.0 x4.

In addition to the pcie 3.0/2.0 x16 slot we've been discussing, the board does have a pcie 2.0 x4 slot at the far left, so I might go that route insted of using the x16 slot.

I think the bracket arrangement had no support at the bottom of the bracket, all cards were in line and the bottoms in a continuous slot. In other words, they relied on the card slot on the bracket for support. I could be wrong but will definitely look tomorrow. If this gets me close to 1gb/port, I'd be very happy. Thanks again for the advice.

12

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 21, 2023, 05:38:43 am »

Agreed. Tomorrow I'm going to take a second hard look at the metalwork and see if this could be done.

Plugging it into the pciex16 would be easy with a ribbon extension but 1) The mainboard manual seems to say 'video card only' but I'm not sure thats what they really mean, and 2) one end of the card bracket would be held captive by the screw, but I'm not sure the bottom would be. I'd certainly prefer a i340 to the pro1000.

I just didn't want to go down this path if it was something else causing the issue. I do sincerely appreciate the input.

13

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 21, 2023, 05:12:49 am »

I suspect I could do that, I thought there was something in the mainboard manual that said that particular slot could *only* be used for video cards. I may be wrong about that. But....

As I said several times above including the very previous post, the only case I had available does not allow that. Hence the reason I said I'd have to get a new case for sure and possibly a mainboard.

As a side note, its interesting to find and understand the problem rather than just sweep it under the rug.

14

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 21, 2023, 12:20:48 am »

If it's not the pci slot and it's not the pcix cards, must be the riser lash-up. Changing that is a mechanical problem. I'll need a different case and likely a different mainboard.

I need 3 ports, so I'm rounding to 4. I was hoping to use off the shelf stuff but that may not be possible.

15

Hardware and Performance / Re: Suggestions for troubleshooting slow nat performance (throughput)?

« on: September 20, 2023, 08:38:14 pm »

Yes, it's a crazy setup but it's the case, cards, and mainboard I had sitting around already. Didn't have to buy anything except a new cpu fan.

I did not try with only one NIC plugged, but can easily do so.

pciconf output is:

Code: [Select]

Enter an option: 8

root@OPNsense:~ # pciconf -lvc
hostb0@pci0:0:0:0:      class=0x060000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0150 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
    cap 09[e0] = vendor (length 12) Intel cap 0 version 1
pcib1@pci0:0:1:0:       class=0x060400 rev=0x09 hdr=0x01 vendor=0x8086 device=0x0151 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[88] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[80] = powerspec 3  supports D0 D3  current D0
    cap 05[90] = MSI supports 1 message
    cap 10[a0] = PCI-Express 2 root port max data 256(256)
                 max read 128
                 link x0(x16) speed 0.0(8.0) ASPM disabled(L0s/L1)
                 slot 1 power limit 75000 mW
    ecap 0002[100] = VC 1 max VC0
    ecap 0005[140] = Root Complex Link Declaration 1
    ecap 0019[d94] = PCIe Sec 1 lane errors 0
vgapci0@pci0:0:2:0:     class=0x030000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0162 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'IvyBridge GT2 [HD Graphics 4000]'
    class      = display
    subclass   = VGA
    cap 05[90] = MSI supports 1 message
    cap 01[d0] = powerspec 2  supports D0 D3  current D0
    cap 13[a4] = PCI Advanced Features: FLR TP
none0@pci0:0:22:0:      class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c3a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family MEI Controller'
    class      = simple comms
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[8c] = MSI supports 1 message, 64 bit
ehci0@pci0:0:26:0:      class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
hdac0@pci0:0:27:0:      class=0x040300 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c20 subvendor=0x1043 subdevice=0x841b
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR NS
                 max read 128
    ecap 0002[100] = VC 1 max VC1
    ecap 0005[130] = Root Complex Link Declaration 1
pcib2@pci0:0:28:0:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c10 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x0(x4) speed 0.0(5.0) ASPM disabled(L0s/L1)
                 slot 0 power limit 250 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib3@pci0:0:28:4:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c18 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 5'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 4 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib4@pci0:0:28:5:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 6'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
                 slot 5 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib5@pci0:0:28:6:      class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1c subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 7'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 6 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
pcib6@pci0:0:28:7:      class=0x060401 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(128)
                 max read 128
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 slot 7 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge subvendor=0x1043 subdevice=0x844d
    cap 01[a0] = powerspec 2  supports D0 D3  current D0
ehci1@pci0:0:29:0:      class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
isab0@pci0:0:31:0:      class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c4a subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = 'H67 Express Chipset LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
    cap 09[e0] = vendor (length 12) Intel cap 1 version 0
                 features: AMT, 4 PCI-e x1 slots
atapci1@pci0:0:31:2:    class=0x01018f rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c00 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 0-3)'
    class      = mass storage
    subclass   = ATA
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 13[b0] = PCI Advanced Features: FLR TP
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c22 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
atapci2@pci0:0:31:5:    class=0x010185 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c08 subvendor=0x1043 subdevice=0x844d
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family Desktop SATA Controller (IDE mode, ports 4-5)'
    class      = mass storage
    subclass   = ATA
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 13[b0] = PCI Advanced Features: FLR TP
atapci0@pci0:3:0:0:     class=0x010185 rev=0x00 hdr=0x00 vendor=0x1106 device=0x0415 subvendor=0x1043 subdevice=0x838f
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT6415 PATA IDE Host Controller'
    class      = mass storage
    subclass   = ATA
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[70] = MSI supports 1 message, 64 bit, vector masks
    cap 10[90] = PCI-Express 1 legacy endpoint max data 128(128)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[130] = Serial 1 004063ffff634000
xhci0@pci0:4:0:0:       class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x1042 subvendor=0x1043 subdevice=0x8488
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1042 SuperSpeed USB Host Controller'
    class      = serial bus
    subclass   = USB
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM disabled(L0s/L1)
    ecap 0002[100] = VC 1 max VC0
re0@pci0:5:0:0: class=0x020000 rev=0x06 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8432
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(256)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM disabled
    cap 11[b0] = MSI-X supports 4 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x800]
    cap 03[d0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0002[140] = VC 1 max VC0
    ecap 0003[160] = Serial 1 14000000684ce000
pcib7@pci0:6:0:0:       class=0x060401 rev=0x01 hdr=0x01 vendor=0x1b21 device=0x1080 subvendor=0x1043 subdevice=0x8489
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1083/1085 PCIe to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 0d[c0] = PCI Bridge subvendor=0x1043 subdevice=0x8489
em0@pci0:7:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em1@pci0:7:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em2@pci0:7:1:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
em3@pci0:7:1:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1010 subvendor=0x8086 subdevice=0x1012
    vendor     = 'Intel Corporation'
    device     = '82546EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 05[f0] = MSI supports 1 message, 64 bit
root@OPNsense:~ #
lspci isn't on stock freebsd, and I don't see it in the plugins or packages. I'd prefer to keep things "per ui".

The paddleboard - you can see it in the previous picture. In PCI1 (left one of the two pci slots) is the riser card, with the two cards plugged into the left. At the right side rear/bottom you can see a few (around six or eight) multicolored wires coming off the riser card. These go to a black single row edge connector, that you can see going to PCI2 (the right slot of the two pci slots). If you look at the right slot closely you can see there is a paddleboard plugged into the slot, it brings those six or eight signals from pci2 to the riser. I dont know if it's just getting additional power from the other slot, or if it provides more than that (addressing, etc). The paddleboard is long and thin, pushed into the entire pci2, with only a fraction of an inch sticking above so the pin connector can attach.