I am trying to enable a 10GbE card based on QLogic 41000 series.
I am using OPNsense 22.7.4 (amd64/OpenSSL).
To do this, I have to enable a driver:
kldload if_qlnxe
The card seems to be detected correctly:
ql0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 34:80:0d:04:fc:2c
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ql1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 34:80:0d:04:fc:2d
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ql2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 34:80:0d:04:fc:2e
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ql3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 34:80:0d:04:fc:2f
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
dmesg, however, shows something is not going great
intsmb0: <AMD FCH SMBus Controller> port 0xff00-0xff0f at device 20.0 on pci0
intsmb0: Could not allocate I/O space
device_attach: intsmb0 attach returned 6
ql0: <Qlogic 10GbE/25GbE/40GbE PCI CNA (AH) Adapter-Ethernet Function v2.0.112> mem 0xea860000-0xea87ffff,0xea000000-0xea7fffff,0xea8b0000-0xea8bffff irq 50 at device 0.0 on pci11
ql0: qlnx_set_personality: ETH_IWARP
ql0: setting parameters required by iWARP dev
ql0: Ethernet address: 34:80:0d:04:fc:2c
ql1: <Qlogic 10GbE/25GbE/40GbE PCI CNA (AH) Adapter-Ethernet Function v2.0.112> mem 0xea840000-0xea85ffff,0xe9800000-0xe9ffffff,0xea8a0000-0xea8affff irq 50 at device 0.1 on pci11
ql1: qlnx_set_personality: ETH_IWARP
ql1: setting parameters required by iWARP dev
ql1: Ethernet address: 34:80:0d:04:fc:2d
ql2: <Qlogic 10GbE/25GbE/40GbE PCI CNA (AH) Adapter-Ethernet Function v2.0.112> mem 0xea820000-0xea83ffff,0xe9000000-0xe97fffff,0xea890000-0xea89ffff irq 50 at device 0.2 on pci11
ql2: qlnx_set_personality: ETH_IWARP
ql2: setting parameters required by iWARP dev
ql2: Ethernet address: 34:80:0d:04:fc:2e
ql3: <Qlogic 10GbE/25GbE/40GbE PCI CNA (AH) Adapter-Ethernet Function v2.0.112> mem 0xea800000-0xea81ffff,0xe8800000-0xe8ffffff,0xea880000-0xea88ffff irq 50 at device 0.3 on pci11
ql3: qlnx_set_personality: ETH_IWARP
ql3: setting parameters required by iWARP dev
ql3: Ethernet address: 34:80:0d:04:fc:2f
But the main problem is that I I get a LOT of ifconfig instances consuming all the CPU.
And the admin website hangs and the machine needs to be rebooted.
Of course, if I try to enable this at bootime in loader.conf, the server does not complete booting.
last pid: 33728; load averages: 8.18, 5.96, 3.02 up 0+00:16:57 16:25:40
86 processes: 9 running, 77 sleeping
CPU: 0.4% user, 0.0% nice, 50.2% system, 0.0% interrupt, 49.4% idle
Mem: 438M Active, 73M Inact, 1086M Wired, 40K Buf, 29G Free
ARC: 109M Total, 25M MFU, 77M MRU, 244K Anon, 811K Header, 5895K Other
56M Compressed, 139M Uncompressed, 2.49:1 Ratio
Swap: 8192M Total, 8192M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
65963 root 1 86 0 13M 3188K CPU12 12 5:11 53.97% ifconfig
99294 root 1 52 0 13M 3188K ecore_ 2 0:08 52.68% ifconfig
11173 root 1 52 0 13M 3188K ecore_ 11 4:09 49.72% ifconfig
5551 root 1 52 0 13M 3188K ecore_ 5 6:37 48.16% ifconfig
83767 root 1 52 0 13M 3188K ecore_ 11 2:16 47.35% ifconfig
86029 root 1 52 0 13M 3188K ecore_ 13 2:11 45.51% ifconfig
46748 root 1 52 0 13M 3188K ecore_ 8 3:09 44.06% ifconfig
55316 root 1 86 0 13M 3188K CPU14 14 0:56 43.60% ifconfig
6093 root 1 52 0 13M 3188K ecore_ 5 6:36 42.95% ifconfig
54580 root 1 52 0 13M 3188K ecore_ 15 0:56 41.80% ifconfig
84789 root 1 52 0 13M 3188K ecore_ 11 2:11 37.66% ifconfig
92072 root 1 52 0 13M 3188K ecore_ 5 0:23 37.36% ifconfig
80609 root 1 52 0 13M 3188K ecore_ 1 4:33 36.12% ifconfig
29653 root 1 50 0 13M 3188K CPU9 9 1:08 35.92% ifconfig
10468 root 1 85 0 13M 3188K CPU10 10 4:11 35.58% ifconfig
Any tips on how to debug this further?
Are there any settings in tunables that I could use?
I am actually unsure the reason behind this.. but if I leave the server running for a while...
I get many of these messages:
"PTT acquire timeout - failed to allocate PTT\n"
This seems to be happening here (in the drivers code):
https://www.leidinger.net/FreeBSD/dox/dev_qlnx/html/d7/d95/ecore__hw_8c_source.html
I suspect this is related to the amount of `ifconfig` instances that are running at the same time:
root@OPNsense:~ # ps aux | grep ifconfig
root 33728 25.4 0.0 13616 3188 - D 16:25 22:50.29 /sbin/ifconfig -m -v
root 83767 22.8 0.0 13616 3188 - D 16:21 25:08.63 /sbin/ifconfig -m -v
root 77400 21.6 0.0 13616 3188 - R 16:29 21:18.76 /sbin/ifconfig -m -v
root 32681 21.1 0.0 13616 3188 - D 16:33 20:27.42 /sbin/ifconfig -m -v
root 6553 21.0 0.0 13616 3188 - D 16:35 19:57.68 /sbin/ifconfig -m -v
root 32443 20.6 0.0 13616 3188 - D 16:33 20:26.96 /sbin/ifconfig -m -v
root 55316 19.8 0.0 13616 3188 - D 16:23 23:42.50 /sbin/ifconfig -m -v
root 86029 19.8 0.0 13616 3188 - D 16:21 24:57.39 /sbin/ifconfig -m -v
root 12236 19.7 0.0 13616 3188 - R 16:35 20:04.06 /sbin/ifconfig -m -v
root 66009 19.5 0.0 13616 3188 - R 16:26 22:21.64 /sbin/ifconfig -m -v
root 94644 19.5 0.0 13616 3188 - D 16:32 20:43.15 /sbin/ifconfig -m -v
root 2511 19.1 0.0 13616 3188 - D 16:33 20:49.90 /sbin/ifconfig -m -v
root 52288 19.1 0.0 13616 3188 - D 16:21 25:06.48 /sbin/ifconfig -m -v
root 84789 18.8 0.0 13616 3188 - R 16:21 24:53.39 /sbin/ifconfig -m -v
root 53510 18.4 0.0 13616 3188 - D 16:31 20:54.66 /sbin/ifconfig -m -v
root 52252 18.2 0.0 13616 3188 - D 16:31 20:47.20 /sbin/ifconfig -m -v
root 46748 18.0 0.0 13616 3188 - D 16:20 25:56.99 /sbin/ifconfig -m -v
root 5551 17.6 0.0 13616 3188 - D 16:17 29:30.01 /sbin/ifconfig -m -v
root 62835 17.4 0.0 13616 3188 - D 16:34 20:21.40 /sbin/ifconfig -m -v
root 19 17.2 0.0 13616 3188 - D 16:27 22:00.12 /sbin/ifconfig -m -v
root 11924 17.1 0.0 13616 3188 - D 16:35 20:12.21 /sbin/ifconfig -m -v
root 76493 16.9 0.0 13616 3188 - R 16:29 21:19.91 /sbin/ifconfig -m -v
root 56359 16.8 0.0 13616 3188 - D 16:23 23:58.49 /sbin/ifconfig -m -v
root 30562 16.5 0.0 13616 3188 - D 16:33 20:38.63 /sbin/ifconfig -m -v
root 11173 16.4 0.0 13616 3188 - D 16:19 26:56.16 /sbin/ifconfig -m -v
root 23544 16.4 0.0 13616 3188 - D 16:22 23:58.20 /sbin/ifconfig -m -v
root 32107 16.4 0.0 13616 3188 - R 16:25 22:43.56 /sbin/ifconfig -m -v
root 92072 16.4 0.0 13616 3188 - D 16:24 23:14.02 /sbin/ifconfig -m -v
root 6093 16.3 0.0 13616 3188 - D 16:17 29:33.10 /sbin/ifconfig -m -v
root 10468 16.3 0.0 13616 3188 - D 16:19 27:14.21 /sbin/ifconfig -m -v
root 78268 16.2 0.0 13616 3188 - D 16:35 19:58.87 /sbin/ifconfig -m -v
root 36836 16.0 0.0 13616 3188 - D 16:28 21:47.77 /sbin/ifconfig -m -v
root 30451 15.9 0.0 13616 3188 - R 16:25 22:50.69 /sbin/ifconfig -m -v
root 65963 15.9 0.0 13616 3188 - D 16:18 27:52.37 /sbin/ifconfig -m -v
root 99176 15.8 0.0 13616 3188 - D 16:27 22:09.12 /sbin/ifconfig -m -v
root 99294 15.6 0.0 13616 3188 - D 16:25 22:55.88 /sbin/ifconfig -m -v
root 78969 15.5 0.0 13616 3188 - D 16:29 21:19.36 /sbin/ifconfig -m -v
root 70286 15.4 0.0 13616 3188 - D 16:27 22:14.18 /sbin/ifconfig -m -v
root 15948 15.3 0.0 13616 3188 - D 16:30 21:31.64 /sbin/ifconfig -m -v
root 29653 15.3 0.0 13616 3188 - D 16:23 23:41.59 /sbin/ifconfig -m -v
root 80609 15.1 0.0 13616 3188 - D 16:19 27:11.15 /sbin/ifconfig -m -v
root 10668 14.8 0.0 13616 3188 - D 16:19 26:48.73 /sbin/ifconfig -m -v
root 46359 14.7 0.0 13616 3188 - D 16:29 21:21.40 /sbin/ifconfig -m -v
root 54580 14.6 0.0 13616 3188 - R 16:23 23:43.37 /sbin/ifconfig -m -v
root 20566 14.4 0.0 13616 3188 - D 16:31 21:00.97 /sbin/ifconfig -m -v
root 51053 14.1 0.0 13616 3188 - D 16:31 20:54.91 /sbin/ifconfig -m -v
root 99674 12.3 0.0 13616 3188 - D 16:27 21:54.40 /sbin/ifconfig -m -v
root 60977 0.0 0.0 12748 2368 1 S+ 18:36 0:00.00 grep ifconfig
root@OPNsense:~ # ps aux | grep ifconfig | wc -l
48
so all these instances seem to be coming from `gateway_status.php`
root 1 0.0 0.0 11408 1020 - ILs 16:08 0:00.01 - /sbin/init
root 386 0.0 0.1 36252 22612 - Is 16:08 0:01.26 |-- /usr/local/bin/python3 /usr/local/opnsense/service/configd.py (python3.9)
root 416 0.0 0.1 229760 38564 - I 16:08 0:01.18 | `-- /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.9)
root 1888 0.0 0.1 50908 29460 - I 16:33 0:00.02 | |-- /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
root 2511 21.7 0.0 13616 3188 - D 16:33 21:39.82 | | `-- /sbin/ifconfig -m -v
Basically, this command hangs
/sbin/ifconfig -m -v
and the weirdest part is that this works:
/sbin/ifconfig
but verbose mode does not:
/sbin/ifconfig -v
what could be the reason that `ifconfig -v` hangs?
sorry for all the posting, but hopefully sharing my investigation helps someone:
The issue I am having sounds very similar to this, it seems:
https://github.com/opnsense/core/issues/5349