Is there any way to check whether OPNsense is using an add-on QAT accelerator properly? I just installed an Intel 8960 QAT accelerator and turned on the setting to enable QAT. Is there a way to check that "it worked" and the card is supported and being utilized? I know the "other" platform explicitly lists which acceleration is being utilized and which ciphers are supported.
root@OPNsense:~ # kldstat
Id Refs Address Size Name
1 96 0xffffffff80200000 215db18 kernel
2 1 0xffffffff8235e000 e4d0 if_bridge.ko
3 2 0xffffffff8236d000 7870 bridgestp.ko
4 1 0xffffffff82375000 ba48 if_gre.ko
5 1 0xffffffff82381000 11920 ipmi.ko
6 3 0xffffffff82393000 3c68 smbus.ko
7 1 0xffffffff82397000 e318 pfsync.ko
8 3 0xffffffff823a6000 741a8 pf.ko
9 1 0xffffffff8241b000 3b18 pflog.ko
10 1 0xffffffff8241f000 4b58 if_enc.ko
11 1 0xffffffff82424000 f460 carp.ko
12 1 0xffffffff82434000 181d0 if_lagg.ko
13 2 0xffffffff8244d000 3538 if_infiniband.ko
14 1 0xffffffff82a10000 2110 pchtherm.ko
15 1 0xffffffff82a13000 5e7c ig4.ko
16 1 0xffffffff82a19000 433c iicbus.ko
17 1 0xffffffff82a1e000 3250 ichsmb.ko
18 1 0xffffffff82a22000 3378 acpi_wmi.ko
19 1 0xffffffff82a26000 2340 uhid.ko
20 1 0xffffffff82a29000 4350 ums.ko
21 1 0xffffffff82a2e000 3380 usbhid.ko
22 1 0xffffffff82a32000 31f8 hidbus.ko
23 1 0xffffffff82a36000 16308 qat.ko
24 1 0xffffffff82a4d000 e2330 qat_c62xfw.ko
25 1 0xffffffff82b30000 20f0 coretemp.ko
26 1 0xffffffff82b33000 4700 nullfs.ko
27 1 0xffffffff82b38000 f418 ipsec.ko
28 1 0xffffffff82b48000 34568 if_wg.ko
29 1 0xffffffff82b7d000 39c0 ng_socket.ko
30 4 0xffffffff82b81000 aac8 netgraph.ko
31 1 0xffffffff82b8c000 31c8 ng_ether.ko
32 1 0xffffffff82b90000 53f8 ng_netflow.ko
33 1 0xffffffff82b96000 31e8 ng_ksocket.ko
and
qat0: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91540000-0x9157ffff,0x91500000-0x9153ffff at device 0.0 on pci3
qat_ae_cluster_intr
qat1: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91440000-0x9147ffff,0x91400000-0x9143ffff at device 0.0 on pci4
qat2: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91340000-0x9137ffff,0x91300000-0x9133ffff at device 0.0 on pci5
qat_ae_cluster_intr
To answer my own question, and just to have the answer for anybody else in the future, the Intel 8960 does indeed seem to be functioning. Running an iPerf3 over an IPSec tunnel and watching vmstat -i | grep qat definitely increments the counters.
Two command outputs below that lead me to believe everything is working as it should.
root@OPNsense:~ # vmstat -i | grep qat
irq144: qat0 296664 2
irq145: qat0 1547616 13
irq146: qat0 142510 1
irq147: qat0 195715 2
irq148: qat0 232107 2
irq149: qat0 255745 2
irq150: qat0 63838 1
irq151: qat0 1632242 13
irq160: qat0 1 0
irq161: qat1 104049 1
irq162: qat1 327807 3
irq163: qat1 57244 0
irq164: qat1 594580 5
irq165: qat1 210576 2
irq166: qat1 27 0
irq167: qat1 189053 2
irq168: qat1 576246 5
irq179: qat2 1430647 12
irq194: qat2 1 0
root@OPNsense:~ # sysctl -a | grep qat
qat0: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91540000-0x9157ffff,0x91500000-0x9153ffff at device 0.0 on pci3
qat_ae_cluster_intr
qat1: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91440000-0x9147ffff,0x91400000-0x9143ffff at device 0.0 on pci4
qat2: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91340000-0x9137ffff,0x91300000-0x9133ffff at device 0.0 on pci5
qat_ae_cluster_intr
irq144: qat0:295 @cpu0(domain0): 296664
irq145: qat0:297 @cpu1(domain0): 1547627
irq146: qat0:299 @cpu2(domain0): 142510
irq147: qat0:301 @cpu3(domain0): 195715
irq148: qat0:303 @cpu4(domain0): 232107
irq149: qat0:305 @cpu5(domain0): 338682
irq150: qat0:307 @cpu6(domain0): 105541
irq151: qat0:309 @cpu7(domain0): 1632242
irq152: qat0:311 @cpu0(domain0): 0
irq153: qat0:313 @cpu1(domain0): 0
irq154: qat0:315 @cpu2(domain0): 0
irq155: qat0:317 @cpu3(domain0): 0
irq156: qat0:319 @cpu4(domain0): 0
irq157: qat0:321 @cpu5(domain0): 0
irq158: qat0:323 @cpu6(domain0): 0
irq159: qat0:325 @cpu7(domain0): 0
irq160: qat0:327 @cpu0(domain0): 1
irq161: qat1:329 @cpu0(domain0): 104049
irq162: qat1:331 @cpu1(domain0): 327807
irq163: qat1:333 @cpu2(domain0): 57244
irq164: qat1:335 @cpu3(domain0): 594580
irq165: qat1:337 @cpu4(domain0): 210576
irq166: qat1:339 @cpu5(domain0): 27
irq167: qat1:341 @cpu6(domain0): 189053
irq168: qat1:343 @cpu7(domain0): 576246
irq169: qat1:345 @cpu0(domain0): 0
irq170: qat1:347 @cpu1(domain0): 0
irq171: qat1:349 @cpu2(domain0): 0
irq172: qat1:351 @cpu3(domain0): 0
irq173: qat1:353 @cpu4(domain0): 0
irq174: qat1:355 @cpu5(domain0): 0
irq175: qat1:357 @cpu6(domain0): 0
irq176: qat1:359 @cpu7(domain0): 0
irq177: qat1:361 @cpu0(domain0): 0
irq178: qat2:363 @cpu0(domain0): 0
irq179: qat2:365 @cpu1(domain0): 1488722
irq180: qat2:367 @cpu2(domain0): 0
irq181: qat2:369 @cpu3(domain0): 0
irq182: qat2:371 @cpu4(domain0): 0
irq183: qat2:373 @cpu5(domain0): 0
irq184: qat2:375 @cpu6(domain0): 0
irq185: qat2:377 @cpu7(domain0): 0
irq186: qat2:379 @cpu0(domain0): 0
irq187: qat2:381 @cpu1(domain0): 0
irq188: qat2:383 @cpu2(domain0): 0
irq189: qat2:385 @cpu3(domain0): 0
irq190: qat2:387 @cpu4(domain0): 0
irq191: qat2:389 @cpu5(domain0): 0
irq192: qat2:391 @cpu6(domain0): 0
irq193: qat2:393 @cpu7(domain0): 0
irq194: qat2:395 @cpu0(domain0): 1
dev.qat.2.stats.sym_alloc_failures: 0
dev.qat.2.stats.ring_full: 0
dev.qat.2.stats.gcm_aad_updates: 0
dev.qat.2.stats.gcm_aad_restarts: 0
dev.qat.2.%parent: pci5
dev.qat.2.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.2.%location: slot=0 function=0 dbsf=pci0:5:0:0
dev.qat.2.%driver: qat
dev.qat.2.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.1.stats.sym_alloc_failures: 0
dev.qat.1.stats.ring_full: 0
dev.qat.1.stats.gcm_aad_updates: 0
dev.qat.1.stats.gcm_aad_restarts: 0
dev.qat.1.%parent: pci4
dev.qat.1.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.1.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.qat.1.%driver: qat
dev.qat.1.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.0.stats.sym_alloc_failures: 0
dev.qat.0.stats.ring_full: 0
dev.qat.0.stats.gcm_aad_updates: 0
dev.qat.0.stats.gcm_aad_restarts: 0
dev.qat.0.%parent: pci3
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.%parent:
Thanks for posting this we are about to install 2 x Supermicro servers and its good to know they get recognised with c620 chips
I tested the QAT speed of my ATOM C3758 which has onboard QAT, using the openssl speed command from this article.
https://stackoverflow.com/questions/64862544/how-to-check-compare-openssl-speed
I also saw a big decrease in CPU util on my openVPN connections after switching to the Q3758 CPU.
without using QAT
Doing aes-256 cbc for 3s on 16 size blocks: 11224359 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 64 size blocks: 2947524 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 744654 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 1024 size blocks: 186214 aes-256 cbc's in 2.98s
Doing aes-256 cbc for 3s on 8192 size blocks: 23475 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 16384 size blocks: 12084 aes-256 cbc's in 3.09s
Results with QAT
Doing aes-256-cbc for 3s on 16 size blocks: 43725132 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 15233718 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 256 size blocks: 4530328 aes-256-cbc's in 3.02s
Doing aes-256-cbc for 3s on 1024 size blocks: 1214440 aes-256-cbc's in 3.06s
Doing aes-256-cbc for 3s on 8192 size blocks: 150648 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 16384 size blocks: 75565 aes-256-cbc's in 3.01s
Hi Guys where is the switch to enable qat in opnsense
Thanks
System: Settings: Miscellaneous: Hardware acceleration
Thanks dude ;D
Quote from: zz00mm on March 10, 2023, 04:26:28 PMI tested the QAT speed of my ATOM C3758 which has onboard QAT, using the openssl speed command from this article.
https://stackoverflow.com/questions/64862544/how-to-check-compare-openssl-speed
@zz00mm , do you happen to know the prerequisite steps you need to install to get the test to run? I just did an OpnSense install and when I log in as root and go to the shell I get library errors when running the test.
I don't even have the basic diagnostic commands like cpuid available. I suppose OpnSense wasn't built for ease of use using the shell.
root@OPNsense:~ # lspci | grep -i "QuickAssist"
lspci: Command not found.
root@OPNsense:~ # openssl speed -engine qat rsa2048
Invalid engine "qat"
00208112871B0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/usr/src/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qat.so): Cannot open "/usr/lib/engines-3/qat.so"
00208112871B0000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/usr/src/crypto/openssl/crypto/dso/dso_lib.c:152:
00208112871B0000:error:13000084:engine routines:dynamic_load:dso not found:/usr/src/crypto/openssl/crypto/engine/eng_dyn.c:442:
00208112871B0000:error:13000074:engine routines:ENGINE_by_id:no such engine:/usr/src/crypto/openssl/crypto/engine/eng_list.c:433:id=qat
00208112871B0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/usr/src/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libqat.so): Shared object "libqat.so" not found, required by "openssl"
00208112871B0000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/usr/src/crypto/openssl/crypto/dso/dso_lib.c:152:
00208112871B0000:error:13000084:engine routines:dynamic_load:dso not found:/usr/src/crypto/openssl/crypto/engine/eng_dyn.c:442:
Doing 2048 bits private rsa's for 10s: 3547 2048 bits private RSA's in 10.05s
Doing 2048 bits public rsa's for 10s: 122119 2048 bits public RSA's in 10.00s
version: 3.0.15
root@OPNsense:~ # cpuid -1 | egrep 'VAES|VPCLM|GFNI|AVX512F|AVX512IFMA'
cpuid: Command not found.
root@OPNsense:~ # vmstat -i | grep qat
root@OPNsense:~ # sysctl -a | grep qat
qat0: <Intel c3xxx QuickAssist> mem 0xdfb40000-0xdfb7ffff,0xdfb00000-0xdfb3ffff irq 16 at device 0.0 on pci1
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
irq140: qat0:b0:291 @cpu0(domain0): 0
irq141: qat0:b1:293 @cpu0(domain0): 0
irq142: qat0:b2:295 @cpu0(domain0): 0
irq143: qat0:b3:297 @cpu0(domain0): 0
irq144: qat0:b4:299 @cpu0(domain0): 0
irq145: qat0:b5:301 @cpu0(domain0): 0
irq146: qat0:b6:303 @cpu0(domain0): 0
irq147: qat0:b7:305 @cpu0(domain0): 0
irq148: qat0:b8:307 @cpu0(domain0): 0
irq149: qat0:b9:309 @cpu0(domain0): 0
irq150: qat0:b10:311 @cpu0(domain0): 0
irq151: qat0:b11:313 @cpu0(domain0): 0
irq152: qat0:b12:315 @cpu0(domain0): 0
irq153: qat0:b13:317 @cpu0(domain0): 0
irq154: qat0:b14:319 @cpu0(domain0): 0
irq155: qat0:b15:321 @cpu0(domain0): 0
irq156: qat0:ae:323 @cpu0(domain0): 0
dev.qat_ocf.0.enable: 1
dev.qat_ocf.0.%iommu:
dev.qat_ocf.0.%parent: nexus0
dev.qat_ocf.0.%pnpinfo:
dev.qat_ocf.0.%location:
dev.qat_ocf.0.%driver: qat_ocf
dev.qat_ocf.0.%desc: QAT engine
dev.qat_ocf.%parent:
dev.qat.0.frequency: 685000000
dev.qat.0.cnv_error:
dev.qat.0.fw_counters:
dev.qat.0.mmp_version: 6.0.0
dev.qat.0.hw_version: 17
dev.qat.0.fw_version: 4.18.0
dev.qat.0.heartbeat: 1
dev.qat.0.heartbeat_failed: 0
dev.qat.0.heartbeat_sent: 1
dev.qat.0.dev_cfg: [GENERAL]
dev.qat.0.num_user_processes: 0
dev.qat.0.cfg_mode: ks
dev.qat.0.cfg_services: sym;dc
dev.qat.0.state: up
dev.qat.0.%iommu: rid=0x100
dev.qat.0.%parent: pci1
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x19e2 subvendor=0x8086 subdevice=0x0000 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel c3xxx QuickAssist
dev.qat.%parent:
root@OPNsense:~ # sudo pkg install cpuid
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'cpuid' have been found in the repositories
Quote from: HeneryH on February 24, 2025, 03:38:56 AMI don't even have the basic diagnostic commands like cpuid available. I suppose OpnSense wasn't built for ease of use using the shell.
Yep, a pure desktop os only, that's FreeBSD :o)
Quoteroot@OPNsense:~ # lspci | grep -i "QuickAssist"
lspci: Command not found.
root@OPNsense:~ # openssl speed -engine qat rsa2048
Invalid engine "qat"
00208112871B0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/usr/src/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qat.so): Cannot open "/usr/lib/engines-3/qat.so"
...
root@OPNsense:~ # cpuid -1 | egrep 'VAES|VPCLM|GFNI|AVX512F|AVX512IFMA'
cpuid: Command not found.
...
root@OPNsense:~ # vmstat -i | grep qat
...
root@OPNsense:~ # sysctl -a | grep qat
qat0: <Intel c3xxx QuickAssist> mem 0xdfb40000-0xdfb7ffff,0xdfb00000-0xdfb3ffff irq 16 at device 0.0 on pci1
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
...
FreeBSD aint Linux ;), no lspci or cpuid.
For the CPU features :
fgrep Features /var/run/dmesg.log Instead of lscpi use
pciconf -lHave you enabled QAT in "System / Miscellaneous"? Your call to `vmstat -i | grep qat` has to show it if enabled.
As for the openssl engine call: you call openssl with any engine name, it will just look for an file with that name in the library folder. There is no qat engine. You would test with QAT-supported ciphers as the user in the link did you mentioned. For the list of supported ciphers have a look at
https://www.intel.com/content/www/us/en/support/articles/000093843/technologies/intel-quickassist-technology-intel-qat.html
@patient0, yes, I think I have it enabled. I was just looking for some way to confirm that it is 1) there and accessible at all and 2) OpnSense is using it.