QAT Accelerator

[antsh415]

Is there any way to check whether OPNsense is using an add-on QAT accelerator properly? I just installed an Intel 8960 QAT accelerator and turned on the setting to enable QAT. Is there a way to check that “it worked” and the card is supported and being utilized? I know the “other” platform explicitly lists which acceleration is being utilized and which ciphers are supported.

Code: [Select]

root@OPNsense:~ # kldstat
Id Refs Address                Size Name
 1   96 0xffffffff80200000  215db18 kernel
 2    1 0xffffffff8235e000     e4d0 if_bridge.ko
 3    2 0xffffffff8236d000     7870 bridgestp.ko
 4    1 0xffffffff82375000     ba48 if_gre.ko
 5    1 0xffffffff82381000    11920 ipmi.ko
 6    3 0xffffffff82393000     3c68 smbus.ko
 7    1 0xffffffff82397000     e318 pfsync.ko
 8    3 0xffffffff823a6000    741a8 pf.ko
 9    1 0xffffffff8241b000     3b18 pflog.ko
10    1 0xffffffff8241f000     4b58 if_enc.ko
11    1 0xffffffff82424000     f460 carp.ko
12    1 0xffffffff82434000    181d0 if_lagg.ko
13    2 0xffffffff8244d000     3538 if_infiniband.ko
14    1 0xffffffff82a10000     2110 pchtherm.ko
15    1 0xffffffff82a13000     5e7c ig4.ko
16    1 0xffffffff82a19000     433c iicbus.ko
17    1 0xffffffff82a1e000     3250 ichsmb.ko
18    1 0xffffffff82a22000     3378 acpi_wmi.ko
19    1 0xffffffff82a26000     2340 uhid.ko
20    1 0xffffffff82a29000     4350 ums.ko
21    1 0xffffffff82a2e000     3380 usbhid.ko
22    1 0xffffffff82a32000     31f8 hidbus.ko
23    1 0xffffffff82a36000    16308 qat.ko
24    1 0xffffffff82a4d000    e2330 qat_c62xfw.ko
25    1 0xffffffff82b30000     20f0 coretemp.ko
26    1 0xffffffff82b33000     4700 nullfs.ko
27    1 0xffffffff82b38000     f418 ipsec.ko
28    1 0xffffffff82b48000    34568 if_wg.ko
29    1 0xffffffff82b7d000     39c0 ng_socket.ko
30    4 0xffffffff82b81000     aac8 netgraph.ko
31    1 0xffffffff82b8c000     31c8 ng_ether.ko
32    1 0xffffffff82b90000     53f8 ng_netflow.ko
33    1 0xffffffff82b96000     31e8 ng_ksocket.ko
and

Code: [Select]

qat0: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91540000-0x9157ffff,0x91500000-0x9153ffff at device 0.0 on pci3
qat_ae_cluster_intr
qat1: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91440000-0x9147ffff,0x91400000-0x9143ffff at device 0.0 on pci4
qat2: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91340000-0x9137ffff,0x91300000-0x9133ffff at device 0.0 on pci5
qat_ae_cluster_intr

[antsh415]

To answer my own question, and just to have the answer for anybody else in the future, the Intel 8960 does indeed seem to be functioning. Running an iPerf3 over an IPSec tunnel and watching

Code: [Select]

vmstat -i | grep qat definitely increments the counters.

Two command outputs below that lead me to believe everything is working as it should.

Code: [Select]

root@OPNsense:~ # vmstat -i | grep qat
irq144: qat0                      296664          2
irq145: qat0                     1547616         13
irq146: qat0                      142510          1
irq147: qat0                      195715          2
irq148: qat0                      232107          2
irq149: qat0                      255745          2
irq150: qat0                       63838          1
irq151: qat0                     1632242         13
irq160: qat0                           1          0
irq161: qat1                      104049          1
irq162: qat1                      327807          3
irq163: qat1                       57244          0
irq164: qat1                      594580          5
irq165: qat1                      210576          2
irq166: qat1                          27          0
irq167: qat1                      189053          2
irq168: qat1                      576246          5
irq179: qat2                     1430647         12
irq194: qat2                           1          0

Code: [Select]

root@OPNsense:~ # sysctl -a | grep qat
qat0: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91540000-0x9157ffff,0x91500000-0x9153ffff at device 0.0 on pci3
qat_ae_cluster_intr
qat1: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91440000-0x9147ffff,0x91400000-0x9143ffff at device 0.0 on pci4
qat2: <Intel C620/Xeon D-2100 QuickAssist PF> mem 0x91340000-0x9137ffff,0x91300000-0x9133ffff at device 0.0 on pci5
qat_ae_cluster_intr
irq144: qat0:295 @cpu0(domain0): 296664
irq145: qat0:297 @cpu1(domain0): 1547627
irq146: qat0:299 @cpu2(domain0): 142510
irq147: qat0:301 @cpu3(domain0): 195715
irq148: qat0:303 @cpu4(domain0): 232107
irq149: qat0:305 @cpu5(domain0): 338682
irq150: qat0:307 @cpu6(domain0): 105541
irq151: qat0:309 @cpu7(domain0): 1632242
irq152: qat0:311 @cpu0(domain0): 0
irq153: qat0:313 @cpu1(domain0): 0
irq154: qat0:315 @cpu2(domain0): 0
irq155: qat0:317 @cpu3(domain0): 0
irq156: qat0:319 @cpu4(domain0): 0
irq157: qat0:321 @cpu5(domain0): 0
irq158: qat0:323 @cpu6(domain0): 0
irq159: qat0:325 @cpu7(domain0): 0
irq160: qat0:327 @cpu0(domain0): 1
irq161: qat1:329 @cpu0(domain0): 104049
irq162: qat1:331 @cpu1(domain0): 327807
irq163: qat1:333 @cpu2(domain0): 57244
irq164: qat1:335 @cpu3(domain0): 594580
irq165: qat1:337 @cpu4(domain0): 210576
irq166: qat1:339 @cpu5(domain0): 27
irq167: qat1:341 @cpu6(domain0): 189053
irq168: qat1:343 @cpu7(domain0): 576246
irq169: qat1:345 @cpu0(domain0): 0
irq170: qat1:347 @cpu1(domain0): 0
irq171: qat1:349 @cpu2(domain0): 0
irq172: qat1:351 @cpu3(domain0): 0
irq173: qat1:353 @cpu4(domain0): 0
irq174: qat1:355 @cpu5(domain0): 0
irq175: qat1:357 @cpu6(domain0): 0
irq176: qat1:359 @cpu7(domain0): 0
irq177: qat1:361 @cpu0(domain0): 0
irq178: qat2:363 @cpu0(domain0): 0
irq179: qat2:365 @cpu1(domain0): 1488722
irq180: qat2:367 @cpu2(domain0): 0
irq181: qat2:369 @cpu3(domain0): 0
irq182: qat2:371 @cpu4(domain0): 0
irq183: qat2:373 @cpu5(domain0): 0
irq184: qat2:375 @cpu6(domain0): 0
irq185: qat2:377 @cpu7(domain0): 0
irq186: qat2:379 @cpu0(domain0): 0
irq187: qat2:381 @cpu1(domain0): 0
irq188: qat2:383 @cpu2(domain0): 0
irq189: qat2:385 @cpu3(domain0): 0
irq190: qat2:387 @cpu4(domain0): 0
irq191: qat2:389 @cpu5(domain0): 0
irq192: qat2:391 @cpu6(domain0): 0
irq193: qat2:393 @cpu7(domain0): 0
irq194: qat2:395 @cpu0(domain0): 1
dev.qat.2.stats.sym_alloc_failures: 0
dev.qat.2.stats.ring_full: 0
dev.qat.2.stats.gcm_aad_updates: 0
dev.qat.2.stats.gcm_aad_restarts: 0
dev.qat.2.%parent: pci5
dev.qat.2.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.2.%location: slot=0 function=0 dbsf=pci0:5:0:0
dev.qat.2.%driver: qat
dev.qat.2.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.1.stats.sym_alloc_failures: 0
dev.qat.1.stats.ring_full: 0
dev.qat.1.stats.gcm_aad_updates: 0
dev.qat.1.stats.gcm_aad_restarts: 0
dev.qat.1.%parent: pci4
dev.qat.1.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.1.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.qat.1.%driver: qat
dev.qat.1.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.0.stats.sym_alloc_failures: 0
dev.qat.0.stats.ring_full: 0
dev.qat.0.stats.gcm_aad_updates: 0
dev.qat.0.stats.gcm_aad_restarts: 0
dev.qat.0.%parent: pci3
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0001 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.%parent:

[jamest65]

Thanks for posting this we are about to install 2 x Supermicro servers and its good to know they get recognised with c620 chips

[zz00mm]

I tested the QAT speed of my ATOM C3758 which has onboard QAT, using the openssl speed  command from this article.
https://stackoverflow.com/questions/64862544/how-to-check-compare-openssl-speed

I also saw a big decrease in CPU util on my openVPN connections after switching to the Q3758 CPU.

without using QAT
Doing aes-256 cbc for 3s on 16 size blocks: 11224359 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 64 size blocks: 2947524 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 744654 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 1024 size blocks: 186214 aes-256 cbc's in 2.98s
Doing aes-256 cbc for 3s on 8192 size blocks: 23475 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 16384 size blocks: 12084 aes-256 cbc's in 3.09s

Results with QAT
Doing aes-256-cbc for 3s on 16 size blocks: 43725132 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 15233718 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 256 size blocks: 4530328 aes-256-cbc's in 3.02s
Doing aes-256-cbc for 3s on 1024 size blocks: 1214440 aes-256-cbc's in 3.06s
Doing aes-256-cbc for 3s on 8192 size blocks: 150648 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 16384 size blocks: 75565 aes-256-cbc's in 3.01s

[jamest65]

Hi Guys where is the switch to enable qat in opnsense

Thanks

[franco]

System: Settings: Miscellaneous: Hardware acceleration

[jamest65]

Thanks dude