Uknown Issue with OPNSense and ATT Fiber

[secdoc]

I am looking for some feedback, possible ideas around how to figure out what may be going on with my throughput between the OPNSense FW and the internet. My current service provider is ATT Fiber Internet, which means I have to have the BGW320 ONT as the gateway and I have IP Passthrough enabled. I subscribed rate is 5Gbps synchronous up until about the beginning of November I was averaging 4.5-5Gbps with no real issues. After then I started to get sweeping performance issues and as such I have speed and Iperf running every 5sec to get metrics and the below shows the overall stats:

Download
1263.2 Mbit/s (current)
3033.3 Mbit/s (average)
4710.9 Mbit/s (maximum)
11.1 Mbit/s (minimum)

Upload
1613.2 Mbit/s (current)
3267.0 Mbit/s (average)
4707.9 Mbit/s (maximum)
7.5 Mbit/s (minimum)

This is the iperf3 now through ATT:

secdoc@maul:~$ iperf3 -c *.*.*.*
Connecting to host *.*.*.*, port 5201
[  5] local 192.168.2.101 port 36846 connected to *.*.*.* port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   218 MBytes  1.83 Gbits/sec   43   2.31 MBytes       
[  5]   1.00-2.00   sec   262 MBytes  2.20 Gbits/sec   83   2.50 MBytes       
[  5]   2.00-3.00   sec   248 MBytes  2.08 Gbits/sec    0   2.58 MBytes       
[  5]   3.00-4.00   sec   215 MBytes  1.80 Gbits/sec    0   2.64 MBytes       
[  5]   4.00-5.00   sec   220 MBytes  1.85 Gbits/sec    0   2.70 MBytes       
[  5]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec  1331   1.39 MBytes       
[  5]   6.00-7.00   sec   351 MBytes  2.95 Gbits/sec    0   1.57 MBytes       
[  5]   7.00-8.00   sec   322 MBytes  2.71 Gbits/sec    7   1.71 MBytes       
[  5]   8.00-9.00   sec   341 MBytes  2.86 Gbits/sec    0   1.84 MBytes       
[  5]   9.00-10.00  sec   330 MBytes  2.77 Gbits/sec    0   1.97 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.59 GBytes  2.23 Gbits/sec  1464             sender
[  5]   0.00-10.04  sec  2.59 GBytes  2.22 Gbits/sec                  receiver

So I have the following setup for OPNSense:

Versions    OPNsense 23.7.10_1-amd64
CPU type    Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz (4 cores, 8 threads)
Memory - 32594 MB
2 - 10Gbase-SR <full-duplex,rxpause,txpause>
4 - 2.5Gpbs Intel NICs

I can run an iperf internally, I get 9.0-9.8Gbps on my network, here is an example:

secdoc@maul:~$ iperf3 -c 192.168.2.10
Connecting to host 192.168.2.10, port 5201
[  5] local 192.168.2.101 port 54084 connected to 192.168.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.36 Gbits/sec    0   1.91 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec   18   1.91 MBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.33 Gbits/sec    5   1.91 MBytes       
[  5]   3.00-4.00   sec  1.06 GBytes  9.10 Gbits/sec    0   1.91 MBytes       
[  5]   4.00-5.00   sec  1.03 GBytes  8.85 Gbits/sec    0   1.91 MBytes       
[  5]   5.00-6.00   sec  1.08 GBytes  9.31 Gbits/sec    7   1.91 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.36 Gbits/sec    0   1.91 MBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.37 Gbits/sec    0   1.92 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.34 Gbits/sec    0   1.93 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.34 Gbits/sec    8   1.94 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.28 Gbits/sec   38             sender
[  5]   0.00-10.05  sec  10.8 GBytes  9.23 Gbits/sec                  receiver

Now I am consistently 1-2Gbps. If I run a speedtest from the BGW320, It is roughly 4.8-5Gbps.

So my question would be is there potentially something that possibly changed with the last set of patches or do you think, which is where I am leaning, they are doing some sort of policy shaping of my traffic and now ATT is messing with me because I am trying to maintain consistent performance.

Any thoughts or possible troubleshooting thoughts would be greatly appreciated.

[secdoc]

*****UPDATE*****

As a set of troubleshooting steps I started disabling various plugins. I did have Zenarmor installed and disabled the engine. Once I disabled the engine, my throughput was restored to previous levels. I opened a ticket with Zenarmor and performed more testing including placing the engine in bypass mode. The throughput dropped to between 2.9-3.9Gbps in bypass. According to Salih at Zenarmor, another customer is also reporting the issue so they will be investigating further. That said, they also pointed to the following:

https://www.zenarmor.com/docs/introduction/hardware-requirements

and according the chart, they would require a min of i9 to have anything greater than the 2gbps throughput. I noted to them that the throughput was not an issue before the latest updates.

Here is a graph and breakdown based on speedtest that are set to run every 5 minutes: