Messages

I'm averaging around 5Gb now, I tried each tweak below, I'll leave them in place for now, I feel like adding RSS helped reduce my latency according to some other speed tests. I am still only averaging around 10-12ms latency on bufferbloat tests, before I upgraded everything I was a consistent 2-4ms scoring A+, now with 2Gb service and 10Gb nics various tweaks etc I cant seem to get below 10ms of latency. That said though my clients on 1Gb nics are scoring A+, it seems like my client running at 10Gb are having the issue scoring A+, if that makes any sense ?



iperf benchmarks -P8 (8 Threads)

Stock Tunables:
[SUM]   0.00-10.01  sec  5.26 GBytes  4.52 Gbits/sec                  receiver

Added RSS Enabled:
[SUM]   0.00-10.02  sec  5.74 GBytes  4.92 Gbits/sec                  receiver

Added kern.ipc.maxsockbuf 16777216
[SUM]   0.00-10.01  sec  5.56 GBytes  4.77 Gbits/sec                  receiver

Added net.inet.tcp.soreceive_stream 1
[SUM]   0.00-10.01  sec  5.71 GBytes  4.90 Gbits/sec                  receiver

Earlier I started messing around with the hardware offloading settings, which made no difference. However, after reverting back to a full stock config and rebooting my iperf scores are looking better in threaded mode. The numbers below should prove all nics are at 10Gb. However I still feel like the scores are low. The netgate hardware store sells a box running a D-1541 Xeon cpu and its rated for 18Gb, im sure I can do better than the below. I'm gonna have to try some more tweaks, its just hard to determine which are really making an impact.

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   535 MBytes   449 Mbits/sec                  sender
[  5]   0.00-10.01  sec   535 MBytes   449 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   550 MBytes   461 Mbits/sec                  sender
[  7]   0.00-10.01  sec   549 MBytes   461 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   529 MBytes   443 Mbits/sec                  sender
[  9]   0.00-10.01  sec   529 MBytes   443 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   562 MBytes   471 Mbits/sec                  sender
[ 11]   0.00-10.01  sec   562 MBytes   471 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   526 MBytes   441 Mbits/sec                  sender
[ 13]   0.00-10.01  sec   526 MBytes   441 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   552 MBytes   463 Mbits/sec                  sender
[ 15]   0.00-10.01  sec   552 MBytes   463 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   534 MBytes   448 Mbits/sec                  sender
[ 17]   0.00-10.01  sec   534 MBytes   448 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   558 MBytes   468 Mbits/sec                  sender
[ 19]   0.00-10.01  sec   558 MBytes   468 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  4.24 GBytes  3.64 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  4.24 GBytes  3.64 Gbits/sec                  receiver

Unless I am reading the output wrong, it seems like adding the switch iperf switch -P8 did not improve the performance, output below, seems like im still capped at 1Gb.

I will try some of the tweaks from that article and report back.



Connecting to host 10.0.0.1, port 45277
[  5] local 10.0.0.240 port 61183 connected to 10.0.0.1 port 45277
[  7] local 10.0.0.240 port 61184 connected to 10.0.0.1 port 45277
[  9] local 10.0.0.240 port 61185 connected to 10.0.0.1 port 45277
[ 11] local 10.0.0.240 port 61186 connected to 10.0.0.1 port 45277
[ 13] local 10.0.0.240 port 61187 connected to 10.0.0.1 port 45277
[ 15] local 10.0.0.240 port 61188 connected to 10.0.0.1 port 45277
[ 17] local 10.0.0.240 port 61189 connected to 10.0.0.1 port 45277
[ 19] local 10.0.0.240 port 61190 connected to 10.0.0.1 port 45277
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.50 MBytes  46.0 Mbits/sec
[  7]   0.00-1.00   sec  7.00 MBytes  58.5 Mbits/sec
[  9]   0.00-1.00   sec  13.0 MBytes   109 Mbits/sec
[ 11]   0.00-1.00   sec  12.9 MBytes   108 Mbits/sec
[ 13]   0.00-1.00   sec  19.9 MBytes   166 Mbits/sec
[ 15]   0.00-1.00   sec  24.9 MBytes   208 Mbits/sec
[ 17]   0.00-1.00   sec  16.0 MBytes   134 Mbits/sec
[ 19]   0.00-1.00   sec  11.6 MBytes  97.2 Mbits/sec
[SUM]   0.00-1.00   sec   111 MBytes   926 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  11.8 MBytes  98.9 Mbits/sec
[  7]   1.00-2.00   sec  14.0 MBytes   118 Mbits/sec
[  9]   1.00-2.00   sec  12.0 MBytes   101 Mbits/sec
[ 11]   1.00-2.00   sec  20.0 MBytes   168 Mbits/sec
[ 13]   1.00-2.00   sec  16.1 MBytes   136 Mbits/sec
[ 15]   1.00-2.00   sec  22.4 MBytes   188 Mbits/sec
[ 17]   1.00-2.00   sec  14.1 MBytes   119 Mbits/sec
[ 19]   1.00-2.00   sec  17.9 MBytes   150 Mbits/sec
[SUM]   1.00-2.00   sec   128 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.01   sec  16.0 MBytes   133 Mbits/sec
[  7]   2.00-3.01   sec  22.4 MBytes   186 Mbits/sec
[  9]   2.00-3.01   sec  13.4 MBytes   111 Mbits/sec
[ 11]   2.00-3.01   sec  21.2 MBytes   177 Mbits/sec
[ 13]   2.00-3.01   sec  11.1 MBytes  92.4 Mbits/sec
[ 15]   2.00-3.01   sec  11.0 MBytes  91.4 Mbits/sec
[ 17]   2.00-3.01   sec  10.1 MBytes  84.1 Mbits/sec
[ 19]   2.00-3.01   sec  25.2 MBytes   210 Mbits/sec
[SUM]   2.00-3.01   sec   130 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.01-4.00   sec  23.5 MBytes   198 Mbits/sec
[  7]   3.01-4.00   sec  16.2 MBytes   137 Mbits/sec
[  9]   3.01-4.00   sec  12.5 MBytes   105 Mbits/sec
[ 11]   3.01-4.00   sec  16.5 MBytes   139 Mbits/sec
[ 13]   3.01-4.00   sec  19.4 MBytes   163 Mbits/sec
[ 15]   3.01-4.00   sec  8.62 MBytes  72.8 Mbits/sec
[ 17]   3.01-4.00   sec  18.9 MBytes   159 Mbits/sec
[ 19]   3.01-4.00   sec  13.0 MBytes   110 Mbits/sec
[SUM]   3.01-4.00   sec   129 MBytes  1.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  20.4 MBytes   171 Mbits/sec
[  7]   4.00-5.00   sec  11.9 MBytes  99.6 Mbits/sec
[  9]   4.00-5.00   sec  11.6 MBytes  97.5 Mbits/sec
[ 11]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec
[ 13]   4.00-5.00   sec  25.6 MBytes   215 Mbits/sec
[ 15]   4.00-5.00   sec  12.8 MBytes   107 Mbits/sec
[ 17]   4.00-5.00   sec  21.1 MBytes   177 Mbits/sec
[ 19]   4.00-5.00   sec  9.88 MBytes  82.8 Mbits/sec
[SUM]   4.00-5.00   sec   128 MBytes  1.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.01   sec  19.6 MBytes   164 Mbits/sec
[  7]   5.00-6.01   sec  13.1 MBytes   109 Mbits/sec
[  9]   5.00-6.01   sec  14.9 MBytes   124 Mbits/sec
[ 11]   5.00-6.01   sec  12.2 MBytes   102 Mbits/sec
[ 13]   5.00-6.01   sec  13.1 MBytes   109 Mbits/sec
[ 15]   5.00-6.01   sec  14.9 MBytes   124 Mbits/sec
[ 17]   5.00-6.01   sec  26.4 MBytes   220 Mbits/sec
[ 19]   5.00-6.01   sec  15.5 MBytes   129 Mbits/sec
[SUM]   5.00-6.01   sec   130 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.01-7.01   sec  26.1 MBytes   219 Mbits/sec
[  7]   6.01-7.01   sec  13.5 MBytes   113 Mbits/sec
[  9]   6.01-7.01   sec  15.8 MBytes   132 Mbits/sec
[ 11]   6.01-7.01   sec  11.0 MBytes  92.2 Mbits/sec
[ 13]   6.01-7.01   sec  13.9 MBytes   116 Mbits/sec
[ 15]   6.01-7.01   sec  11.0 MBytes  92.2 Mbits/sec
[ 17]   6.01-7.01   sec  19.1 MBytes   160 Mbits/sec
[ 19]   6.01-7.01   sec  18.9 MBytes   158 Mbits/sec
[SUM]   6.01-7.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.01-8.01   sec  11.8 MBytes  98.3 Mbits/sec
[  7]   7.01-8.01   sec  20.9 MBytes   175 Mbits/sec
[  9]   7.01-8.01   sec  15.8 MBytes   132 Mbits/sec
[ 11]   7.01-8.01   sec  17.9 MBytes   150 Mbits/sec
[ 13]   7.01-8.01   sec  18.0 MBytes   151 Mbits/sec
[ 15]   7.01-8.01   sec  13.1 MBytes   110 Mbits/sec
[ 17]   7.01-8.01   sec  17.6 MBytes   148 Mbits/sec
[ 19]   7.01-8.01   sec  14.0 MBytes   117 Mbits/sec
[SUM]   7.01-8.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.01-9.01   sec  8.00 MBytes  67.1 Mbits/sec
[  7]   8.01-9.01   sec  32.1 MBytes   269 Mbits/sec
[  9]   8.01-9.01   sec  12.8 MBytes   107 Mbits/sec
[ 11]   8.01-9.01   sec  9.88 MBytes  82.8 Mbits/sec
[ 13]   8.01-9.01   sec  22.5 MBytes   189 Mbits/sec
[ 15]   8.01-9.01   sec  19.0 MBytes   159 Mbits/sec
[ 17]   8.01-9.01   sec  10.5 MBytes  88.0 Mbits/sec
[ 19]   8.01-9.01   sec  14.1 MBytes   118 Mbits/sec
[SUM]   8.01-9.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.01-10.01  sec  18.0 MBytes   152 Mbits/sec
[  7]   9.01-10.01  sec  17.8 MBytes   150 Mbits/sec
[  9]   9.01-10.01  sec  13.4 MBytes   113 Mbits/sec
[ 11]   9.01-10.01  sec  8.25 MBytes  69.5 Mbits/sec
[ 13]   9.01-10.01  sec  25.4 MBytes   214 Mbits/sec
[ 15]   9.01-10.01  sec  15.2 MBytes   129 Mbits/sec
[ 17]   9.01-10.01  sec  12.8 MBytes   107 Mbits/sec
[ 19]   9.01-10.01  sec  17.4 MBytes   146 Mbits/sec
[SUM]   9.01-10.01  sec   128 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   161 MBytes   135 Mbits/sec                  sender
[  5]   0.00-10.02  sec   160 MBytes   134 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   169 MBytes   142 Mbits/sec                  sender
[  7]   0.00-10.02  sec   168 MBytes   141 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   135 MBytes   113 Mbits/sec                  sender
[  9]   0.00-10.02  sec   135 MBytes   113 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   145 MBytes   121 Mbits/sec                  sender
[ 11]   0.00-10.02  sec   144 MBytes   121 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   185 MBytes   155 Mbits/sec                  sender
[ 13]   0.00-10.02  sec   185 MBytes   155 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   153 MBytes   128 Mbits/sec                  sender
[ 15]   0.00-10.02  sec   152 MBytes   128 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   167 MBytes   140 Mbits/sec                  sender
[ 17]   0.00-10.02  sec   166 MBytes   139 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   158 MBytes   132 Mbits/sec                  sender
[ 19]   0.00-10.02  sec   157 MBytes   131 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  1.24 GBytes  1.07 Gbits/sec                  sender
[SUM]   0.00-10.02  sec  1.24 GBytes  1.06 Gbits/sec                  receiver

iperf Done.

I added a 10Gb NIC to my OPNsense box and its negotiating the 10Gb link just fine, however when I run an iperf3 test from another client to the lan nic I only achieve 1Gb speeds, (931 Mbits/sec). I was expecting to at least hit 6000 Mbits/sec, not 931. I have IPS disabled for this test, I am not using Jumbo Frames. What am I missing here, even without Jumbo Frames I should be getting at least double or 5x what I am seeing. Details and pics below, open to trying anything. Even weirder is that I have 2Gb internet and my windows client hits 2Gb downloads on a speedtest. How could the WAN connected at 2.5Gb be faster than an internal speed test using iperf on the 10Gb lan nic.


Current OPNsense box Specs:

Intel D-1541 XEON 8 Core + Hyperthreading
32GB Ram
128GB M.2 SSD
Intel X550-T2 PCIe 4X 10Gb NIC

Client:
Windows 11
11th Gen i9-11900k
32GB ram
Intel X550-T2

Switch:
Brocade ICX-7250 (8x SFP+ 10Gb ports)

Well I think I answered my own questions at least partially.

I just created a new PS routine to download an EICAR file and this time the blocked traffic appeared in the logs as IPV6. So I guess some rules for sure support IPV6, I'm just really surprised that no other rules are firing for IPV6 at the moment. I posted PS routine elsewhere but I might as well share the testing process below, I guess its useful for both testing IPS mode and IPV6 :)



Here are some cleaner instructions for validating that IPS is blocking for the average windows user:


Step 1: Enable the rule ""OPNsense-App-detect/test", located at: Services/Intrusion Detection/Administration/Download

Step 2: Open the PowerShell ISE

Step 3: Paste in the following code

$url = "pkg.opnsense.org/test/eicar.com.txt"
$dest = "C:\temp\eicar.com.txt"
Invoke-RestMethod -Uri $url -OutFile $dest

Step 4: Click the Green Run arrow

Step 5: Check your IPS Alerts, located at: Services/Intrusion Detection/Administration/Alerts. You should see a hit for "OPNsense test eicar virus"

Step 6: Check C:\Temp\ for the creation of any new files named "eicar.com.txt", you should have none and your Powershell ISE should be just sitting there hung looking like its doing nothing, well thats because your IDS is blocking the download.

Congrats.

Here are some cleaner instructions for validating that IPS is blocking for the average windows user:


Step 1: Enable the rule ""OPNsense-App-detect/test", located at: Services/Intrusion Detection/Administration/Download

Step 2: Open the PowerShell ISE

Step 3: Paste in the following code

$url = "pkg.opnsense.org/test/eicar.com.txt"
$dest = "C:\temp\eicar.com.txt"
Invoke-RestMethod -Uri $url -OutFile $dest

Step 4: Click the Green Run arrow

Step 5: Check your IPS Alerts, located at: Services/Intrusion Detection/Administration/Alerts. You should see a hit for "OPNsense test eicar virus"

Step 6: Check C:\Temp\ for the creation of any new files named "eicar.com.txt", you should have none and your Powershell ISE should be just sitting there hung looking like its doing nothing, well thats because your IDS is blocking the download.

Congrats.

I have IPV6 configured and working very nicely, confirmed on all test sites. I also have IPS in blocking mode on both the WAN & LAN working very nicely, verified via EICAR being blocked.

However, in my IPS logs I have yet to see any WAN traffic associated to my IPv6 Address. Is that normal, does IPS support IPV6, I average about 10 blocked attacks per minute on IPv4, I cant imagine no one is scanning my IPv6 address ?

I did not clear the states, did not know I had too

10.X is correct I said 192 for no good reason other than security through obscurity

I tried putting the block rule on the top and on the bottom and in testing nothing is working, the client can still reach 8.8.8.8 not sure what is going on here.

If I wanted to make this logic a little more complex how should I order the below rules.


Sodo Logic:

Block IP (192.168.1.242) to (ANY) port (53)

Allow IP (192.168.1.242) to (1.1.1.3) Port (53)


Goal:

I want the IP address (192.168.1.242) to only have the ability to query the DNS server 1.1.1.3 for DNS requests. So for example is the user of the PC with the IP address of (192.168.1.242) decided to change their local network card settings to use the DNS server 8.8.8.8 their traffic should get blocked and ultimately web pages will fail to load.

Thanks for the help

I simply wanted to block an internal client with the IP address of (192.168.1.242) from accessing 8.8.8.8 (google dns) where would be the best place to put that rule.

Testing and checking has failed hence why I ask, not sure what I am overlooking here, would a floating rule be best ?

Im also running into an issue where if the client has IPv6 all DNS filtering fails. I don't see any way to exclude certain clients from getting an IPV6 address ?

I also had an idea where instead of using DHCP to set a dns value, I would leave it to my defaults and create a port forward NAT rule to catch any DNS requests from protected clients and redirect them to 1.1.1.3 for dns traffic. However my experiment with that failed and the clients can still get to undesired sites. Not sure what I am doing wrong here.

In effort easily block inappropriate sites I would like to leverage cloudfare's DNS 1.1.1.3. My OPNsense firewall is currently configured at a global level for DNS/TSL (DOT) via 1.1.1.1 and its ipv6 counterpart, that setup is working just fine, I don't wish to block all devices. So, via DHCP I cant set a reservation for my children's MAC Address and in DHCP I can override the DNS value and set it to 1.1.1.3 (Family DNS). This effectively accomplishes the goal but its a weak solution that can be easily bypassed.

1. Is there anyway to achieve the same goal above but without having to leverage the DHCP reservation trick. One draw back I have found is that it forces the client to only use IPV4, ISC DHCP does not seem to allow IPV6 values in the config. In addition it also breaks DNS over TLS as it reverts back to simple DNS mode. I guess what I would like is some way in Unbound DNS to apply a rule that says these MAC addresses should leverage this (DNS over TLS) config, and all other clients use the global values, I am not seeing any way to do that today, perhaps I am wrong ?

2. If the above request is not possible what are some firewall rules you all would recommend to better harden this approach. Its on the tip of my tongue but I cant place it, I am looking for a rule that could prevent the (child / protected) pc from leveraging any other DNS server. For example if the protected pc had its local network settings modified and the DNS forced to 8.8.8.8, I would not want to allow that traffic to pass, again just not sure on the proper order of rules and types I would need to achieve that while also allowing all other clients to work normally.

Thanks.

I have a spare QAT card that I could add to my Opensense box, would enabling QAT provide any benefits outside of VPN use cases. For example would it accelerate or improve performance of SSL or TLS in regards to clients browsing websites such as youtube. Just trying to understand if there would be any benefit at all besides accelerating VPN tunnels, the network in question has no VPN or IPSEC tunnels of any kind, it a clean flat network. Web Proxy, IDS, etc. anything that could warrant the extra power draw ?

Thanks.