Messages

226

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 02, 2020, 11:37:08 pm »

Just wanted to post here due to the excellent testing from OP and to corroborate the numbers that OP is seeing.

My testing setup is as follows:
ESXi 6.7u3, host has an E3 1220v3 and 32GB of RAM
All Firewall VMs have 2vCPU. 5GB of RAM allocated to OPNsense.
VMXnet3 NICs negotiated at 10gbps

In pfSense and OPNsense, I disabled all of the hardware offloading features. I am using client and server VMs on the WAN and LAN sides of the firewall VMs. This means I am pushing/pulling traffic through the firewalls, I am not running iperf directly on any of the firewalls themselves. Because I am doing this on a single ESXi host and the traffic is within the same host/vSwitch, the traffic is never routed to my physical network switch and therefore I can test higher throughput.

pfSense and OPNsense were both out of the box installs with their default rulesets. I did not add any packages or make any config changes outside of making sure that all hardware offloading was disabled. All iperf3 tests were run with the LAN side client pulling traffic through the WAN side interface, to simulate a large download. However, if I perform upload tests, my throughput results are the same. All iperf3 tests were run for 60 seconds and used the default MTU of 1500. The results below show the average of the 60 second runs. I ran each test twice, and used the final result to allow the firewalls to "warm up" and stabilize with their throughput during testing.

Code: [Select]

pfSense 2.4.5p1 1500MTU receiving from WAN, vmx3 NICs, all hardware offloading disabled, default ruleset
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  31.5 GBytes  4.50 Gbits/sec  11715             sender
[  5]   0.00-60.00  sec  31.5 GBytes  4.50 Gbits/sec                  receiver

OpenWRT 19.07.3 1500MTU receiving from WAN, vmx3 NICs, default ruleset
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  47.5 GBytes  6.81 Gbits/sec  44252             sender
[  5]   0.00-60.00  sec  47.5 GBytes  6.81 Gbits/sec                  receiver

OPNsense 20.7.2 1500MTU receiving from WAN, vmx3 NICs, all hardware offloading disabled, default ruleset
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  6.83 GBytes   977 Mbits/sec  459             sender
[  5]   0.00-60.00  sec  6.82 GBytes   977 Mbits/sec                  receiver

I also notice that while doing a throughput test on OPNsense, one of the vCPUs is completely consumed. I did not see this behavior with Linux or pfSense on my testing, screenshot attached shows the CPU usage I'm seeing while the iperf3 test is running.

227

20.7 Legacy Series / Re: Unbound DNS query & assistance

« on: August 14, 2020, 04:08:37 pm »

Yes, I should have clarified that in my response. What I was trying to convey is that most people will choose one of the 3 available large providers. Google, CloudFlare, or Quad9.

Regardless of which one you prefer, I would only recommend configuring Unbound to use one provider at a time. If you do want to run multiple different providers, I would align them so that you aren't using a combination of filtered/unfiltered results so that you can keep the client experience consistent and potentially reduce troubleshooting if there's a DNS issue.

228

20.7 Legacy Series / Re: Unbound DNS query & assistance

« on: August 13, 2020, 04:41:46 pm »

Can Unbound DNS probe every server I have listed and serve up the result which responded first? If so, how would I configure this?

This used to be possible with DNSMASQ, there was a separate ability to query sequentially, or with with a round robin style for all specified DNS servers.

However, for Unbound, I'm only aware of it using a round robin style query by default.

It's also worth noting, your config mixes two DNS providers with different use cases. Your Google DNS and CloudFlare DNS will do DNSSEC/DoT, but no filtering. Your Quad9 will do DNSSEC/DoT, and malware filtering. Due to the way Unbound will randomly query either one, you may get inconsistent results back to your clients. It's very likely that google may recommend one CDN location, while Quad9 may provide results for another. You'd be better off picking one of those two services only. Which one is another discussion entirely but, Quad9 has a much better stance on user privacy so I know which one I'd go with. 

229

20.1 Legacy Series / Re: Unbound Plus Plugin and DoT hostname validation?

« on: May 08, 2020, 03:54:59 pm »

Thanks for the reply. If it is helpful, I am happy to test future versions. I have a few OPNsense VMs in a lab that I can demo stuff on before I push it to production.

230

20.1 Legacy Series / Unbound Plus Plugin and DoT hostname validation?

« on: May 08, 2020, 06:11:18 am »

I had a question for @mimugmail or anyone else that may know how the Unbound Plus plugin is doing hostname validation for DoT implementations?

Currently, I'm using regular Unbound with the following entries in the Advanced section:

Code: [Select]

# TLS Config
tls-cert-bundle: "/etc/ssl/cert.pem"
# Forwarding Config
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 1.1.1.1@853#1dot1dot1dot1.cloudflare-dns.com
forward-addr: 1.0.0.1@853#1dot1dot1dot1.cloudflare-dns.com
I would like to convert to using Unbound Plus plugin and input my DoT servers there. However, it does not appear to use the hostname for validation? Only the IP and Port?

231

20.1 Legacy Series / Re: Packets are being ignored - why?

« on: April 21, 2020, 12:42:26 am »

You mention initial problems when migrating rule sets, how were these migrated originally? Did you have to manually re-create them in the GUI or was it some other method?

On my OPNsense test box and on my production box, when I connect via SSH and drop to a console, I get a full output of the rules present when I run "pfctl -sr". I'm running OPNsense 20.1.4 AMD64/OpenSSL.

232

20.1 Legacy Series / Re: Packets are being ignored - why?

« on: April 20, 2020, 04:02:20 pm »

It's hard to say without getting more detail on your rulesets (screenshots, including the "automatically generated" rules). Have you tried an external port scanner to see if you have other holes on the WAN interface that should not be open? Something like the ShieldsUP scanner on grc.com can be useful to make sure that the rule you want to work is actually doing what its supposed to do.

What I can say is that by default, OPNsense will block all unsolicited incoming connections, just like pfSense does. So I suspect this is less of an OPNsense issue and more of a tweaking issue that will need to be reviewed line by line to find the offending rule.

233

20.1 Legacy Series / Re: Install files verification fails

« on: April 14, 2020, 09:19:58 pm »

I downloaded just now from the same mirror in your first post and the filehash appears to match for me. This is on a windows box without openssl so I can't run the other verification steps that you list.

Code: [Select]

Get-FileHash .\OPNsense-20.1-OpenSSL-vga-amd64.img.bz2 -algorithm sha256

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          019A877C4B4CB96CFDA62D041774A91C030C5A8ECD58F8C3FD0067C7AC392982       D:\downloads\OPNsense-20.1-Op...

PS D:\downloads> cat .\OPNsense-20.1-OpenSSL-checksums-amd64.sha256
SHA256 (OPNsense-20.1-OpenSSL-dvd-amd64.iso.bz2) = 4b15e9b3d72732d325c5eaf46ba34575d4de8cdc3e3ac1b10666c7372563be6d
SHA256 (OPNsense-20.1-OpenSSL-nano-amd64.img.bz2) = 27544a78ae03d480a483cfd2e7cfa703b60e50938a1ed188ec3ccde6c426fefe
SHA256 (OPNsense-20.1-OpenSSL-serial-amd64.img.bz2) = f93bbcbe92059c5de49f22d485da292952b48658a28d1cdaf83191e8c95c03c2
SHA256 (OPNsense-20.1-OpenSSL-vga-amd64.img.bz2) = 019a877c4b4cb96cfda62d041774a91c030c5a8ecd58f8c3fd0067c7ac392982

234

20.1 Legacy Series / Re: Default route persists when upstream gateway down

« on: April 14, 2020, 05:30:10 am »

I presume this is a scenario in which you have multiple gateways defined within the router and you want the router to switch to a new gateway if another one fails? Do you have any gateway groups and gateway weighting defined yet? That should accomplish what you want if you have multiple WAN interfaces and you want one of them to fail over if one is marked "down".

Another option that is helpful for multiple WAN (gateways) is to enable state killing on gateway failure. This prevents some clients on the LAN from re-using an existing connection through a gateway that has failed. You can set this under Firewall/Settings/Advanced/Gateway Monitoring.

235

Hardware and Performance / Re: Constant high load on idle install

« on: February 01, 2020, 08:10:56 pm »

I saw similar issues on a fresh virtualized install. In my case, I was also seeing pflog0 promiscuous enabled/disable messages spamming the logs many times per second. This seemed to be related to IPV6 unable to pull a prefix delegation on the WAN interface of the OPNsense VM.

Try disabling IPV6 on WAN and see if this clears up? If so, it's likely related to the issue I saw in my LAB.

236

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 29, 2020, 06:53:02 pm »

Yes, I see what you mean with pfSense being able to sustain a higher throughput under the same circumstance.

Unfortunately, I'm not sure if this is useful data because it isn't telling us how fast either solution can actually route packets. In actual use, we'd be using pfSense or OPNsense as a firewall/router setup and we would want to see how quickly they can push traffic through themselves, rather than serving traffic directly through one interface.

I suppose the last things I would check would be to verify that OPNsense and pfSense both have OpenVMTools installed/running. And run a 'top -aSCHIP' on an SSH console for both of them and see what their CPU usage is when running your transfer test. Watching them under load may reveal a bottleneck, especially on the OPNsense router since that one seems to be under performing in your tests.

Finally, if you feel like digging in a bit more, I would recommend doing a test using Proxmox client and server VMs, and have two switches inside Proxmox. One can be used for the WAN port and one switch is a private switch with no physical uplinks, we'll use this for the LAN port. Doing this method will allow you to simulate actual routing performance of both solutions and you won't need an extra physical client.

237

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 29, 2020, 04:40:42 pm »

Just to confirm, does your setup look like the below diagram? OPNsense is not hosting the client or server portion of iperf, correct?

238

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 19, 2020, 10:37:43 pm »

A quick reply regarding the OPNsense CPU utilization. In my case this seemed to be related to DHCP6 being enabled out of the box. I'm not sure if OPNsense was trying to delegate a prefix to the LAN side over and over and causing high CPU usage on unbound? My logs are filled with this:

Code: [Select]

kernel: pflog0: promiscuous mode disabled
kernel: pflog0: promiscuous mode enabled
I was seeing these events spamming the logs constantly every second. As soon as I disabled DHCP6 on WAN, these errors went away and idle CPU usage on OPNsense returned to normal.

Here are the results of a current iperf3 test, using the same VMs described in my post above. These throughput numbers are much more consistent now that OPNsense has normal CPU usage.

Code: [Select]

Accepted connection from 192.168.1.232, port 4084
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 24664
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   1.00-2.00   sec   323 MBytes  2.71 Gbits/sec
[  5]   2.00-3.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   3.00-4.00   sec   344 MBytes  2.88 Gbits/sec
[  5]   4.00-5.00   sec   316 MBytes  2.65 Gbits/sec
[  5]   5.00-6.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   6.00-7.00   sec   353 MBytes  2.96 Gbits/sec
[  5]   7.00-8.00   sec   349 MBytes  2.93 Gbits/sec
[  5]   8.00-9.00   sec   356 MBytes  2.98 Gbits/sec
[  5]   9.00-10.00  sec   345 MBytes  2.89 Gbits/sec
[  5]  10.00-11.00  sec   305 MBytes  2.56 Gbits/sec
[  5]  11.00-12.00  sec   348 MBytes  2.92 Gbits/sec
[  5]  12.00-13.00  sec   341 MBytes  2.86 Gbits/sec
[  5]  13.00-14.00  sec   343 MBytes  2.87 Gbits/sec
[  5]  14.00-15.00  sec   331 MBytes  2.77 Gbits/sec
[  5]  15.00-15.04  sec  14.8 MBytes  3.04 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  4.81 GBytes  2.75 Gbits/sec                  receiver


239

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 19, 2020, 08:48:02 pm »

Here are my numbers. Both of these are fresh out of the box installs, OPNsense 19.7.9 and pfSense 2.4.4p3, both are X86_64.

Hypervisor Specs:
VMware ESXi 6.7u3
2x Intel Xeon E5620
All VMs are running open-vm-tools, including the firewalls

Specs on both firewall VMs are as follows:
2x CPU
4GB RAM
2x VMXnet3 NICs (one WAN, one LAN)

I have two other VMs running as iperf3 server and client. The "server" VM is on the WAN side of these firewalls, the client VM is on the "LAN" side. This is to test traffic throughput of the router itself. Never try to run these tests with the router/firewall acting as a client or server, you will not get accurate results.

pfSense 2.4.4p3:

Code: [Select]

Accepted connection from 192.168.1.230, port 56492
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.230 port 45828
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   314 MBytes  2.64 Gbits/sec
[  5]   1.00-2.00   sec   459 MBytes  3.85 Gbits/sec
[  5]   2.00-3.00   sec   407 MBytes  3.41 Gbits/sec
[  5]   3.00-4.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   4.00-5.00   sec   351 MBytes  2.94 Gbits/sec
[  5]   5.00-6.00   sec   372 MBytes  3.12 Gbits/sec
[  5]   6.00-7.00   sec   424 MBytes  3.55 Gbits/sec
[  5]   7.00-8.00   sec   410 MBytes  3.44 Gbits/sec
[  5]   8.00-9.00   sec   443 MBytes  3.71 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-11.00  sec   448 MBytes  3.76 Gbits/sec
[  5]  11.00-12.00  sec   428 MBytes  3.59 Gbits/sec
[  5]  12.00-13.00  sec   404 MBytes  3.39 Gbits/sec
[  5]  13.00-14.00  sec   419 MBytes  3.51 Gbits/sec
[  5]  14.00-15.00  sec   445 MBytes  3.73 Gbits/sec
[  5]  15.00-15.04  sec  16.1 MBytes  3.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  5.98 GBytes  3.42 Gbits/sec                  receiver
OPNsense 19.7.9 (no tuning, Unbound using lots of CPU)

Code: [Select]

Accepted connection from 192.168.1.232, port 15150
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 46858
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   304 MBytes  2.55 Gbits/sec
[  5]   1.00-2.00   sec  88.9 MBytes   746 Mbits/sec
[  5]   2.00-3.00   sec   371 MBytes  3.11 Gbits/sec
[  5]   3.00-4.00   sec   164 MBytes  1.38 Gbits/sec
[  5]   4.00-5.00   sec   420 MBytes  3.52 Gbits/sec
[  5]   5.00-6.00   sec  79.4 MBytes   666 Mbits/sec
[  5]   6.00-7.00   sec   400 MBytes  3.36 Gbits/sec
[  5]   7.00-8.00   sec  97.7 MBytes   820 Mbits/sec
[  5]   8.00-9.00   sec   403 MBytes  3.38 Gbits/sec
[  5]   9.00-10.00  sec   399 MBytes  3.35 Gbits/sec
[  5]  10.00-11.00  sec   104 MBytes   872 Mbits/sec
[  5]  11.00-12.00  sec   374 MBytes  3.14 Gbits/sec
[  5]  12.00-13.00  sec  74.0 MBytes   621 Mbits/sec
[  5]  13.00-14.00  sec   289 MBytes  2.42 Gbits/sec
[  5]  14.00-15.00  sec   135 MBytes  1.13 Gbits/sec
[  5]  15.00-15.04  sec  3.24 MBytes   675 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  3.62 GBytes  2.07 Gbits/sec                  receiver

OPNsense 19.7.9 (set unbound to use Quad9 DoT using forwarding mode)

Code: [Select]

Accepted connection from 192.168.1.232, port 58840
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 16760
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   214 MBytes  1.80 Gbits/sec
[  5]   1.00-2.00   sec   268 MBytes  2.25 Gbits/sec
[  5]   2.00-3.00   sec   312 MBytes  2.61 Gbits/sec
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   4.00-5.00   sec   273 MBytes  2.29 Gbits/sec
[  5]   5.00-6.00   sec   259 MBytes  2.17 Gbits/sec
[  5]   6.00-7.00   sec   201 MBytes  1.69 Gbits/sec
[  5]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec
[  5]   8.00-9.00   sec   311 MBytes  2.61 Gbits/sec
[  5]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec
[  5]  10.00-11.00  sec   237 MBytes  1.99 Gbits/sec
[  5]  11.00-12.00  sec   298 MBytes  2.50 Gbits/sec
[  5]  12.00-13.00  sec   322 MBytes  2.70 Gbits/sec
[  5]  13.00-14.00  sec   291 MBytes  2.44 Gbits/sec
[  5]  14.00-15.00  sec   303 MBytes  2.54 Gbits/sec
[  5]  15.00-15.03  sec  8.95 MBytes  2.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.03  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.03  sec  3.92 GBytes  2.24 Gbits/sec                  receiver

As we can see, OPNsense does seem to have some throughput limits out of the box. Still, I am seeing much higher throughput values than you are so it's important to make sure your tests are using servers/clients on the WAN and LAN sides of the firewall.

Finally, here's a screenshot of what a 'top -aSCHIP' looks like on the OPNsense 19.7.9 VM, you can see the high CPU usage for some reason with unbound. You may want to check if your OPNsense VM exhibits the same high CPU behavior, as that can also take away from the overall throughput.

240

19.7 Legacy Series / Re: Python 3.7 using max memory and filling disk space

« on: December 15, 2019, 07:22:24 pm »

I did a System/Firmware/Health audit and below are the results of this:

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
>>> Check installed kernel version
Version 19.7.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 19.7.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for and install missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
***DONE***
At this point I'm stumped. Not sure what caused this? I've run memcheck on the firewall as well and have no errors, so I don't think its hardware related. It has been stable for months until the odd Python CPU usage and disk fill yesterday.