Messages

1

20.1 Production Series / Re: Unbound Plus Plugin and DoT hostname validation?

« on: May 08, 2020, 03:54:59 pm »

Thanks for the reply. If it is helpful, I am happy to test future versions. I have a few OPNsense VMs in a lab that I can demo stuff on before I push it to production.

2

20.1 Production Series / Unbound Plus Plugin and DoT hostname validation?

« on: May 08, 2020, 06:11:18 am »

I had a question for @mimugmail or anyone else that may know how the Unbound Plus plugin is doing hostname validation for DoT implementations?

Currently, I'm using regular Unbound with the following entries in the Advanced section:

Code: [Select]

# TLS Config
tls-cert-bundle: "/etc/ssl/cert.pem"
# Forwarding Config
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 1.1.1.1@853#1dot1dot1dot1.cloudflare-dns.com
forward-addr: 1.0.0.1@853#1dot1dot1dot1.cloudflare-dns.com
I would like to convert to using Unbound Plus plugin and input my DoT servers there. However, it does not appear to use the hostname for validation? Only the IP and Port?

3

20.1 Production Series / Re: Packets are being ignored - why?

« on: April 21, 2020, 12:42:26 am »

You mention initial problems when migrating rule sets, how were these migrated originally? Did you have to manually re-create them in the GUI or was it some other method?

On my OPNsense test box and on my production box, when I connect via SSH and drop to a console, I get a full output of the rules present when I run "pfctl -sr". I'm running OPNsense 20.1.4 AMD64/OpenSSL.

4

20.1 Production Series / Re: Packets are being ignored - why?

« on: April 20, 2020, 04:02:20 pm »

It's hard to say without getting more detail on your rulesets (screenshots, including the "automatically generated" rules). Have you tried an external port scanner to see if you have other holes on the WAN interface that should not be open? Something like the ShieldsUP scanner on grc.com can be useful to make sure that the rule you want to work is actually doing what its supposed to do.

What I can say is that by default, OPNsense will block all unsolicited incoming connections, just like pfSense does. So I suspect this is less of an OPNsense issue and more of a tweaking issue that will need to be reviewed line by line to find the offending rule.

5

20.1 Production Series / Re: Install files verification fails

« on: April 14, 2020, 09:19:58 pm »

I downloaded just now from the same mirror in your first post and the filehash appears to match for me. This is on a windows box without openssl so I can't run the other verification steps that you list.

Code: [Select]

Get-FileHash .\OPNsense-20.1-OpenSSL-vga-amd64.img.bz2 -algorithm sha256

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          019A877C4B4CB96CFDA62D041774A91C030C5A8ECD58F8C3FD0067C7AC392982       D:\downloads\OPNsense-20.1-Op...

PS D:\downloads> cat .\OPNsense-20.1-OpenSSL-checksums-amd64.sha256
SHA256 (OPNsense-20.1-OpenSSL-dvd-amd64.iso.bz2) = 4b15e9b3d72732d325c5eaf46ba34575d4de8cdc3e3ac1b10666c7372563be6d
SHA256 (OPNsense-20.1-OpenSSL-nano-amd64.img.bz2) = 27544a78ae03d480a483cfd2e7cfa703b60e50938a1ed188ec3ccde6c426fefe
SHA256 (OPNsense-20.1-OpenSSL-serial-amd64.img.bz2) = f93bbcbe92059c5de49f22d485da292952b48658a28d1cdaf83191e8c95c03c2
SHA256 (OPNsense-20.1-OpenSSL-vga-amd64.img.bz2) = 019a877c4b4cb96cfda62d041774a91c030c5a8ecd58f8c3fd0067c7ac392982

6

20.1 Production Series / Re: Default route persists when upstream gateway down

« on: April 14, 2020, 05:30:10 am »

I presume this is a scenario in which you have multiple gateways defined within the router and you want the router to switch to a new gateway if another one fails? Do you have any gateway groups and gateway weighting defined yet? That should accomplish what you want if you have multiple WAN interfaces and you want one of them to fail over if one is marked "down".

Another option that is helpful for multiple WAN (gateways) is to enable state killing on gateway failure. This prevents some clients on the LAN from re-using an existing connection through a gateway that has failed. You can set this under Firewall/Settings/Advanced/Gateway Monitoring.

7

Hardware and Performance / Re: Constant high load on idle install

« on: February 01, 2020, 08:10:56 pm »

I saw similar issues on a fresh virtualized install. In my case, I was also seeing pflog0 promiscuous enabled/disable messages spamming the logs many times per second. This seemed to be related to IPV6 unable to pull a prefix delegation on the WAN interface of the OPNsense VM.

Try disabling IPV6 on WAN and see if this clears up? If so, it's likely related to the issue I saw in my LAB.

8

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 29, 2020, 06:53:02 pm »

Yes, I see what you mean with pfSense being able to sustain a higher throughput under the same circumstance.

Unfortunately, I'm not sure if this is useful data because it isn't telling us how fast either solution can actually route packets. In actual use, we'd be using pfSense or OPNsense as a firewall/router setup and we would want to see how quickly they can push traffic through themselves, rather than serving traffic directly through one interface.

I suppose the last things I would check would be to verify that OPNsense and pfSense both have OpenVMTools installed/running. And run a 'top -aSCHIP' on an SSH console for both of them and see what their CPU usage is when running your transfer test. Watching them under load may reveal a bottleneck, especially on the OPNsense router since that one seems to be under performing in your tests.

Finally, if you feel like digging in a bit more, I would recommend doing a test using Proxmox client and server VMs, and have two switches inside Proxmox. One can be used for the WAN port and one switch is a private switch with no physical uplinks, we'll use this for the LAN port. Doing this method will allow you to simulate actual routing performance of both solutions and you won't need an extra physical client.

9

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 29, 2020, 04:40:42 pm »

Just to confirm, does your setup look like the below diagram? OPNsense is not hosting the client or server portion of iperf, correct?

10

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 19, 2020, 10:37:43 pm »

A quick reply regarding the OPNsense CPU utilization. In my case this seemed to be related to DHCP6 being enabled out of the box. I'm not sure if OPNsense was trying to delegate a prefix to the LAN side over and over and causing high CPU usage on unbound? My logs are filled with this:

Code: [Select]

kernel: pflog0: promiscuous mode disabled
kernel: pflog0: promiscuous mode enabled
I was seeing these events spamming the logs constantly every second. As soon as I disabled DHCP6 on WAN, these errors went away and idle CPU usage on OPNsense returned to normal.

Here are the results of a current iperf3 test, using the same VMs described in my post above. These throughput numbers are much more consistent now that OPNsense has normal CPU usage.

Code: [Select]

Accepted connection from 192.168.1.232, port 4084
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 24664
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   1.00-2.00   sec   323 MBytes  2.71 Gbits/sec
[  5]   2.00-3.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   3.00-4.00   sec   344 MBytes  2.88 Gbits/sec
[  5]   4.00-5.00   sec   316 MBytes  2.65 Gbits/sec
[  5]   5.00-6.00   sec   357 MBytes  2.99 Gbits/sec
[  5]   6.00-7.00   sec   353 MBytes  2.96 Gbits/sec
[  5]   7.00-8.00   sec   349 MBytes  2.93 Gbits/sec
[  5]   8.00-9.00   sec   356 MBytes  2.98 Gbits/sec
[  5]   9.00-10.00  sec   345 MBytes  2.89 Gbits/sec
[  5]  10.00-11.00  sec   305 MBytes  2.56 Gbits/sec
[  5]  11.00-12.00  sec   348 MBytes  2.92 Gbits/sec
[  5]  12.00-13.00  sec   341 MBytes  2.86 Gbits/sec
[  5]  13.00-14.00  sec   343 MBytes  2.87 Gbits/sec
[  5]  14.00-15.00  sec   331 MBytes  2.77 Gbits/sec
[  5]  15.00-15.04  sec  14.8 MBytes  3.04 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  4.81 GBytes  2.75 Gbits/sec                  receiver


11

Hardware and Performance / Re: OPNsense 4x slower than PFSense on same hardware

« on: January 19, 2020, 08:48:02 pm »

Here are my numbers. Both of these are fresh out of the box installs, OPNsense 19.7.9 and pfSense 2.4.4p3, both are X86_64.

Hypervisor Specs:
VMware ESXi 6.7u3
2x Intel Xeon E5620
All VMs are running open-vm-tools, including the firewalls

Specs on both firewall VMs are as follows:
2x CPU
4GB RAM
2x VMXnet3 NICs (one WAN, one LAN)

I have two other VMs running as iperf3 server and client. The "server" VM is on the WAN side of these firewalls, the client VM is on the "LAN" side. This is to test traffic throughput of the router itself. Never try to run these tests with the router/firewall acting as a client or server, you will not get accurate results.

pfSense 2.4.4p3:

Code: [Select]

Accepted connection from 192.168.1.230, port 56492
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.230 port 45828
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   314 MBytes  2.64 Gbits/sec
[  5]   1.00-2.00   sec   459 MBytes  3.85 Gbits/sec
[  5]   2.00-3.00   sec   407 MBytes  3.41 Gbits/sec
[  5]   3.00-4.00   sec   393 MBytes  3.30 Gbits/sec
[  5]   4.00-5.00   sec   351 MBytes  2.94 Gbits/sec
[  5]   5.00-6.00   sec   372 MBytes  3.12 Gbits/sec
[  5]   6.00-7.00   sec   424 MBytes  3.55 Gbits/sec
[  5]   7.00-8.00   sec   410 MBytes  3.44 Gbits/sec
[  5]   8.00-9.00   sec   443 MBytes  3.71 Gbits/sec
[  5]   9.00-10.00  sec   393 MBytes  3.30 Gbits/sec
[  5]  10.00-11.00  sec   448 MBytes  3.76 Gbits/sec
[  5]  11.00-12.00  sec   428 MBytes  3.59 Gbits/sec
[  5]  12.00-13.00  sec   404 MBytes  3.39 Gbits/sec
[  5]  13.00-14.00  sec   419 MBytes  3.51 Gbits/sec
[  5]  14.00-15.00  sec   445 MBytes  3.73 Gbits/sec
[  5]  15.00-15.04  sec  16.1 MBytes  3.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  5.98 GBytes  3.42 Gbits/sec                  receiver
OPNsense 19.7.9 (no tuning, Unbound using lots of CPU)

Code: [Select]

Accepted connection from 192.168.1.232, port 15150
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 46858
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   304 MBytes  2.55 Gbits/sec
[  5]   1.00-2.00   sec  88.9 MBytes   746 Mbits/sec
[  5]   2.00-3.00   sec   371 MBytes  3.11 Gbits/sec
[  5]   3.00-4.00   sec   164 MBytes  1.38 Gbits/sec
[  5]   4.00-5.00   sec   420 MBytes  3.52 Gbits/sec
[  5]   5.00-6.00   sec  79.4 MBytes   666 Mbits/sec
[  5]   6.00-7.00   sec   400 MBytes  3.36 Gbits/sec
[  5]   7.00-8.00   sec  97.7 MBytes   820 Mbits/sec
[  5]   8.00-9.00   sec   403 MBytes  3.38 Gbits/sec
[  5]   9.00-10.00  sec   399 MBytes  3.35 Gbits/sec
[  5]  10.00-11.00  sec   104 MBytes   872 Mbits/sec
[  5]  11.00-12.00  sec   374 MBytes  3.14 Gbits/sec
[  5]  12.00-13.00  sec  74.0 MBytes   621 Mbits/sec
[  5]  13.00-14.00  sec   289 MBytes  2.42 Gbits/sec
[  5]  14.00-15.00  sec   135 MBytes  1.13 Gbits/sec
[  5]  15.00-15.04  sec  3.24 MBytes   675 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.04  sec  3.62 GBytes  2.07 Gbits/sec                  receiver

OPNsense 19.7.9 (set unbound to use Quad9 DoT using forwarding mode)

Code: [Select]

Accepted connection from 192.168.1.232, port 58840
[  5] local 192.168.1.231 port 5201 connected to 192.168.1.232 port 16760
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   214 MBytes  1.80 Gbits/sec
[  5]   1.00-2.00   sec   268 MBytes  2.25 Gbits/sec
[  5]   2.00-3.00   sec   312 MBytes  2.61 Gbits/sec
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   4.00-5.00   sec   273 MBytes  2.29 Gbits/sec
[  5]   5.00-6.00   sec   259 MBytes  2.17 Gbits/sec
[  5]   6.00-7.00   sec   201 MBytes  1.69 Gbits/sec
[  5]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec
[  5]   8.00-9.00   sec   311 MBytes  2.61 Gbits/sec
[  5]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec
[  5]  10.00-11.00  sec   237 MBytes  1.99 Gbits/sec
[  5]  11.00-12.00  sec   298 MBytes  2.50 Gbits/sec
[  5]  12.00-13.00  sec   322 MBytes  2.70 Gbits/sec
[  5]  13.00-14.00  sec   291 MBytes  2.44 Gbits/sec
[  5]  14.00-15.00  sec   303 MBytes  2.54 Gbits/sec
[  5]  15.00-15.03  sec  8.95 MBytes  2.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-15.03  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-15.03  sec  3.92 GBytes  2.24 Gbits/sec                  receiver

As we can see, OPNsense does seem to have some throughput limits out of the box. Still, I am seeing much higher throughput values than you are so it's important to make sure your tests are using servers/clients on the WAN and LAN sides of the firewall.

Finally, here's a screenshot of what a 'top -aSCHIP' looks like on the OPNsense 19.7.9 VM, you can see the high CPU usage for some reason with unbound. You may want to check if your OPNsense VM exhibits the same high CPU behavior, as that can also take away from the overall throughput.

12

19.7 Legacy Series / Re: Python 3.7 using max memory and filling disk space

« on: December 15, 2019, 07:22:24 pm »

I did a System/Firmware/Health audit and below are the results of this:

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
>>> Check installed kernel version
Version 19.7.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 19.7.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for and install missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
***DONE***
At this point I'm stumped. Not sure what caused this? I've run memcheck on the firewall as well and have no errors, so I don't think its hardware related. It has been stable for months until the odd Python CPU usage and disk fill yesterday.

13

19.7 Legacy Series / Re: Python 3.7 using max memory and filling disk space

« on: December 15, 2019, 06:17:09 pm »

When I manually run that command I receive back "status" : "ok"

This is a stock install, I think I started with an early version of 19.7 and have upgraded it as updates were made available through the OPNsense repos. I have not made any customizations to packages and have only installed two packages available on the mirrors, WOL and Smart.

One thing I also wanted to add, the "python3.7.core" file that was filling the disk was not removed on a reboot. I shutdown the firewall gracefully and powered it back up. When it came up, the disk space was completely exhausted and I had to manually remove the "python3.7.core" file to get the space back.

14

19.7 Legacy Series / Re: Python 3.7 using max memory and filling disk space

« on: December 14, 2019, 08:56:17 pm »

Here's the output of my filter_tables.conf file.

15

19.7 Legacy Series / Re: Python 3.7 using max memory and filling disk space

« on: December 14, 2019, 07:21:40 pm »

Thank you for the response. This is a fairly basic install and I don't use aliases. Here's a screenshot of my firewall aliases table.

The only packages I have installed are wake-on-lan and smart tools. I'm not using Suricata or Sensei on this install.