Messages

I've started to tinker around with rules and noticed every 20-24 hours I have to allow all traffic into my network again in order for my wireguard services (which have been running for months with no issue) to function again.

Currently I am blocking all services (LAN) other then ports 53, 853, 80 and 443 with the ssh port and wireguard ports open. ICMP is also open.

Wireguard external IP(s) are whitelisted and can bypass all of the lan rules.

This problem also goes away instantly when all traffic is allowed on the interface...

It's a constant 20-24 hour cycle.

The services over wireguard don't stop working. They just start taking minutes to resolve and load. At first I thought it was a peering issue between myself and the server but then noticed that allowing all traffic fixes the problem..

My assumption is because I have such a tight restriction on LAN traffic there may be a cache issue or communication issue between all of the servers on my LAN. Perhaps an ARP cache issue?

I'm running the latest opnsense version, upgraded today.

Which ports/protocols do I need to open on LAN for servers to communicate with each other internally?

I have two nics (one is intel, the other is a virtio bridge). Is there a way to enable hardware offloading only for the intel nic?

If not I'll probably need to use a dedicated server instead of VM. The CPU on the VM hits 70% with 90Mbps on speedtest.net which I'm assuming is due to hardware offloading being disabled.

You are on devel and not stable branch. You shouldn't use devel immediately after a major release:

   opnsense-devel-20.1.a_44 (4 MiB: 100.00% of the 4 MiB to download)

Code Select Expand

Installed packages to be REMOVED:
opnsense-devel-19.7.r_1

New packages to be INSTALLED:
opnsense: 19.7.1

You are on devel and not stable branch. You shouldn't use devel immediately after a major release:

   opnsense-devel-20.1.a_44 (4 MiB: 100.00% of the 4 MiB to download)

I was on devel yes. I changed back to production and was upgrading to stable.

i440fx is for windows machines.

Q35 with passthrough works with pfsense 2.4 and opnsense 19.1, so I don't think is a problem with my configuration

"For windows machines" yet works fine with Opnsense 19.7.

I've never been able to get Q35 4.0+ working with Opnsense. It never detected the NICs. If you managed to (and it's not Q35 3.0) then congrats.

Just a FYI the upgrade to 19.7.1 destroyed the system on reboot. While likely fixable, it was easy enough to start from scratch.

Output of install, followed by boot (attached):

Code Select Expand

***GOT REQUEST FOR TYPE: opnsense***
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
opnsense-devel-20.1.a_44 (4 MiB: 100.00% of the 4 MiB to download)

Number of packages to be fetched: 2

The process will require 4 MiB more space.
4 MiB to be downloaded.
Fetching opnsense-devel-20.1.a_44.txz: .......... done
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (2 conflicting)
  - opnsense-19.7.1 conflicts with opnsense-devel-19.7.r_1 on /boot/brand-opnsense.4th
  - opnsense-19.7.1 conflicts with opnsense-devel-20.1.a_44 on /boot/brand-opnsense.4th
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
opnsense-devel-19.7.r_1

New packages to be INSTALLED:
opnsense: 19.7.1

Number of packages to be removed: 1
Number of packages to be installed: 1
[1/2] Deinstalling opnsense-devel-19.7.r_1...
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
[1/2] Deleting files for opnsense-devel-19.7.r_1: .......... done
[2/2] Installing opnsense-19.7.1...
[2/2] Extracting opnsense-19.7.1: .......... done
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
Keep version OPNsense\Backup\NextcloudSettings (1.0.0)
Keep version OPNsense\CaptivePortal\CaptivePortal (1.0.0)
Keep version OPNsense\Cron\Cron (1.0.1)
Keep version OPNsense\Diagnostics\Netflow (1.0.0)
Keep version OPNsense\Firewall\Alias (1.0.0)
Keep version OPNsense\IDS\IDS (1.0.3)
*** OPNsense\Monit\Monit Migration failed, check log for details
Keep version OPNsense\OpenVPN\Export (0.0.1)
Keep version OPNsense\Proxy\Proxy (1.0.3)
Migrated OPNsense\Routes\Route from  <unversioned>  to 1.0.0
Keep version OPNsense\Syslog\Syslog (1.0.0)
Keep version OPNsense\Dnscryptproxy\Forward (0.1.0)
Keep version OPNsense\Dnscryptproxy\General (0.1.0)
Keep version OPNsense\Dnscryptproxy\Whitelist (0.1.0)
Keep version OPNsense\Dnscryptproxy\Server (1.0.0)
Keep version OPNsense\Dnscryptproxy\Dnsbl (1.0.0)
Keep version OPNsense\Dnscryptproxy\Cloak (0.1.0)
Keep version OPNsense\iperf\FakeInstance (0.0.0)
Keep version OPNsense\ARPscanner\ARPscanner (1.0.0)
Keep version OPNsense\TrafficShaper\TrafficShaper (1.0.3)
Writing firmware setting...done.
Configuring login behaviour...done.
Configuring system logging...done.
Message from opnsense-19.7.1:

Roar!
Your system is up to date.
Starting web GUI...done.
Generating RRD graphs...done.
***DONE***

Q35 does not detect nics (at least 4.0+). You'll need to use i440fx which works fine.

Just a FYI the upgrade to 19.7.1 destroyed the system on reboot. While likely fixable, it was easy enough to start from scratch.

Output of install:

Code Select Expand

***GOT REQUEST FOR TYPE: opnsense***
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
opnsense-devel-20.1.a_44 (4 MiB: 100.00% of the 4 MiB to download)

Number of packages to be fetched: 2

The process will require 4 MiB more space.
4 MiB to be downloaded.
Fetching opnsense-devel-20.1.a_44.txz: .......... done
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (2 conflicting)
  - opnsense-19.7.1 conflicts with opnsense-devel-19.7.r_1 on /boot/brand-opnsense.4th
  - opnsense-19.7.1 conflicts with opnsense-devel-20.1.a_44 on /boot/brand-opnsense.4th
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
opnsense-devel-19.7.r_1

New packages to be INSTALLED:
opnsense: 19.7.1

Number of packages to be removed: 1
Number of packages to be installed: 1
[1/2] Deinstalling opnsense-devel-19.7.r_1...
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
[1/2] Deleting files for opnsense-devel-19.7.r_1: .......... done
[2/2] Installing opnsense-19.7.1...
[2/2] Extracting opnsense-19.7.1: .......... done
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
Keep version OPNsense\Backup\NextcloudSettings (1.0.0)
Keep version OPNsense\CaptivePortal\CaptivePortal (1.0.0)
Keep version OPNsense\Cron\Cron (1.0.1)
Keep version OPNsense\Diagnostics\Netflow (1.0.0)
Keep version OPNsense\Firewall\Alias (1.0.0)
Keep version OPNsense\IDS\IDS (1.0.3)
*** OPNsense\Monit\Monit Migration failed, check log for details
Keep version OPNsense\OpenVPN\Export (0.0.1)
Keep version OPNsense\Proxy\Proxy (1.0.3)
Migrated OPNsense\Routes\Route from  <unversioned>  to 1.0.0
Keep version OPNsense\Syslog\Syslog (1.0.0)
Keep version OPNsense\Dnscryptproxy\Forward (0.1.0)
Keep version OPNsense\Dnscryptproxy\General (0.1.0)
Keep version OPNsense\Dnscryptproxy\Whitelist (0.1.0)
Keep version OPNsense\Dnscryptproxy\Server (1.0.0)
Keep version OPNsense\Dnscryptproxy\Dnsbl (1.0.0)
Keep version OPNsense\Dnscryptproxy\Cloak (0.1.0)
Keep version OPNsense\iperf\FakeInstance (0.0.0)
Keep version OPNsense\ARPscanner\ARPscanner (1.0.0)
Keep version OPNsense\TrafficShaper\TrafficShaper (1.0.3)
Writing firmware setting...done.
Configuring login behaviour...done.
Configuring system logging...done.
Message from opnsense-19.7.1:

Roar!
Your system is up to date.
Starting web GUI...done.
Generating RRD graphs...done.
***DONE***

Boot had a require_once (script/load_phalcon.php) failed to open stream warning in legacy_bindings.inc line 29.

Yep, but the hardest part is building a plugin around it so you can manage it via GUI.

I ended up bypassing bind and making a bash script to convert blacklists into unbound format. I now have basically all of the pihole blacklists running.

My only concern is how large blacklists may impact performance and if it would be a better idea to run them in a ramdisk.

I'm just going to go make a feature request in 19.7 for DNSBL on Unbound. Thanks for taking the time to reply though!

You'd not be the first one :)

So I'm not exactly familiar with how DNSBL works. This - https://github.com/oznu/dns-zone-blacklist/tree/master/unbound - for unbound does the same thing as the DNSBL on bind/dnscrypt, correct?

If so, it probably wouldn't be hard to convert the existing blacklists into unbound format, yeah?

Can we get DNSBL for unbound? It's available for both bind and dnscrypt.

Unbound can blacklist based on config files so this should be possible.

I had a report the daemon always crashes when you have IPv6 enabled in plugin but not on interface.
I'm using it at home perfectly fine all the time.

I'm pretty sure it was disabled on the plugin side as well.

I'm just going to go make a feature request in 19.7 for DNSBL on Unbound. Thanks for taking the time to reply though!

Why not using dnscrypt plugin and DoH?

dnscrypt w/wo DoH stopped working constantly and needed to be restarted each time. So, the plugin seems a little borked.

Besides unbound is great with tls. It would be nice if it had the same dnsbl bind/dnscrypt have though.

I'd like to use the DNSBL in bind, but want to continue using unbound as my DNS server. BIND has a dns-forward option, but it's very specific about it only being for unknown dns queries.

Ideally, I'd like it to go like so for every DNS query:

BIND (53) / DNSBL -> Unbound (port 5353) / ad-blacklist, domain blacklist config files, etc -> DNSEC/TLS via port 853 to 1.1.1.1

Would setting dns-forward in bind be all I need to do?

No, the switch is not vlan capable.

After testing some more it does function correctly when the FIOS router's DHCP is disabled.

Mmm. Looks like I can't do both without vlan functionality.